A Comprehensive Overview of Kernels in Machine Learning: Mathematical Foundations and Applications
Keywords:
Kernel Applications, Mathematical Foundations of Kernels, Kernel Trick, Kernel PropertiesAbstract
Kernels play a fundamental role in machine learning, enabling algorithms to operate efficiently and effectively in high-dimensional spaces. In this paper, we provide a comprehensive overview of regression kernels in machine learning, focusing on their mathematical foundations, properties, and practical applications. We begin with an introduction to the concept of regression kernels and their significance in machine learning. Then, we delve into the mathematical formulation of regression kernels, exploring Mercer's theorem and positive semi-definite (PSD) kernels. Next, we discuss popular kernel functions with their respective properties and applications. After that we apply regression kernel to the bike sharing demand dataset as a case study and compare the different kernel functions. Finally, we explore kernel limitations and current research trends and emerging directions in kernel-based learning, offering insights into the future potential of this powerful methodology. This work aims to serve as a resource for both researchers and practitioners seeking a thorough understanding of regression kernel-based approaches in machine learning.
References
Antoine Bordes, Seyda Ertekin, Jason Weston, Le ?on Bottou “Fast Kernel Classifiers with Online and Active Learning”. Journal of Machine Learning Research 6, pages 1579-1619, (2005).
Bernhard Scho ?lkopf, Alexander J. Smola, “Leaning with Kernels”. The MIT Press, 2002. ISBN 0-262-19475-9
K.-R. Muller, S. Mika, G. Ratsch, K. Tsuda and B. Scholkopf, "An introduction to kernel-based learning algorithms," in IEEE Transactions on Neural Networks, vol. 12, no. 2, pp. 181-201, March 2001
Mercer, J., "Functions of positive and negative type and their connection with the theory of integral equations", Philosophical Transactions of the Royal Society A, 209 (441–458): 415–446, 1909.
Cortes, C. and Vapnik, V., “Support-Vector Networks”. Machine Learning, 20, 273-297, 1995.
http://dx.doi.org/10.1007/BF00994018.
Saunders, C., Gammerman, A. and Vovk V., “Ridge Regression Learning Algorithm in Dual Variables”. ICML 98: Proceedings of the Fifteenh International Conference on Machine Learning. Pages 515-521, 1998.
Schölkopf et al., “Kernel Methods in Computational Biology”. 2004 MIT Press (July 16, 2004)
Schölkopf B. and Smola, A., “Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond”. The MIT Press, Cabridge Massachusetts, London, England. 2004.
Genton, M. G., “Classes of Kernels for Machine Learning: A Statistics Perspective”. Journal of Machine Learning Research 2, 299-312, 2001.
Bergstra, J. and BengioRandom, Y., “Search for Hyper-Parameter Optimization”. Journal of Machine Learning Research 13, 281-305, 2012.
Snoek , J., Larochelle, H. and Adams, R. P. “Practical Bayesian Optimization of Machine Learning Algorithms”. Curran Associates, Inc.Vol 25, 2012.
Weinberger., K., “Kernel Notes from Cornell CS4780 class”, Internet: https://www.cs.cornell.edu/courses/cs4780/2018fa/lectures/lecturenote13.html. Spring 2017
Zareapoor, M., Shamsolmoali, P., Jain, D. K., Hoaxiang W. and Jie Y., “Kernelized support vector machine with deep learning: An efficient approach for extreme multiclass dataset”. Pattern Recognition Letterss, volume 115, pages 4-13, 2018.
Herbrich, R., “Learning Kernel Classifiers Theory and Algorithms”. The MIT Press, 2001, ISBN: 9780262256339.
Schölkopf, B., and Smola, A. J., “Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond”. MIT Press, 2002.
Shawe-Taylor, J. and Cristianini, N. “Kernel Methods for Pattern Analysis”. Cambridge University Press, 2004.
Hofmann, T., Scholkopf, B. and Smola, A. J., “A Review of Kernel Methods in Machine Learning”. Max Planck Institute for Biological Cybernetics, Technical report No. 156, 2006.
Hofmann, T., Schölkopf. B., and Smola, A. J., "Kernel methods in machine learning" Annals of Statistics 36 (3) 1171 - 1220, June 2008.
Schölkopf, B., Smola, Alex; and Müller, Klaus-Robert, "Nonlinear Component Analysis as a Kernel Eigenvalue Problem". Neural Computation. 10 (5): 1299–1319, 1998. doi:10.1162/089976698300017467. S2CID 6674407.
Schölkopf, B., Smola, Alex. and Müller, Klaus-Robert, Nonlinear Component Analysis as a Kernel Eigenvalue Problem (PDF) (Technical report). Max-Planck-Institut für biologische Kybernetik. 44, 1996.
Wilson A., Hu Zhiting, Salakhutdinov R. and Xing P. E., “Deep Kernel Learning”. Published of the 19th International Conference on Artificial Intelligence and Statistics, 6 November 2015.
Majumdar, A. “Kernelized Linear Autoencoder”. Neural Process Lett 53, 1597–1614 (2021). https://doi.org/10.1007/s11063-021-10467-0
Zheng, X., Ni, Z., Zhong X. and Luo Y., "Kernelized Deep Learning for Matrix Factorization Recommendation System Using Explicit and Implicit Information," in IEEE Transactions on Neural Networks and Learning Systems, vol. 35, no. 1, pp. 1205-1216, Jan. 2024, doi: 10.1109/TNNLS.2022.3182942.
Cho, Y. and Saul, L., “Kernel methods for deep learning”. Advances in neural information processing systems, 2022.
Wu, C., Khan, Z., Chang, Y., Ioannidis, S. and Dy, J., “Deep Kernel Learning for Clustering”. Proceedings of the 2020 SIAM International Conference on Data Mining, pages 640-648, 2020.
Murphy, K. P., “Machine Learning: A probabilistic Perspective”. The MIT Press, Chapter 14.4.3, pp. 492-493, 2012.
Ali Rahimi and Ben Recht, “Random Feature for Large-Scale-Kernel Machines”. Advances in Neural Information Processing Systems, 2017.
Williams, C. K. I. and Seeger, M, “Using the Nyström Method to Speed Up Kernel Machines”. In T. K. Leen, T. G. Dietterich, & V. Tresp (Eds.), Advances in Neural Information Processing Systems 13 (NIPS) (pp. 682-688), 2001.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Omar EL Khatib, Nabeel Alkhatib
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who submit papers with this journal agree to the following terms.
