Networks for Learning: Regression and Classification

Bertero, M., T. Poggio, and V. Torre. “Ill-posed Problems in Early Vision.” Proc. of the IEEE 76 (1988): 869-889.

Girosi, F., M. Jones, and T. Poggio. “Regularization Theory and Neural Network Architectures.” Neural Computation 7 (1995): 219-269.

Vapnik, V. The Nature of Statistical Learning Theory . Springer, 1995.

Chapter 1 is a readable first-hand introduction to the subject.

Bertero, M. “Regularization Methods for Linear Inverse Problems.” In Inverse Problems . Edited by G. Talenti. Lecture Notes in Mathematics. Vol. 1225. 1986, pp. 52-112.

Tikhonov, A. N. and V. Y. Arsenin. Solutions of Ill-posed Problems . W. H. Winston, 1977.

Vapnik, V. Statistical Learning Theory . Wiley, 1998.

Browse the first chapters of this book if you want to go deeper into the foundations of SLT.

Kolmogorov, N., and S.V. Fomine. Elements of the Theory of Functions and Functional Analysis. Dover, 1975.

Strang, G. Calculus. Wellesley-Cambridge Press, 1991.

Chapter 13 contains an excellent exposition of the Lagrange multipliers technique.

Bertero, M. “Regularization Methods for Linear Inverse Problems” In Inverse Problems. Edited by G. Talenti. Lecture Notes in Mathematics. Vol. 1225. 1986, pp. 52-112.

Tikhonov, A. N., and V. Y. Arsenin. Solutions of Ill-posed Problems . W. H. Winston, 1977.

Kolmogorov, N. and S. V. Fomine. Elements of the Theory of Functions and Functional Analysis . Dover, 1975.

Strang, G. Introduction to Linear Algebra . Wellesley-Cambridge Press, 1993.

Chapter 6 contains the matrix algebra used in this class (and more!). ** Further Readings:**

Aronszajn, N. “Theory of Reproducing Kernels.” Trans. Amer. Math. Soc. 686 (1950): 337-404.

Girosi, F. “An Equivalence Between Sparse Approximation and Support Vector Machines.” Neural Computation 10 (1998): 1455-1480.

Wahba, G. Spline Models for Observational Data. SIAM, 1990.

Chapter 1 introduces you to the world of RKHS.

Strang, G. Calculus . Wellesley-Cambridge Press, 1991.

Vapnik, V. N. Estimation of Dependences Based on Empirical Data. Springer, 1982.

Chapter 9 contains a discussion of the Parzen windows method within the framework of Regularization Theory.

Bishop, C. M. Neural Networks for Pattern Recognition . Clarendon, 1995.

Chapters 3 and 4 discuss single and multi-layer perceptrons at length.

Hertz, J., A. Krogh, and R. G. Palmer. Introduction to the Theory of Neural Computation. Addison Wesley, 1991.

Hertz, J., A. Krogh, and R. G. Palmer. Introduction to the Theory of Neural Computation . Addison Wesley, 1991.

Vapnik, V. Statistical Learning Theory. Wiley, 1998.

Chapter 3 contains all the material covered in this class (and much more!). Several parts of Chapter 2 give you the perspective behind the theory, but if you want to appreciate the difference between stating a result and proving it, browse chapter 14…

Chapter 4 contains all the material covered in this class (and much more!)

Alon, N., et al. “Scale Sensitive Dimensions, Uniform Convergence, and Learnability.” Symposium on Foundation of Computer Science (1993).

Evgeniou, T., M. Pontil, and T. Poggio. “Regularization Networks and Support Vector Machines.” Advances in Computational Mathematics 13 (2000): 1-50.

Chapters 5 and 6 tell you most but not the whole story about the results discussed in this class.

This class will cover part of chapter 10. You may want to go through chapter 8 for putting SVMs in perspective with respect to other techniques.

Girosi, F. “An Equivalence between Sparse Approximation and Support Vector Machines.” Neural Computation 10 (1998): 1455-1480.

This class will cover part of chapters 11 and 13.

Chen, S., D. Donoho, and M. Saunders. “Atomic Decomposition by Basis Pursuit.” Tech Rep 479. Dept. of Statistics. Stanford University. 1995.

Daubechies, I. “Time Frequency Localization Operators: a Geometric Phase Space Approach.” IEEE Trans. on Information Theory 34 (1988): 605-612.

Mallat, S., and S. Zhang. “Matching Pursuits with Time-Frequency Dictionaries.” IEEE. Trans. on Signal Proc. 41 (1993): 3397-3415.

Pontil, M., S. Mukherjee, and F. Girosi. “On the Noise Model of Support Vector Machine Regression.” CBCL Paper #168, AI Memo #1651, Massachusetts Institute of Technology, Cambridge, MA (1998).

Cristianini, N., and J. Shawe-Taylor. Support Vector Machines and Other Kernel-based Learning Methods. Cambridge, 2000.

Chapter 3 of this book covers kernels in depth.

You’ll find kernels and ideas on kernels throughout chapters 10, 11 and 12.

The title is intimidating, but chapter 3 is easy to read and contains a lucid introduction to positive definite functions.

Jaakkola, T., and D. Haussler. “Exploiting Generative Models in Discriminative Classifiers.” NIPS (1998).

Niyogi, P., T. Poggio, and F. Girosi. “Incorporating Prior Information in Machine Learning by Creating Virtual Examples.” IEEE Proceedings on Intelligent Signal Processing 86 (1998): 2196-2209.

Logothetis, N. K., J. Pauls, and T. Poggio. “Viewered-Centered Object Recognition in Monkeys .” AI Memo 1472, CBCL Paper 95 (1994).

Logothetis, N. K., T. Vetter, A. Hulbert, and T. Poggio. “View-Based Models of 3D Object Recognition and Class-Specific Invariances.” AI Memo 1473, CBCL Paper 94 (1994).

Riesenhuber, M., and T. Poggio. “Hierarchical Models of Object Recognition in Cortex.” Nature Neuroscience 2 (1999): 1019-1025.

Lorentz, G. G. “Approximation of Functions.” Chelsea Pub co, 1986.

Not easy but very compact and rigorous introduction to the subject (Chapters 1, 5 and 8-10 in particular).

Niyogi, P., and F. Girosi. “On the Relationship between Generalization Error, Hypothesis Complexity, and Sample Complexity for Radial Basis Functions.” Neural Computation 8 (1996): 819-842.

Bazaraa, Sherali, and Shetty. Nonlinear Programming, Theory and Techniques. John Wiley & Sons, 1993. A textbook on Optimization Theory.

Is the SVM solution unique? Burges, and Crisp. Uniqueness of the SVM Solution NIPS 12 (1999).

**The Decomposition Method for SVMs: **Osuna, Edgar. Support Vector Machines: Training And Applications. Ph.D. Thesis (1998).

Optimizing over 2 variables at a time: Platt, John C. “Sequential Minimal Optimization: A Fast Algorithm For Training Support Vector Machines.” Microsoft Research MST-TR-98-14 (1998).

Analysis of the Decomposition Method: Chang, Chih-Chung, Chih-Wei Hsu, and Chih-Jen Lin. “The Analysis of Decomposition Methods For Support Vector Machines.” Proceedings of IJCAI99 , SVM workshop (1999).

Keerthi, S. S., and E. G. Gilbert. Convergence of a Generalized SMO Algorithm For SVM Classifier Design Control Division . Dept. of Mechanical and Production Engineering, National University of Singapore CD-00-01 (2000).

Keerthi, S. S., S. K. Shevade, C. Bhattacharyya, and K. R. K. Murthy. Improvements to Platt’s SMO Algorithm for SVM Classifier Design Control Division. Dept. of Mechanical and Production Engineering, National University of Singapore CD-99-14 (1999).

Sparsity Control: Osuna, Freund, and Girosi. “Reducing Run-time Complexity in SVMs.” Proceedings of the 14th Int’l Conference on Pattern Recognition.

Breiman, L. “Bagging Predictors.” Machine Learning 26 (1996): 123-140.

Schapire, R. E., Y. Freund, P. Bartlett, and W. S. Lee. “Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods.” The Annals of Statistics 26 (1998): 1651-1686.

Stollnitz, E. J., T. D. DeRose, and D. H. Salesin. “Wavelets for Computer Graphics: A Primer Department of Computer Science and Engineering.” University of Washington Tech Rep 94-09-11 (1994).

Daubechies, I. “Ten Lectures on Wavelets.” CBMS-NSF Regional Conferences Series in Applied Mathematics , SIAM, Philadelphia PA (1992).