Special Topics: Spectral Techniques for Machine Learning

Course number: TTIC 41000
Instructor: Karl Stratos
Time: M 3-4:20pm
Location: TTIC Room 526
Office hours: M 4:30-5pm

Course description. This special topics course will examine techniques that use eigenvalues/eigenvectors of a matrix (and more generally, any linear algebraic tools) to solve or understand problems in modern machine learning. The course will be accompanied by lectures on technical materials required to understand and derive spectral techniques. As part of the course, students will take turn to present papers on recent research in this area.

  1. Achieving an understanding of the foundational concepts and tools used in modern spectral methods
  2. Obtaining an ability to accurately evaluate new works in this area at conferences
  3. Finding new research projects that persist beyond this course

Audience and prerequisites. The expected audience is student researchers in machine learning who are interested in learning about and doing novel research on this topic. The main prerequisite is general mathematical maturity and familiarity with basic notions in linear algebra, statistics, and optimization.

Grading. The course will be pass-fail. Students who actively engage in the class will receive a pass. A significant part of this evaluation will also be the technical quality of the student presentation.

Tentative plan.
Oct 1 Introduction, Vector Space Review
Oct 3 (Wed) Hilbert Space, Matrix Decomposition (Dudeja and Hsu, 2018)
Oct 8 Spectral Learning of HMMs (Hsu et al., 2008; Foster et al., 2012; Balle et al., 2014)
Oct 15 Canonical Correlation Analysis (Golub and Zha, 1992; Andrew et al., 2013)
Some notes on deep CCA
Comparing matrix ranges
Oct 22 Paper Discussion (Wang et al., 2015b)
Oct 29 Paper Discussion (Chapter 2 of Peyre and Cuturi)
Nov 5 Paper Discussion (Chapter 4 of Peyre and Cuturi; Peyre et al., 2016; Alvarez-Melis and Jaakkola, 2018; Zhang et al., 2017)
Nov 12 Paper Discussion (Von Luxburg, 2007; Belkin and Niyogi, 2002)
Nov 19 Paper Discussion (Sedoc et al., 2017; Halko et al., 2011)
Nov 26 Paper Discussion (Goodfellow et al., 2014; Arjovsky et al., 2017; Arora et al., 2018; Miyato et al., 2018)
Dec 3 Paper Discussion (Anandkumar et al., 2014; Sedghi et al., 2016)

Papers. Please expand recursively for a more complete list.
Topic Papers
Efficient algorithms
Analysis of CCA
Subspace identification
Tensor decomposition
Optimal transport
Neural networks
Clustering/graph cuts
Kernel approximation

Other resources.