CS 533: Natural Language Processing (Spring 2021)
Instructor: Karl Stratos (karl.stratos@rutgers.edu)
TA: TBD
Instructor office hours: Tuesday 45pm, or by appointment
TA office hours: TBD
Course format.
The course will be asynchronous remote.
Instead of having specific lecture times, there will be constant online interactions on Canvas along with weekly virtual office hours.
The course will nonetheless be highly structured. Here is what to expect:
 One or more video lectures will be uploaded on Canvas every week. An advantage of this format is that you can watch the lecture at your own convenience.
 Along with lectures, there will be optional reading from publicly available online textbooks and other resources (the slides are selfcontained).
 The course will be driven mainly by written/programming assignments based on the materials covered in the lectures.
You can ask questions regarding lectures/assignments anytime on Canvas and in the virtual office hours.
There will be a dedicated turnaround time of two days (if not sooner) for any question posted on Canvas.
To guarantee fast turnaround and also to involve other students, please use the Canvas discussion board and only use other means of communication like email for private matters.
 In addition to assignments, there will be an entrance quiz, a series of short timed quizzes, and a project.
Entrance quiz.
This is a technical course. On the first day of the class at a designated time, we will have a timed entrance quiz on Canvas to assess if you have a suitable background.
The entrance quiz will count as 5% of the final grade.
Project.
The last month of the course will be used for projects.
The lectures will present selected conference papers to illustrate the current research landscape of the field.
You will select a recent paper (approved by the instructor as acceptable), replicate and possibly build on its results,
and submit a short video presentation and a final report of your findings.
Academic integrity policy.

Assignments: collaboration is allowed and encouraged, as long as you (1) write your own solution entirely on your own, and (2) specify names of student(s) you collaborated with in your writeup.
If you find a solution online, clearly acknowledge the source and still write your own solution entirely on your own.
Copying solutions from others or from the internet is strictly prohibited.
 Quizzes: cheating is strictly prohibited.
 Project: collaboration up to 3 is allowed.
If the student is caught in cheating/plagiarism, the incident will be reported to the office of student conduct and he/she will get zero point for the assignment/quiz/exam, which will result in a low final grade.
Course description.
This graduatelevel course will cover technical foundations of modern natural language processing (NLP).
The course will cast NLP as an application of machine learning, in particular deep learning,
and focus on deriving general mathematical principles that underlie stateoftheart NLP systems today.
Goals.
 Understanding the goals, capabilities, and principles of NLP
 Acquiring mathematical tools to formalize NLP problems
 Acquiring implementation skills to build practical NLP systems
 Obtaining an ability to critically read and accurately evaluate conference papers in NLP
Audience and prerequisites.
No previous exposure to NLP is assumed.
However, the course will be most beneficial for students with some programming experience and familiarity with basic concepts in probability and statistics, calculus, and linear algebra.
Examples of such concepts include
 Random variables (continuous or discrete), expectation, mean/variance
 Matrix and vector operations
 Derivatives, partial derivatives, gradients
 Programming (in Python): familiarity with data structures and algorithms
If you are an undergraduate, prerequisites are as follows:
 Required: M250 (linear algebra), 112 (data structures), 206 (discrete II)
 Recommended: M251 (multivariable calculus), 533 (machine learning)
 Alternatives to 206: M477 (probability), S379 (basic probability theory), or instructor's permission
The entrance quiz will be useful for evaluating whether you meet the prerequisites.
Grading.
 Assignments: 50%
 Entrance quiz: 5%
 Quizzes (excluding the entrance quiz): 15%
 Project: 30%
The assignment report must be written in LaTeX using a provided assignment report template.
Similarly, the project report must be written in LaTeX using a provided project report template.
If you have never used LaTeX before, you can pick it up quickly ( tutorial, style guide).
The project will be due at the end of the semester and graded as follows:
 10minute presentation: 1/3 of the project grade. The structure should strictly adhere to
paperwriting tips by Jennifer Widom
and clearly explain (1) what the problem is, (2) why it is interesting and important,
(3) why it is hard, (4) why previous approaches fail, and (5) what the key components of your approach and results are.
 4page report (excluding references): 2/3 of the project grade. This should flesh out the presentation with details like a conference paper.
Tentative plan.
Date 
Topics 
Week 1 (Jan 18) 
General introduction and review of prerequisites 
Week 2 (Jan 25) 
Linear classification: score function, softmax, loss function, stochastic gradient descent, regularization 
Week 3 (Feb 1) 
Deep learning: nonliniearity and universality of neural networks, backpropagation 
Week 4 (Feb 8) 
Sequencetosequence models: chain rule, perplexity, training and beam search 
Week 5 (Feb 15) 
Neural architectures for sequences: convolutional, recurrent, and transformer 
Week 6 (Feb 22) 
Structured prediction: Markov random fields, conditional random fields, Viterbi, CKY 
Week 7 (Mar 1) 
Selfsupervised representation learning: word embeddings, contextual word embeddings, masked language modeling 
Week 8 (Mar 8) 
Unsupervised learning: latentvariable models, expectation maximization, variational autoencoders 
Week 9 (Mar 22) 
Information extraction: knowledge base, entity linking, relation extraction, question answering 
Week 10 (Mar 29) 
Text generation: translation, summarization, image captioning, datatotext generation, dialogue 
Week 11 (Apr 5) 
Project phase: selected conference paper 
Week 12 (Apr 12) 
Project phase: selected conference paper 
Week 13 (Apr 19) 
Project phase: selected conference paper 
Week 14 (Apr 26) 
Project phase: selected conference paper 
Online resources.
 Natural Language Processing by Jacob Eisenstein
 A Primer on Neural Network Models for Natural Language Processing by Yoav Goldberg
