Computer techniques for processing human languages, with a focus on textual data. Students will develop deep and intuitive understandings of quantitative and algorithmic principles behind Natural Language Processing techniques and use those techniques creatively to solve theoretical and applied problems in linguistics and language studies, including language technology.
Additional Requirements for Graduate Students: Graduate students will complete a pilot study for an original research project connected to their research interests, with a formal write-up in the form of a short (four-page) paper for the Association of Computational Linguistics Anthology. Regular assignments over the course of the semester will include additional questions for graduate students to stimulate thinking about the application of Natural Language Processing techniques to theoretical problems in linguistics.
Athena Title
Natural Language Processing
Undergraduate Prerequisite
(LING 2100 or LING 2100E or LING 2100H) and (CSCI 1300-1300L or CSCI 1360 or CSCI 1360E or LING 2200)
Graduate Prerequisite
Permission of department
Grading System
A - F (Traditional)
Student Learning Outcomes
Students will be able to name the basic components of a computational model (e.g. training data, model architecture, training objective) and describe the components of specific computational models.
Students will be able to describe and implement simple algorithms for model training and inference, such as expectation maximization, the Viterbi algorithm, and gradient descent.
Students will be able to evaluate NLP models quantitatively and qualitatively in terms of characteristics like accuracy, generalization ability, and social impact.
Students will be able to explain why different models work well for different types of tasks and data.
Students will be able to describe how different NLP model architectures are applied to different NLP tasks, such as text classification, part-of-speech tagging, and natural language generation.
Students will be able to apply NLP tasks to significant theoretical and applied problems in linguistics and adjacent fields.
Topical Outline
1. Introduction to NLP: linguistic annotation tasks; downstream NLP tasks; anatomy of an NLP model; NLP and linguistics; NLP and AI ethics
2. Language as sequences: n-grams and smoothing; word-level tokenization; Viterbi algorithm; subword tokenization
3. Static word vectors: vector spaces; singular value decomposition; latent semantic analysis; GLoVE; imbalance in word embeddings; debiasing
4. Shallow classification: Bayes’ theorem and the noisy channel model; sentiment analysis; word sense disambiguation; naïve Bayes; logistic regression; gradient descent; regularization
5. Feed-forward neural networks: the perceptron; word2vec and the projection layer; stochastic gradient descent; the chain rule and backpropagation
6. Recurrent neural networks: simple recurrent networks; backpropagation through time; bidirectional RNNs; dynamic word embeddings; question-answering systems; seq2seq models and attention; long short-term memory networks
7. Transformers and large language models: pre-training and fine-tuning; self-attention; encoder-only transformers; BERT; chatbots; decoder-only transformers; reinforcement learning from human feedback; retrieval-augmented generation
8. Social impacts of NLP: fairness and representation; environmental costs; security and privacy issues; labour issues; toxicity and hallucination
