Provides a rigorous foundation in linear model theory, covering estimation, inference, and prediction. Topics include a review of matrix algebra, distribution theory, least squares and maximum likelihood estimation methods, inference, misspecification, ANOVA, reparameterization, non-full rank models, and linear mixed models. Emphasis is on theory and its implications for applications.
Athena Title
Theory of Linear Models
Prerequisite
(STAT 6420 and STAT 6421) or permission of department
Pre or Corequisite
STAT 6820 or STAT 6520
Semester Course Offered
Offered spring
Grading System
A - F (Traditional)
Student learning Outcomes
Students will apply concepts of vector and matrix algebra, including eigenvalues, eigenvectors, and quadratic forms, to formulate and solve linear model problems.
Students will analyze the distributional properties of least squares estimators using multivariate normal theory, quadratic forms, and the Wishart distribution.
Students will evaluate the assumptions and properties of the classical linear model, demonstrating an understanding of the Gauss-Markov theorem and Best Linear Unbiased Estimators (BLUE).
Students will implement and distinguish between estimation techniques such as ordinary, weighted, and generalized least squares, as well as maximum likelihood and restricted maximum estimation, to derive and interpret estimators of model parameters.
Students will formulate hypothesis tests and confidence intervals/regions in linear and linear mixed-effect models based on t-tests, F-tests, likelihood ratio tests, and approximations thereof, including Satterthwaite and Kenward-Roger approximations.
Students will construct and interpret ANOVA models, including fixed, random, and mixed-effects models for experimental designs, and apply contrasts and multiple comparison methods for statistical inference.
Students will communicate statistical findings effectively through written assignments, using rigorous mathematical justification and appropriate interpretation of model results in real-world applications.
Topical Outline
A Review of Vector and Matrix Algebra
• Vectors and vector spaces
• Matrices and their properties (rank, trace, determinant)
• Special matrices (idempotent, orthogonal, projection matrices)
• Eigenvalues, eigenvectors, and spectral decomposition
• Quadratic forms and their applications in statistics
Distribution Theory
• Multivariate normal distribution
• Distribution of quadratic forms
• Wishart distribution and its applications
• Central limit theorem in a multivariate setting
• Properties of least squares estimators from a distributional perspective
The Classical Linear Model – Full Rank Case
• General form and assumptions of the classical linear model.
• Properties of least squares estimators (unbiasedness, efficiency)
• Gauss-Markov theorem and the Best Linear Unbiased Estimator (BLUE)
• Geometric interpretation of least squares estimation
Estimation in Linear Models
• Ordinary Least Squares (OLS): Normal equations, properties, and interpretation
• Weighted Least Squares (WLS): Motivation, estimation, and applications
• Generalized Least Squares (GLS): Handling correlated and heteroscedastic errors
• Maximum Likelihood Estimation (MLE): Derivation, properties, and asymptotic results
Inference for Linear Models
• Confidence intervals for regression coefficients
• Hypothesis testing in linear models (t-tests, F-tests)
• Likelihood ratio, Wald, and score tests
• Model diagnostics and goodness-of-fit measures
• Multicollinearity and its impact on inference
Reparameterization in Linear Models
• Reasons for reparameterization
• Orthogonalization techniques
• Centering and standardization of predictors
• Constraints in parameter estimation
Non-Full Rank Case
• Identifiability and the concept of estimability
• Constraints and testability in rank-deficient models
• Generalized inverse solutions for estimation
• Applications to overparameterized models (e.g., effects model in ANOVA)
Analysis of Variance (ANOVA)
• One-way and two-way ANOVA models
• Fixed, random, and mixed-effects models
• Decomposing total variation and interpreting sums of squares
• Inference on contrasts and marginal means.
• Multiple comparisons and post-hoc tests (Tukey, Bonferroni, Scheffé)
• ANOVA and regression relationship
Prediction in Linear and Linear Mixed Models
• Point prediction and interval estimation
• Prediction in full-rank vs. non-full-rank models
• Prediction under model uncertainty
• Mean squared error of prediction and shrinkage estimators
Theory of Linear Mixed Models (LMMs)
• Introduction to random effects and hierarchical models
• Specification of LMMs and variance components estimation
• Restricted Maximum Likelihood (REML) estimation
• Fixed effect estimation vs. prediction of random effects
• Constrained inference on variance-covariance parameters.
• Approximate inference on fixed effects, including Satterthwaite and Kenward-Roger inference.
Institutional Competencies Learning Outcomes
Analytical Thinking
The ability to reason, interpret, analyze, and solve problems from a wide array of authentic contexts.
