Introduction to Regularization: Ridge Regression and the LASSO

Dr. Gilbert

September 1, 2024

Reminders from Last Time

Cross-validation is a procedure that we can use to obtain more reliable performance assessments for our models

In \(k\)-fold cross validation, we create \(k\) models and obtain \(k\) performance estimates – the average of these performance estimates can be referred to as the cross-validation performance estimate

Our cross-validation procedure does NOT result in a fitted model, but results in the cross-validation performance estimate and an estimated standard error for that model performance

Cross-validation makes our choices and inferences less susceptible to random chance (the randomly chosen training and test observations)

Big Picture Recap

Our approach to linear regression so far has perhaps led us to the intuition that we should start with a large model and then reduce it down to include only statistically significant terms

This approach, called backward elimination, is commonly utilized

There is also an opposite approach, called forward selection

Playing Along

We’ll switch to using the ames dataset for this discussion

That dataset contains features and selling prices for 2,930 homes sold in Ames, Iowa between 2006 and 2010

  1. Open your MAT300 project in RStudio and create a new Quarto document

  2. Use a setup chunk to load the {tidyverse} and {tidymodels}

  3. The ames data set is contained in the {modeldata} package, which is loaded with {tidymodels} – take a preliminary look at this dataset

  4. Split your data into training and test sets

  5. Create five or ten cross-validation folds

A Shopping Analogy

Consider the model (or us, as modelers) as a shopper in a market that sells predictors

  • (Backward elimination) Our model begins by putting every item in the store into its shopping cart, and then puts back the items it doesn’t need

  • (Forward selection) Our model begins with an empty cart and wanders the store, finding the items it needs most to add to its cart one-by-one

Okay, So What?

At first, these approaches may seem reasonable, if inefficient

Statistical Standpoint: We’re evaluating lots of \(t\)-tests in determining statistical significance of predictors

  • The probability of making at least one Type I error (claiming a term is significant when it truly isn’t) becomes inflated – even with just three tests, the probability is over 14% when using the common \(\alpha = 0.05\)

Model-Fit Perspective: The more predictors a model has access to, the more flexible it is, the better it will fit the training data, and the more likely it is to become overfit

via GIPHY

Back to the Shopping Analogy

By allowing a model to “shop” freely for its predictors, we are encouraging our model to become overfit

Giving our model a “budget” to spend on its shopping trip would force our model to be more selective about the predictors it chooses, and lowers the likelihood that it becomes overfit

A Look Under the Hood

We’ve hidden the math that fits our models up until this point, but its worth a look now

\[\mathbb{E}\left[y\right] = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_k x_k\]

The optimization procedure we’ve been using to find the \(\beta\)-coefficients is called Ordinary Least Squares

Ordinary Least Squares: Find \(\beta_0, \beta_1, \cdots, \beta_k\) in order to minimize

\[\sum_{i = 1}^{n}{\left(y_{\text{obs}_i} - y_{\text{pred}_i}\right)^2}\]

A Look Under the Hood

We’ve hidden the math that fits our models up until this point, but its worth a look now

\[\mathbb{E}\left[y\right] = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_k x_k\]

The optimization procedure we’ve been using to find the \(\beta\)-coefficients is called Ordinary Least Squares

Ordinary Least Squares: Find \(\beta_0, \beta_1, \cdots, \beta_k\) in order to minimize

\[\sum_{i = 1}^{n}{\left(y_{\text{obs}_i} - \left(\beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \cdots + \beta_k x_{ik}\right)\right)^2}\]

A Look Under the Hood

We’ve hidden the math that fits our models up until this point, but its worth a look now

\[\mathbb{E}\left[y\right] = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_k x_k\]

The optimization procedure we’ve been using to find the \(\beta\)-coefficients is called Ordinary Least Squares

Ordinary Least Squares: Find \(\beta_0, \beta_1, \cdots, \beta_k\) in order to minimize

\[\sum_{i = 1}^{n}{\left(y_{\text{obs}_i} - \left(\beta_0 + \sum_{j = 1}^{k}{\beta_j x_{ij}}\right)\right)^2}\]

This is the procedure that allows our model to shop freely for predictors

Regularization

Regularization refers to techniques design to constrain models and reduce the likelihood of overfitting

For linear regression, there are two commonly used methods

  • Ridge Regression
  • The LASSO (least absolute shrinkage and selection operator)

Each of these methods makes an adjustment to the Ordinary Least Squares procedure we just saw

Regularization: Ridge Regression

\[\mathbb{E}\left[y\right] = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_k x_k\]

Ridge Regression: Find \(\beta_0, \beta_1, \cdots, \beta_k\) in order to minimize

\[\sum_{i = 1}^{n}{\left(y_{\text{obs}_i} - \left(\beta_0 + \sum_{j = 1}^{k}{\beta_j x_{ij}}\right)\right)^2}\]

subject to the constraint

\[\sum_{j = 1}^{k}{\beta_j^2} \leq C\]

Note: \(C\) is a constraint which can be thought of as our budget for coefficients

The Result: Ridge regression encourages very small coefficients on unimportant predictors

Regularization: The LASSO

\[\mathbb{E}\left[y\right] = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_k x_k\]

LASSO: Find \(\beta_0, \beta_1, \cdots, \beta_k\) in order to minimize

\[\sum_{i = 1}^{n}{\left(y_{\text{obs}_i} - \left(\beta_0 + \sum_{j = 1}^{k}{\beta_j x_{ij}}\right)\right)^2}\]

subject to the constraint

\[\sum_{j = 1}^{k}{\left|\beta_j\right|} \leq C\]

Note: Like with Ridge Regression, \(C\) is a constraint which can be thought of as our budget for coefficients

The Result: The LASSO pushes coefficients of unimportant predictors to \(0\)

LASSO or Ridge?

The choice between Ridge Regression and the LASSO depends on your goals

The LASSO is better at variable selection because it sends coefficients on unimportant predictors to exactly \(0\)

LASSO won’t always out-perform Ridge though, so you might try both and see which is better for your use case

Feature PreProcessing Requirements

If a predictor \(x_i\) is on a larger scale than the response \(y\), then that predictor becomes artificially cheap to include in a model

Similarly, if a predictor \(x_j\) is on a smaller scale than the response, then that predictor becomes artificially expensive to include in a model

We don’t want any of our predictors to be artificially advantaged or disadvantaged, so we must ensure that all of our numerical predictors are on the same scale as one another

Min-Max Scaling: Projects each numerical predictor down to the interval \(\left[0, 1\right]\) via \(\displaystyle{\frac{x - \min\left(x\right)}{\max\left(x\right) - \min\left(x\right)}}\)

  • We include step_range() in a recipe to utilize min-max scaling

Standard Scaling: Converts observed measurements into standard deviations (\(z\)-scores) via \(\displaystyle{\frac{x - \text{mean}\left(x\right)}{\text{sd}\left(x\right)}}\)

  • We include step_normalize() in a recipe to utilize standard scaling

Switching Engines

The {tidymodels} framework is great because it provides us with standardized structure for defining and fitting models

Ridge and the LASSO are still linear regression models, but they’re no longer fit using OLS

This puts them in a class of models called Generalized Linear Models . . .

We’ll need to change our fitting engine from "lm" to something that can fit these GLMs – we’ll use "glmnet"

Required Parameters for "glmnet":

  • mixture can be set to any value between \(0\) and \(1\)

    • Setting mixture = 0 results in Ridge Regression
    • Setting mixture = 1 results in the LASSO

  • penalty is the amount of regularization being applied

    • You can think of this parameter as being tied to our coefficient budget

Choosing a penalty

For now, we’ll just pick a value and see how it performs

We can experiment with several if we like, and then choose the one that results in the best performance

We’ll talk about a better strategy next time

A Few Additional Concerns

  • The "glmnet" engine requires that no missing values are included in any fold

    • We could omit any rows with missing values
    • We could omit any features with missing entries
    • We could impute missing values with a step_impute_*() function added to a recipe

Implementing Ridge and LASSO with the Ames Data

We’ll use the ames housing data set that we’ve seen from time to time this semester

We’ll read it in, remove the rows with missing Sale_Price (our response), split the data into training and test sets, and create our cross-validation folds

ames_known_price <- ames %>%
  filter(!is.na(Sale_Price))

set.seed(123)

ames_split <- initial_split(ames_known_price, prop = 0.9)
ames_train <- training(ames_split)
ames_test <- testing(ames_split)

ames_folds <- vfold_cv(ames_train, v = 5)

Ridge Regression

ridge_reg_spec <- linear_reg(mixture = 0, penalty = 1e4) %>%
  set_engine("glmnet")

ridge_reg_rec <- recipe(Sale_Price ~ ., data = ames_train) %>%
  step_impute_knn(all_predictors()) %>%
  step_normalize(all_numeric_predictors()) %>%
  step_other(all_nominal_predictors()) %>%
  step_dummy(all_nominal_predictors())

ridge_reg_wf <- workflow() %>%
  add_model(ridge_reg_spec) %>%
  add_recipe(ridge_reg_rec)

ridge_reg_results <- ridge_reg_wf %>%
  fit_resamples(ames_folds)

ridge_reg_results %>%
  collect_metrics()
.metric .estimator mean n std_err .config
rmse standard 33512.4783290 5 3528.8858182 Preprocessor1_Model1
rsq standard 0.8285515 5 0.0232516 Preprocessor1_Model1

Seeing the Estimated Ridge Regression Model

While we wouldn’t generally fit the Ridge Regression model at this time, you can see how to do that and examine the estimated model below.

ridge_reg_fit <- ridge_reg_wf %>%
  fit(ames_train)

ridge_reg_fit %>%
  tidy()
term estimate penalty
(Intercept) 170590.91440 10000
Lot_Frontage 1458.08999 10000
Lot_Area 1919.73420 10000
Year_Built 4443.44715 10000
Year_Remod_Add 5652.52473 10000
Mas_Vnr_Area 6469.12318 10000
BsmtFin_SF_1 -109.24172 10000
BsmtFin_SF_2 619.57697 10000
Bsmt_Unf_SF -1897.35902 10000
Total_Bsmt_SF 7471.51817 10000
First_Flr_SF 8231.08414 10000
Second_Flr_SF 8576.05018 10000
Gr_Liv_Area 13259.91733 10000
Bsmt_Full_Bath 2910.80533 10000
Bsmt_Half_Bath -872.28855 10000
Full_Bath 4132.02383 10000
Half_Bath 2520.03561 10000
Bedroom_AbvGr -2600.41177 10000
Kitchen_AbvGr -3311.98370 10000
TotRms_AbvGrd 4547.30499 10000
Fireplaces 4625.12646 10000
Garage_Cars 5750.22932 10000
Garage_Area 4248.50579 10000
Wood_Deck_SF 1899.00078 10000
Open_Porch_SF 224.51413 10000
Enclosed_Porch 1142.26429 10000
Three_season_porch 487.29049 10000
Screen_Porch 3569.00946 10000
Pool_Area -1114.39753 10000
Misc_Val -5085.90458 10000
Mo_Sold -203.12359 10000
Year_Sold -835.49586 10000
Longitude 650.40625 10000
Latitude 5328.30675 10000
MS_SubClass_One_Story_1945_and_Older -4126.27103 10000
MS_SubClass_One_and_Half_Story_Finished_All_Ages -1861.67028 10000
MS_SubClass_Two_Story_1946_and_Newer -1514.91540 10000
MS_SubClass_One_Story_PUD_1946_and_Newer -4117.65295 10000
MS_SubClass_other -1733.26832 10000
MS_Zoning_Residential_Medium_Density -5316.31128 10000
MS_Zoning_other -5176.60141 10000
Street_other -13837.22636 10000
Alley_other -1461.83621 10000
Lot_Shape_Slightly_Irregular 3700.17083 10000
Lot_Shape_other 961.05230 10000
Land_Contour_other 3230.49668 10000
Utilities_other -8536.41564 10000
Lot_Config_CulDSac 11616.63150 10000
Lot_Config_Inside 357.65609 10000
Lot_Config_other -5147.76184 10000
Land_Slope_other 3944.23456 10000
Neighborhood_College_Creek 1894.52790 10000
Neighborhood_Old_Town -1468.07869 10000
Neighborhood_Edwards -7179.02156 10000
Neighborhood_Somerset 12865.77564 10000
Neighborhood_Northridge_Heights 35341.47415 10000
Neighborhood_Gilbert -19167.19093 10000
Neighborhood_other 8049.59728 10000
Condition_1_Norm 10577.33452 10000
Condition_1_other 2555.17042 10000
Condition_2_other 7230.83188 10000
Bldg_Type_TwnhsE -11776.64761 10000
Bldg_Type_other -15016.92568 10000
House_Style_One_Story 2517.46869 10000
House_Style_Two_Story -1834.73368 10000
House_Style_other -3583.23800 10000
Overall_Cond_Above_Average 387.29301 10000
Overall_Cond_Good 5453.63364 10000
Overall_Cond_other -198.67916 10000
Roof_Style_Hip 9826.95184 10000
Roof_Style_other -6285.85166 10000
Roof_Matl_other 2956.86795 10000
Exterior_1st_MetalSd 2401.59277 10000
Exterior_1st_Plywood -423.31025 10000
Exterior_1st_VinylSd 430.87774 10000
Exterior_1st_Wd.Sdng 364.15513 10000
Exterior_1st_other 9216.05135 10000
Exterior_2nd_MetalSd 2810.59106 10000
Exterior_2nd_Plywood -4782.09685 10000
Exterior_2nd_VinylSd 1979.06774 10000
Exterior_2nd_Wd.Sdng 3586.14793 10000
Exterior_2nd_other 1975.61474 10000
Mas_Vnr_Type_None 7067.96660 10000
Mas_Vnr_Type_Stone 6432.96754 10000
Mas_Vnr_Type_other -12999.85758 10000
Exter_Cond_Typical -1223.58374 10000
Exter_Cond_other -8374.06611 10000
Foundation_CBlock -3752.43518 10000
Foundation_PConc 4777.91201 10000
Foundation_other 12.67181 10000
Bsmt_Cond_other -1096.68254 10000
Bsmt_Exposure_Gd 18454.01931 10000
Bsmt_Exposure_Mn -6292.37350 10000
Bsmt_Exposure_No -9993.27851 10000
Bsmt_Exposure_other -3349.21932 10000
BsmtFin_Type_1_BLQ -979.66229 10000
BsmtFin_Type_1_GLQ 9866.03456 10000
BsmtFin_Type_1_LwQ -4371.15616 10000
BsmtFin_Type_1_Rec -2205.86621 10000
BsmtFin_Type_1_Unf -2137.50633 10000
BsmtFin_Type_1_other -54.65570 10000
BsmtFin_Type_2_other -1995.42781 10000
Heating_other -1333.94062 10000
Heating_QC_Good -4130.03350 10000
Heating_QC_Typical -7280.75514 10000
Heating_QC_other -10421.28767 10000
Central_Air_Y 305.63110 10000
Electrical_SBrkr -843.81162 10000
Electrical_other 145.95150 10000
Functional_other -16837.88549 10000
Garage_Type_BuiltIn 2912.19245 10000
Garage_Type_Detchd -1782.91691 10000
Garage_Type_No_Garage 1405.71932 10000
Garage_Type_other -13424.85604 10000
Garage_Finish_No_Garage 2074.34345 10000
Garage_Finish_RFn -7304.15017 10000
Garage_Finish_Unf -3707.86157 10000
Garage_Cond_Typical 587.45190 10000
Garage_Cond_other -4325.25632 10000
Paved_Drive_Paved 1759.89335 10000
Paved_Drive_other 3091.17128 10000
Pool_QC_other 12835.93973 10000
Fence_No_Fence -1541.73821 10000
Fence_other -1034.11743 10000
Misc_Feature_other 6558.20291 10000
Sale_Type_WD. -6374.52035 10000
Sale_Type_other -7784.24102 10000
Sale_Condition_Normal 6003.00979 10000
Sale_Condition_Partial 13836.66751 10000
Sale_Condition_other 3630.29900 10000

The LASSO

lasso_reg_spec <- linear_reg(mixture = 1, penalty = 1e4) %>%
  set_engine("glmnet")

lasso_reg_rec <- recipe(Sale_Price ~ ., data = ames_train) %>%
  step_impute_knn(all_predictors()) %>%
  step_normalize(all_numeric_predictors()) %>%
  step_other(all_nominal_predictors()) %>%
  step_dummy(all_nominal_predictors())

lasso_reg_wf <- workflow() %>%
  add_model(lasso_reg_spec) %>%
  add_recipe(lasso_reg_rec)

lasso_reg_results <- lasso_reg_wf %>%
  fit_resamples(ames_folds)

lasso_reg_results %>%
  collect_metrics()
.metric .estimator mean n std_err .config
rmse standard 40537.9245852 5 3303.1924456 Preprocessor1_Model1
rsq standard 0.7821723 5 0.0229465 Preprocessor1_Model1

Seeing the Estimated LASSO Model

Again, we wouldn’t generally fit the LASSO model at this time, however you can see how to do that and examine the estimated model below.

lasso_reg_fit <- lasso_reg_wf %>%
  fit(ames_train)

lasso_reg_fit %>%
  tidy()
term estimate penalty
(Intercept) 174537.449 10000
Lot_Frontage 0.000 10000
Lot_Area 0.000 10000
Year_Built 7478.097 10000
Year_Remod_Add 6606.480 10000
Mas_Vnr_Area 3398.018 10000
BsmtFin_SF_1 0.000 10000
BsmtFin_SF_2 0.000 10000
Bsmt_Unf_SF 0.000 10000
Total_Bsmt_SF 12271.407 10000
First_Flr_SF 0.000 10000
Second_Flr_SF 0.000 10000
Gr_Liv_Area 26200.798 10000
Bsmt_Full_Bath 0.000 10000
Bsmt_Half_Bath 0.000 10000
Full_Bath 0.000 10000
Half_Bath 0.000 10000
Bedroom_AbvGr 0.000 10000
Kitchen_AbvGr 0.000 10000
TotRms_AbvGrd 0.000 10000
Fireplaces 3226.727 10000
Garage_Cars 6992.504 10000
Garage_Area 4368.112 10000
Wood_Deck_SF 0.000 10000
Open_Porch_SF 0.000 10000
Enclosed_Porch 0.000 10000
Three_season_porch 0.000 10000
Screen_Porch 0.000 10000
Pool_Area 0.000 10000
Misc_Val 0.000 10000
Mo_Sold 0.000 10000
Year_Sold 0.000 10000
Longitude 0.000 10000
Latitude 0.000 10000
MS_SubClass_One_Story_1945_and_Older 0.000 10000
MS_SubClass_One_and_Half_Story_Finished_All_Ages 0.000 10000
MS_SubClass_Two_Story_1946_and_Newer 0.000 10000
MS_SubClass_One_Story_PUD_1946_and_Newer 0.000 10000
MS_SubClass_other 0.000 10000
MS_Zoning_Residential_Medium_Density 0.000 10000
MS_Zoning_other 0.000 10000
Street_other 0.000 10000
Alley_other 0.000 10000
Lot_Shape_Slightly_Irregular 0.000 10000
Lot_Shape_other 0.000 10000
Land_Contour_other 0.000 10000
Utilities_other 0.000 10000
Lot_Config_CulDSac 0.000 10000
Lot_Config_Inside 0.000 10000
Lot_Config_other 0.000 10000
Land_Slope_other 0.000 10000
Neighborhood_College_Creek 0.000 10000
Neighborhood_Old_Town 0.000 10000
Neighborhood_Edwards 0.000 10000
Neighborhood_Somerset 0.000 10000
Neighborhood_Northridge_Heights 22598.017 10000
Neighborhood_Gilbert 0.000 10000
Neighborhood_other 0.000 10000
Condition_1_Norm 0.000 10000
Condition_1_other 0.000 10000
Condition_2_other 0.000 10000
Bldg_Type_TwnhsE 0.000 10000
Bldg_Type_other 0.000 10000
House_Style_One_Story 0.000 10000
House_Style_Two_Story 0.000 10000
House_Style_other 0.000 10000
Overall_Cond_Above_Average 0.000 10000
Overall_Cond_Good 0.000 10000
Overall_Cond_other 0.000 10000
Roof_Style_Hip 0.000 10000
Roof_Style_other 0.000 10000
Roof_Matl_other 0.000 10000
Exterior_1st_MetalSd 0.000 10000
Exterior_1st_Plywood 0.000 10000
Exterior_1st_VinylSd 0.000 10000
Exterior_1st_Wd.Sdng 0.000 10000
Exterior_1st_other 0.000 10000
Exterior_2nd_MetalSd 0.000 10000
Exterior_2nd_Plywood 0.000 10000
Exterior_2nd_VinylSd 0.000 10000
Exterior_2nd_Wd.Sdng 0.000 10000
Exterior_2nd_other 0.000 10000
Mas_Vnr_Type_None 0.000 10000
Mas_Vnr_Type_Stone 0.000 10000
Mas_Vnr_Type_other 0.000 10000
Exter_Cond_Typical 0.000 10000
Exter_Cond_other 0.000 10000
Foundation_CBlock 0.000 10000
Foundation_PConc 3534.049 10000
Foundation_other 0.000 10000
Bsmt_Cond_other 0.000 10000
Bsmt_Exposure_Gd 10297.300 10000
Bsmt_Exposure_Mn 0.000 10000
Bsmt_Exposure_No 0.000 10000
Bsmt_Exposure_other 0.000 10000
BsmtFin_Type_1_BLQ 0.000 10000
BsmtFin_Type_1_GLQ 7603.684 10000
BsmtFin_Type_1_LwQ 0.000 10000
BsmtFin_Type_1_Rec 0.000 10000
BsmtFin_Type_1_Unf 0.000 10000
BsmtFin_Type_1_other 0.000 10000
BsmtFin_Type_2_other 0.000 10000
Heating_other 0.000 10000
Heating_QC_Good 0.000 10000
Heating_QC_Typical 0.000 10000
Heating_QC_other 0.000 10000
Central_Air_Y 0.000 10000
Electrical_SBrkr 0.000 10000
Electrical_other 0.000 10000
Functional_other 0.000 10000
Garage_Type_BuiltIn 0.000 10000
Garage_Type_Detchd 0.000 10000
Garage_Type_No_Garage 0.000 10000
Garage_Type_other 0.000 10000
Garage_Finish_No_Garage 0.000 10000
Garage_Finish_RFn 0.000 10000
Garage_Finish_Unf 0.000 10000
Garage_Cond_Typical 0.000 10000
Garage_Cond_other 0.000 10000
Paved_Drive_Paved 0.000 10000
Paved_Drive_other 0.000 10000
Pool_QC_other 0.000 10000
Fence_No_Fence 0.000 10000
Fence_other 0.000 10000
Misc_Feature_other 0.000 10000
Sale_Type_WD. 0.000 10000
Sale_Type_other 0.000 10000
Sale_Condition_Normal 0.000 10000
Sale_Condition_Partial 0.000 10000
Sale_Condition_other 0.000 10000

Non-zero LASSO Coefficients

Here are only the predictors with non-zero coefficients

lasso_reg_fit %>%
  tidy() %>%
  filter(estimate != 0)
term estimate penalty
(Intercept) 174537.449 10000
Year_Built 7478.097 10000
Year_Remod_Add 6606.480 10000
Mas_Vnr_Area 3398.018 10000
Total_Bsmt_SF 12271.407 10000
Gr_Liv_Area 26200.798 10000
Fireplaces 3226.727 10000
Garage_Cars 6992.504 10000
Garage_Area 4368.112 10000
Neighborhood_Northridge_Heights 22598.017 10000
Foundation_PConc 3534.049 10000
Bsmt_Exposure_Gd 10297.300 10000
BsmtFin_Type_1_GLQ 7603.684 10000

Summary

  • The more predictors we include in a model, the more flexible that model is

  • We can use regularization methods to constrain our models and make overfitting less likely

  • Two techniques commonly used with linear regression models are Ridge Regression and the LASSO

  • These methods alter the optimization problem that obtains the estimated \(\beta\)-coefficients for our model

  • Ridge Regression attaches very small coefficients to uninformative predictors, while the LASSO attaches coefficients of \(0\) to them

    • This means that the LASSO can be used for variable selection

  • Both Ridge Regression and the LASSO require all numerical predictors to be scaled

  • We can fit/cross-validate these models in nearly the same way that we have been working with ordinary linear regression models

    • We set_engine("glmnet") rather than set_engine("lm") for ridge and LASSO
    • We set mixture = 0 for Ridge Regression and mixture = 1 for the LASSO
    • We define a penalty parameter which determines the amount of regularization (constraint) applied to the model

Next Time…


Other Classes of Regression Model