# How to use this file AI for Data and Documents — TOPIC 2 Modeling and Code You Can Trust Two models for Y1: (A) Linear Regression (B) Regression Tree (CART) GOAL Build two predictive models for the numeric outcome Y1 using the predictors X1-X20, judge how good they are with RMSE, and compare that RMSE to the standard deviation of Y1. A model is only useful if its prediction error (RMSE) is clearly SMALLER than simply guessing the average of Y1 (whose error is about the standard deviation of Y1). HOW TO USE THIS FILE 1. From the "GOV Datasets" page, download one dataset (GOV1.csv ... GOV23.csv) and its description. 2. Open a new chat at claude.ai and UPLOAD the CSV. 3. Run the prompts below IN ORDER. Part A is regression; Part B is the tree. 4. After each model, fill in the RESULTS LOG at the bottom. DATA STRUCTURE REMINDER - Identifiers (never predictors): country, iso3, year - Outcome to predict: Y1 (numeric). Ignore Y2 in this Topic. - Predictors: X1 ... X20 - X20 = ht_region = world region, a CATEGORICAL variable with 10 categories. It must be treated as categorical (one-hot / dummy encoded), NOT as a number. (Confirm in the description file; in a few datasets region may be X19.) THE GOLDEN RULE OF THIS TOPIC After every step, add: "Show me the exact code you ran." You are learning to produce an analysis you can defend, not a black box. --- # Set up the problem (run this first) I have uploaded a CSV. We are going to predict the numeric outcome Y1 from the predictors X1 through X20. Please set up the problem exactly as follows and show all code: 1. Load the data. Confirm the columns: country, iso3, year, Y1, Y2, X1..X20. 2. DROP the columns country, iso3, and Y2. Keep year for the split (next step), but do NOT use year, country, or iso3 as predictors in the model. 3. Treat X20 (ht_region) as CATEGORICAL. Confirm how many categories it has. All other predictors X1..X19 are numeric. 4. Apply complete-case handling: for the modeling dataset, keep only rows that have no missing values in Y1 or in any predictor X1..X20. Report how many rows you started with and how many remain. Do not build any model yet. Just report the setup and wait. --- # Train / test split by time Split the data into a TRAINING set and a TEST set using the variable "year": - TRAINING = all rows with year <= the 80th percentile of year - TEST = the remaining later years (This is a time-based split: we train on earlier years and test on later years, which is the honest way to check prediction.) Report: - the exact year cutoff you used - the number of training rows and test rows - the first and last year in each set Then compute and report the STANDARD DEVIATION of Y1 IN THE TRAINING SET. Call this number SD_train. We will compare every model's error to it. Show the code. --- # Fit the linear regression On the TRAINING set only, fit a linear (ordinary least squares) regression that predicts Y1 from all predictors X1 through X20, with X20 one-hot encoded as a set of region dummy variables (drop one category as the reference). Report: - the regression coefficient for each predictor, with its p-value - the R-squared on the training set - a short, plain-English reading of the 3 predictors with the largest absolute standardized effect (and whether each pushes Y1 up or down) Remind me that these associations are not proof of causation. Show the code. --- # Evaluate with RMSE and compare to SD Now judge the regression honestly: 1. Use the fitted model to predict Y1 on the TRAINING set. Compute the training RMSE (root mean squared error). 2. Use the SAME fitted model to predict Y1 on the TEST set (the later years). Compute the test RMSE. 3. Show a small comparison table with three numbers: SD_train (std. dev. of Y1 in training) ...... [from step 2A.2] Training RMSE ................................ Test RMSE .................................... 4. Compute the ratio Test RMSE / SD_train and report it. Interpret the result for me in plain English: - If RMSE is much smaller than SD_train (ratio well below 1), the model predicts Y1 better than just guessing the training mean -- it has learned something useful. - If RMSE is close to or above SD_train (ratio near or above 1), the model is NOT beating the naive "always predict the average" baseline. - Comment on whether the test RMSE is much larger than the training RMSE, which would signal overfitting. Show all code. --- # A picture of the fit On the TEST set, make a scatterplot of predicted Y1 (x-axis) versus actual Y1 (y-axis), and draw the 45-degree line where prediction = actual. Describe in two sentences how close the points sit to that line. Show the code. --- # Part B — Regression Tree (CART): setup Use the SAME training and test sets, the SAME complete-case rows, and the SAME SD_train from Part A, so the two models are compared fairly. --- # Fit a regression tree (CART) On the TRAINING set only, fit a regression tree (CART) that predicts Y1 from the predictors X1 through X20. Treat X20 (ht_region) as categorical. To avoid an overgrown tree, start with these settings and state them explicitly: - maximum depth = 4 - minimum samples per leaf = 20 (or the nearest sensible value for this data) Report what you did and show the code. Do not evaluate yet. --- # Show and read the tree Draw the fitted regression tree as a clear diagram (boxes with the split rule, the number of samples, and the predicted Y1 at each leaf). Then: - Translate the top two or three splits into plain English (e.g. "the first split is on X7: if X7 is below ___, predicted Y1 is ___"). - List the predictor importances (which variables the tree actually used most). Show the code. --- # Evaluate the tree with RMSE and compare Judge the tree the same way we judged the regression: 1. Training RMSE of the tree on the TRAINING set. 2. Test RMSE of the tree on the TEST set (later years). 3. Comparison table: SD_train (std. dev. of Y1 in training) ....... [same as before] Tree Training RMSE ........................... Tree Test RMSE ............................... 4. Ratio Tree Test RMSE / SD_train. Interpret: does the tree beat the naive "predict the average" baseline (ratio below 1)? Is there a big gap between training and test RMSE (overfitting)? Show all code. --- # Does depth help? (a small, honest experiment) Refit the regression tree at maximum depths 2, 4, 6, and 8 (keeping everything else the same). For each depth, report the training RMSE and the test RMSE in one table, and plot both against depth. Tell me which depth gives the LOWEST TEST RMSE, and point out where training RMSE keeps falling while test RMSE starts rising -- that gap is overfitting. Show the code. --- # Compare the two models Put the regression and the best regression tree side by side in ONE table: Model | Test RMSE | Test RMSE / SD_train Linear regression | ... | ... Regression tree (CART)| ... | ... Baseline (predict mean)| SD_train | 1.00 State clearly which model predicts Y1 better on the later (test) years, and by how much relative to the naive baseline. If neither model clearly beats the baseline, say so honestly. Show the code. --- # A defensible paragraph Write one honest paragraph I could put in a report: what outcome we modeled, how we split the data, the test RMSE of each model versus the standard deviation of Y1, which model we would choose and why, and the main limitation (including that these are associations, not causal effects, and that results are specific to this dataset and time split). --- # RESULTS LOG (fill this in as you go) Dataset used (GOV #): ______ Y1 is: __________________________ Year cutoff for split: ______ Train rows: ______ Test rows: ______ SD_train (std dev of Y1, training): ____________ Linear regression: Train RMSE ______ Test RMSE ______ Test/SD ______ Regression tree: Train RMSE ______ Test RMSE ______ Test/SD ______ Best tree depth: ______ Which model wins on test data? ________________________________ Does it beat the "predict the average" baseline? YES / NO --- # VERIFICATION HABITS (Topic 2) - Always confirm the SAME rows, split, and SD_train are used for both models. - Make sure X20 (region) is treated as categorical, not as a number. - Be suspicious of a training RMSE far below the test RMSE: that is overfitting. - A model that cannot beat RMSE = SD_train has learned nothing useful. - Ask Claude to show the code every time, and re-run one number to confirm it.