senior-data-scientist

What this skill does

# Senior Data Scientist

## Overview

Build end-to-end data science workflows from data exploration through model deployment. This skill covers data preprocessing, feature engineering, model selection, hyperparameter tuning, cross-validation, experiment tracking with MLflow/W&B, statistical testing, visualization with matplotlib/seaborn/plotly, and Jupyter notebook best practices.

**Announce at start:** "I'm using the senior-data-scientist skill for data science workflow."

---

## Phase 1: Data Understanding

**Goal:** Profile the dataset and establish a baseline before any modeling.

### Actions

1. Load and profile the dataset (shape, types, distributions)
2. Identify missing values, outliers, and data quality issues
3. Perform exploratory data analysis (EDA)
4. Define the target variable and success metrics
5. Establish baseline performance

### Baseline Models (Always Start Here)

| Task | Baseline Model | Why |
|------|---------------|-----|
| Classification | Majority class classifier | Lower bound for accuracy |
| Classification | Logistic regression | Simple, interpretable |
| Regression | Mean predictor | Lower bound for RMSE |
| Regression | Linear regression | Simple, interpretable |
| Time series | Naive forecast (previous value) | Lower bound for MAE |
| Time series | Seasonal naive | Captures basic seasonality |

### STOP — Do NOT proceed to Phase 2 until:
- [ ] Dataset is profiled (shape, types, distributions)
- [ ] Missing values and outliers are documented
- [ ] Target variable is defined
- [ ] Success metrics are chosen
- [ ] Baseline performance is established

---

## Phase 2: Feature Engineering

**Goal:** Transform raw data into features that improve model performance.

### Actions

1. Handle missing values (imputation strategy)
2. Encode categorical variables
3. Scale/normalize numerical features
4. Create derived features
5. Feature selection (remove redundant/irrelevant)

### Missing Value Strategy Decision Table

| Strategy | When to Use | Implementation |
|----------|-------------|---------------|
| Drop rows | < 5% missing, MCAR | `df.dropna()` |
| Mean/Median | Numerical, no outliers | `SimpleImputer(strategy='median')` |
| Mode | Categorical | `SimpleImputer(strategy='most_frequent')` |
| KNN Imputer | Structured missing patterns | `KNNImputer(n_neighbors=5)` |
| Iterative | Complex relationships | `IterativeImputer()` |
| Flag + Impute | Missingness is informative | Add `is_missing` column + impute |

### Categorical Encoding Decision Table

| Method | When | Cardinality |
|--------|------|-------------|
| One-Hot | Nominal, low cardinality | < 10 categories |
| Label/Ordinal | Ordinal features | Any |
| Target Encoding | High cardinality nominal | > 10 categories |
| Frequency Encoding | When frequency matters | Any |
| Binary Encoding | Very high cardinality | > 50 categories |

### Scaling Decision Table

| Scaler | When | Robust to Outliers? |
|--------|------|-------------------|
| StandardScaler | Default choice (mean=0, std=1) | No |
| RobustScaler | Outliers present (median/IQR) | Yes |
| MinMaxScaler | Neural networks, distance-based [0,1] | No |

### Feature Types and Engineering

| Feature Type | Techniques |
|-------------|-----------|
| Numerical | Log transform, polynomial, binning, interactions (A*B, A/B) |
| Temporal | Hour, day-of-week, is_weekend, time_since_event, cyclical (sin/cos), lags |
| Text | TF-IDF, word count, sentiment scores, named entities, embeddings |
| Categorical | Encoding (above), interaction with numerical features |

### Feature Selection Decision Table

| Method | Type | Use When |
|--------|------|----------|
| Correlation matrix | Filter | Initial exploration |
| Mutual information | Filter | Non-linear relationships |
| Recursive Feature Elimination | Wrapper | Model-specific selection |
| L1 Regularization | Embedded | Linear models |
| Feature importance | Embedded | Tree-based models |
| Permutation importance | Model-agnostic | Final validation |

### STOP — Do NOT proceed to Phase 3 until:
- [ ] Missing values are handled with justified strategy
- [ ] Categorical variables are encoded appropriately
- [ ] Numerical features are scaled
- [ ] Feature engineering is done BEFORE train/test split on training data only
- [ ] Feature selection has reduced dimensionality if needed

---

## Phase 3: Modeling

**Goal:** Select, train, and evaluate candidate models.

### Actions

1. Select candidate algorithms
2. Set up cross-validation strategy
3. Train and evaluate candidates
4. Hyperparameter tuning
5. Final model selection and evaluation

### Algorithm Decision Table

| Data Characteristics | Try First | Also Consider |
|---------------------|-----------|---------------|
| Tabular, < 10K rows | Random Forest, XGBoost | Logistic/Linear Regression |
| Tabular, > 10K rows | XGBoost, LightGBM | CatBoost, Neural Network |
| High dimensionality | Lasso/Ridge, SVM | Random Forest with selection |
| Time series | Prophet, ARIMA | LSTM, XGBoost with lag features |
| Text classification | Fine-tuned transformer | TF-IDF + Logistic Regression |
| Image classification | Pre-trained CNN (ResNet, EfficientNet) | Vision Transformer |
| Regression | XGBoost, Random Forest | Linear Regression, Neural Network |
| Anomaly detection | Isolation Forest | LOF, Autoencoder |

### Cross-Validation Strategy Decision Table

| Strategy | When | Code |
|----------|------|------|
| K-Fold (k=5) | Default, balanced data | `KFold(n_splits=5)` |
| Stratified K-Fold | Classification, imbalanced | `StratifiedKFold(n_splits=5)` |
| Time Series Split | Temporal data | `TimeSeriesSplit(n_splits=5)` |
| Group K-Fold | Grouped observations | `GroupKFold(n_splits=5)` |
| Leave-One-Out | Very small datasets | `LeaveOneOut()` |

### Evaluation Metrics Decision Table

| Task | Primary Metric | Secondary Metrics |
|------|---------------|-------------------|
| Binary Classification | AUC-ROC | F1, Precision, Recall, AP |
| Multiclass | Macro F1 | Accuracy, Confusion Matrix |
| Regression | RMSE | MAE, R-squared, MAPE |
| Ranking | NDCG | MAP, MRR |
| Anomaly Detection | F1, AP | Precision@K, Recall@K |

### Hyperparameter Tuning Decision Table

| Method | Compute Budget | Search Space | Implementation |
|--------|---------------|-------------|----------------|
| Grid Search | Low (< 100 combos) | Small, known ranges | `GridSearchCV` |
| Random Search | Medium | Large, uncertain | `RandomizedSearchCV` |
| Bayesian (Optuna) | Any | Large, expensive | `optuna.create_study()` |
| Successive Halving | Large | Many candidates | `HalvingRandomSearchCV` |

### Common Hyperparameters (XGBoost/LightGBM)

```python
param_space = {
    'n_estimators': [100, 300, 500, 1000],
    'max_depth': [3, 5, 7, 9],
    'learning_rate': [0.01, 0.05, 0.1],
    'subsample': [0.7, 0.8, 0.9],
    'colsample_bytree': [0.7, 0.8, 0.9],
    'min_child_weight': [1, 3, 5],
}
```

### STOP — Do NOT proceed to Phase 4 until:
- [ ] At least 2 candidate models are evaluated
- [ ] Cross-validation is used (not just train/test split)
- [ ] Results beat the baseline from Phase 1
- [ ] Best model is selected with justification
- [ ] Overfitting is checked (train vs validation gap)

---

## Phase 4: Deployment

**Goal:** Serialize, serve, and monitor the model in production.

### Actions

1. Serialize model and preprocessing pipeline
2. Create prediction API or batch pipeline
3. Set up monitoring for data drift and model degradation
4. Document model card (inputs, outputs, limitations, biases)

### STOP — Deployment complete when:
- [ ] Model is serialized with preprocessing pipeline
- [ ] Prediction API or batch pipeline works end-to-end
- [ ] Monitoring is configured for data drift
- [ ] Model card is documented

---

## Experiment Tracking

### MLflow Pattern

```python
import mlflow

mlflow.set_experiment("customer-churn-prediction")

with mlflow.start_run(run_name="xgboost-v2"):
    mlflow.log_params(params)
    mlflow.log_metrics({"auc": auc_score, "f1":