Feature Engineering of Turku Weather Data

Code Structure

class TurkuWeatherAnalysis:
    - load_and_preprocess(filepath)  # Loads and preprocesses the dataset
    - visualize_data()               # Generates weather-related plots
    - generate_statistics()          # Computes annual statistics
    - check_data_period()            # Displays the available time period

Data Period

• Start Date: 2021-01-01 00:00:00
• End Date: 2024-09-23 16:00:00

Annual Weather Statistics

Visualizations

The following image presents key insights from the weather dataset, including temperature trends, precipitation distribution, humidity relationships, and wind speed analysis.

Description of Visualizations:

1. Monthly Temperature Trends (Top Left): Shows how average temperature varies across different months from 2021 to 2024. Warmer months are clearly visible in June-August, while the coldest periods occur in December-February.
2. Precipitation Distribution (Top Right): Highlights the frequency and distribution of precipitation values. The majority of precipitation is below 2mm, with some extreme values up to 12mm.
3. Temperature vs Humidity (Bottom Left): A scatter plot that shows how temperature correlates with humidity. Higher temperatures tend to have lower humidity, while lower temperatures are associated with higher humidity.
4. Wind Speed Distribution (Bottom Right): A box plot summarizing the distribution of wind speeds. The median wind speed is around 3 m/s, with some extreme values reaching up to 10 m/s.

Point Session Data Analysis

Data Period

• Start Date: 2019-01-23 14:00:31.857000
• End Date: 2024-06-28 10:19:29.179000

Basic Statistics

• Total records: 2,038,060
• Unique sessions: 581,137
• Unique points: 74

Daily Session Statistics

• Average sessions per day: 440
• Maximum sessions in one day: 1,127
• Minimum sessions in one day: 1

Property Type Distribution

Top 10 Most Active Points

Top 10 Busiest Days

Common Sequential Patterns

Visualizations

The following image presents key insights from the session dataset, including session distribution, property type occurrences, and busiest points.

Description of Visualizations:

1. Session Distribution Over Time: Shows the total number of sessions per day across the dataset timeframe, highlighting peak usage periods.
2. Property Type Occurrences: Displays the frequency of different property types such as dishWeight, menuDeduction, and wasteWeight to understand session interactions.
3. Busiest Points Activity: Identifies the most active points where session interactions take place, helping to analyze high-traffic areas.
4. Sequential Patterns: Reveals the most common sequences in session interactions, showing how different activities relate to each other.

Overfitting Analysis and Model Performance Comparison

Overview

This document provides an analysis of overfitting issues in different XGBoost models (M1, M2, M3, M4, M5) and offers recommendations for improvement. The key focus is identifying models that generalize well to unseen data and addressing those that suffer from overfitting.

Training and Evaluation Process

Training Process

1. Data Preprocessing:

   • Convert datetime columns to numeric values.
   • Ensure all features are numerical and handle missing values.
   • Standardize features using StandardScaler to normalize input values.

2. Model Selection and Tuning:

   • Hyperparameter tuning performed using Optuna, optimizing for RMSE.
   • Models trained using XGBoost Regressor with different hyperparameter settings.
   • Training conducted using GPU acceleration (tree_method='hist').

3. Training Steps:

   • Train-test split with 80% training data and 20% test data.
   • Model iterates through 500 optimization trials for hyperparameter tuning.
   • Final model is selected based on the best RMSE score.
   • Models are evaluated using cross-validation and learning curves.

4. K-Fold Cross-Validation:

   • 5-fold cross-validation (cv=5) is applied to assess model generalization.
   • The dataset is split into 5 parts, with each part serving as the validation set once.
   • The final performance score is averaged across all 5 iterations.
   • Helps ensure the model is robust and does not overfit to a single train-test split.

5. Evaluation Metrics:

   • Mean Absolute Error (MAE) to measure absolute prediction error.
   • Root Mean Squared Error (RMSE) to penalize large errors.
   • Training vs. Test Performance Comparison to detect overfitting.

6. Learning Curve Analysis:

   • Training and validation scores analyzed over increasing dataset sizes.
   • Helps identify if models are underfitting or overfitting.

7. Learning Curves Interpretation:

   • M1 & M2: Show significant overfitting as training error remains very low while validation error is high.
   • M3: Best balance between training and validation errors, indicating good generalization.
   • M4 & M5: Some degree of overfitting, but better than M1 & M2.
   • Visualization:
     • Blue line: Training score (lower is better)
     • Red line: Cross-validation score (lower is better)
     • Shaded region: Variance in cross-validation error.

8. Learning Curve Visualizations:

   • Below are the learning curves for each model:
     • M1:
     • M2:
     • M3:
     • M4:
     • M5:

Model Performance Metrics

Ranking Based on Test MAE

Overfitting Indicators

Key Takeaways

• M2 is the most overfit model, with a MAE gap of 66.70 and RMSE gap of 103.00.
• M5 also exhibits significant overfitting, with large discrepancies between training and test performance.
• M3 is the best model, balancing test accuracy with minimal overfitting.
• M4 shows relatively stable performance and may also be a good option.

Recommendations to Address Overfitting

1️⃣ Increase Regularization

Adding L1 (Lasso) or L2 (Ridge) regularization can penalize large weights and reduce overfitting:

'reg_alpha': trial.suggest_float('reg_alpha', 0.0, 10.0),  # L1 regularization
'reg_lambda': trial.suggest_float('reg_lambda', 0.0, 10.0)  # L2 regularization

2️⃣ Reduce Model Complexity

• Lower max_depth to 3–6 instead of 9–10.
• Reduce n_estimators to 100–150 instead of 200–300.
• Increase min_child_weight to 5–7 to make splits more selective.

3️⃣ Use More Training Data

• Expand the dataset or apply synthetic data augmentation.
• Implement k-fold cross-validation with cv=5 to improve generalization.

4️⃣ Adjust Learning Rate

• Use a lower learning rate (0.01–0.1) with a higher number of trees (n_estimators 150–200) to smooth decision boundaries.

5️⃣ Feature Selection & Engineering

• Remove highly correlated or redundant features.
• Use SHAP values or feature importance scores to identify and remove unnecessary features.

Final Recommendation

• Use M3 as the best model since it has the lowest test MAE and RMSE with minimal overfitting.
• Apply regularization and hyperparameter tuning to further optimize M3.
• If stability is a priority, M4 is also a viable option.

Next Steps

• Implement suggested changes and re-evaluate models.
• Generate new hyperparameter settings to further reduce overfitting.
• Validate improved models with additional test data.

By following these recommendations, we can improve the model's generalization while maintaining strong predictive accuracy.

📌 End of Document