In this project we predict the forest cover type using the cartographic variables in the training/test datasets.

Overview

Kaggle Competition: Forest Cover Type Prediction

In this project we predict the forest cover type (the predominant kind of tree cover) using the cartographic variables given in the training/test datasets. You can find more about this project at Forest Cover Type Prediction.

This project and its detailed notebooks were created and published on Kaggle.

Project Objective

  • We are given raw unscaled data with both numerical and categorical variables.
  • First, we performed Exploratory Data Analysis in order to visualize the characteristics of our given variables.
  • We constructed various models to train our data - utilizing Optuna hyperparameter tuning to get parameters that maximize the model accuracies.
  • Using feature engineering techniques, we built new variables to help improve the accuracy of our models.
  • Using the strategies above, we built our final model and generated forest cover type predictions for the test dataset.

Links to Detailed Notebooks

EDA Summary

The purpose of the EDA is to provide an overview of how python visualization tools can be used to understand the complex and large dataset. EDA is the first step in this workflow where the decision-making process is initiated for the feature selection. Some valuable insights can be obtained by looking at the distribution of the target, relationship to the target and link between the features.

Visualize Numerical Variables

  • Using histograms, we can visualize the spread and values of the 10 numeric variables.
  • The Slope, Vertical Distance to Hydrology, Horizontal Distance to Hydrology, Roadways and Firepoints are all skewed right.
  • Hillshade 9am, Noon, and 3pm are all skewed left. visualize numerical variables histograms

Visualize Categorical Variables

  • The plots below the number of observations of the different Wilderness Areas and Soil Types.
  • Wilderness Areas 3 and 4 have the most presence.
  • Wilderness Area 2 has the least amount of observations.
  • The most observations are seen having Soil Type 10 followed by Soil Type 29.
  • The Soil Types with the least amount of observations are Soil Type 7 and 15. # of observations of wilderness areas # of observations of soil types

Feature Correlation

With the heatmap excluding binary variables this helps us visualize the correlations of the features. We were also able to provide scatterplots for four pairs of features that had a positive correlation greater than 0.5. These are one of the many visualization that helped us understand the characteristics of the features for future feature engineering and model selection.

heatmap scatterplots

Summary of Challenges

EDA Challenges

  • This project consists of a lot of data and can have countless of patterns and details to look at.
  • The training data was not a simple random sample of the entire dataset, but a stratified sample of the seven forest cover type classes which may not represent the final predictions well.
  • Creating a "story" to be easily incorporated into the corresponding notebooks such as Feature Engineering, Models, etc.
  • Manipulating the Wilderness_Area and Soil_Type (one-hot encoded variables) to visualize its distribution compared to Cover_Type.

Feature Engineering Challenges

  • Adding new variables during feature engineering often produced lower accuracy.
  • Automated feature engineering using entities and transformations amongst existing columns from a single dataset created many new columns that did not positively contribute to the model's accuracy - even after feature selection.
  • Testing the new features produced was very time consuming, even with the GPU accelerator.
  • After playing around with several different sets of new features, we found that only including manually created new features yielded the highest results.

Modeling Challenges

  • Ensemble and stacking methods initially resulted in models yielding higher accuracy on the test set, but as we added features and refined the parameters for each individual model, an individual model yielded a better score on the test set.
  • Performing hyperparameter tuning and training for several of the models was computationally expensive. While we were able to enable GPU acceleration for the XGBoost model, activating the GPU accelerator seemed to increase the tuning and training for the other models in the training notebook.
  • Optuna worked to reduce the time to process hyperparameter trials, but some of the hyperparameters identified through this method yielded weaker models than the hyperparameters identified through GridSearchCV. A balance between the two was needed.

Summary of Modeling Techniques

We used several modeling techniques for this project. We began by training simple, standard models and applying the predictions to the test set. This resulted in models with only 50%-60% accuracy, necessitating more complex methods. The following process was used to develop the final model:

  • Scaling the training data to perform PCA and identify the most important features (see the Feature_Engineering Notebook for more detail).
  • Preprocessing the training data to add in new features.
  • Performing GridSearchCV and using the Optuna approach (see the ModelParams Notebook for more detail) for identifying optimal parameters for the following models with corresponding training set accuracy scores:
    • Logistic Regression (.7126)
    • Decision Tree (.9808)
    • Random Forest (1.0)
    • Extra Tree Classifier (1.0)
    • Gradient Boosting Classifier (1.0)
    • Extreme Gradient Boosting Classifier (using GPU acceleration; 1.0)
    • AdaBoost Classifier (.5123)
    • Light Gradient Boosting Classifier (.8923)
    • Ensemble/Voting Classifiers (assorted combinations of the above models; 1.0)
  • Saving and exporting the preprocessor/scaler and each each version of the model with the highest accuracy on the training set and highest cross validation score (see the Training notebook for more detail).
  • Calculating each model's predictions for the test set and submitting to determine accuracy on the test set:
    • Logistic Regression (.6020)
    • Decision Tree (.7102)
    • Random Forest (.7465)
    • Extra Tree Classifier (.7962)
    • Gradient Boosting Classifier (.7905)
    • Extreme Gradient Boosting Classifier (using GPU acceleration; .7803)
    • AdaBoost Classifier (.1583)
    • Light Gradient Boosting Classifier (.6891)
    • Ensemble/Voting Classifier (assorted combinations of the above models; .7952)

Summary of Final Results

The model with the highest accuracy on the out of sample (test set) data was selected as our final model. It should be noted that the model with the highest accuracy according to 10-fold cross validation was not the most accurate model on the out of sample data (although it was close). The best model was the Extra Tree Classifier with an accuracy of .7962 on the test set. The Extra Trees model outperformed our Ensemble model (.7952), which had been our best model for several weeks. See the Submission Notebook and FinalModelEvaluation Notebook for additional detail.

Owner
Marianne Joy Leano
A recent graduate with a Master's in Data Science. Excited to explore data and create projects!
Marianne Joy Leano
MemStream: Memory-Based Anomaly Detection in Multi-Aspect Streams with Concept Drift

MemStream Implementation of MemStream: Memory-Based Anomaly Detection in Multi-Aspect Streams with Concept Drift . Siddharth Bhatia, Arjit Jain, Shivi

Stream-AD 61 Dec 02, 2022
A super lightweight Lagrangian model for calculating millions of trajectories using ERA5 data

Easy-ERA5-Trck Easy-ERA5-Trck Galleries Install Usage Repository Structure Module Files Version iteration Easy-ERA5-Trck is a super lightweight Lagran

Zhenning Li 26 Nov 19, 2022
A Pytorch implementation of MoveNet from Google. Include training code and pre-train model.

Movenet.Pytorch Intro MoveNet is an ultra fast and accurate model that detects 17 keypoints of a body. This is A Pytorch implementation of MoveNet fro

Mr.Fire 241 Dec 26, 2022
An Active Automata Learning Library Written in Python

AALpy An Active Automata Learning Library AALpy is a light-weight active automata learning library written in pure Python. You can start learning auto

TU Graz - SAL Dependable Embedded Systems Lab (DES Lab) 78 Dec 30, 2022
Lazy, a tool for running things in idle time

Lazy, a tool for running things in idle time Mostly used to stop CUDA ML model training from making my desktop unusable. Simply monitors keyboard/mous

N Shepperd 46 Nov 06, 2022
Hummingbird compiles trained ML models into tensor computation for faster inference.

Hummingbird Introduction Hummingbird is a library for compiling trained traditional ML models into tensor computations. Hummingbird allows users to se

Microsoft 3.1k Dec 30, 2022
Offline Reinforcement Learning with Implicit Q-Learning

Offline Reinforcement Learning with Implicit Q-Learning This repository contains the official implementation of Offline Reinforcement Learning with Im

Ilya Kostrikov 125 Dec 31, 2022
Unofficial PyTorch code for BasicVSR

Dependencies and Installation The code is based on BasicSR, Please install the BasicSR framework first. Pytorch=1.51 Training cd ./code CUDA_VISIBLE_

Long 59 Dec 06, 2022
Direct design of biquad filter cascades with deep learning by sampling random polynomials.

IIRNet Direct design of biquad filter cascades with deep learning by sampling random polynomials. Usage git clone https://github.com/csteinmetz1/IIRNe

Christian J. Steinmetz 55 Nov 02, 2022
Deep Learning for Computer Vision final project

Deep Learning for Computer Vision final project

grassking100 1 Nov 30, 2021
Character Controllers using Motion VAEs

Character Controllers using Motion VAEs This repo is the codebase for the SIGGRAPH 2020 paper with the title above. Please find the paper and demo at

Electronic Arts 165 Jan 03, 2023
这是一个利用facenet和retinaface实现人脸识别的库,可以进行在线的人脸识别。

Facenet+Retinaface:人脸识别模型在Pytorch当中的实现 目录 注意事项 Attention 所需环境 Environment 文件下载 Download 预测步骤 How2predict 参考资料 Reference 注意事项 该库中包含了两个网络,分别是retinaface和

Bubbliiiing 102 Dec 30, 2022
This repository implements variational graph auto encoder by Thomas Kipf.

Variational Graph Auto-encoder in Pytorch This repository implements variational graph auto-encoder by Thomas Kipf. For details of the model, refer to

DaehanKim 215 Jan 02, 2023
A simple rest api that classifies pneumonia infection weather it is Normal, Pneumonia Virus or Pneumonia Bacteria from a chest-x-ray image.

This is a simple rest api that classifies pneumonia infection weather it is Normal, Pneumonia Virus or Pneumonia Bacteria from a chest-x-ray image.

crispengari 3 Jan 08, 2022
Implementation of Artificial Neural Network Algorithm

Artificial Neural Network This repository contain implementation of Artificial Neural Network Algorithm in several programming languanges and framewor

Resha Dwika Hefni Al-Fahsi 1 Sep 14, 2022
Keyword2Text This repository contains the code of the paper: "A Plug-and-Play Method for Controlled Text Generation"

Keyword2Text This repository contains the code of the paper: "A Plug-and-Play Method for Controlled Text Generation", if you find this useful and use

57 Dec 27, 2022
TCNN Temporal convolutional neural network for real-time speech enhancement in the time domain

TCNN Pandey A, Wang D L. TCNN: Temporal convolutional neural network for real-time speech enhancement in the time domain[C]//ICASSP 2019-2019 IEEE Int

凌逆战 16 Dec 30, 2022
Steer OpenAI's Jukebox with Music Taggers

TagBox Steer OpenAI's Jukebox with Music Taggers! The closest thing we have to VQGAN+CLIP for music! Unsupervised Source Separation By Steering Pretra

Ethan Manilow 34 Nov 02, 2022
Flow is a computational framework for deep RL and control experiments for traffic microsimulation.

Flow Flow is a computational framework for deep RL and control experiments for traffic microsimulation. See our website for more information on the ap

867 Jan 02, 2023
This repository contains the code and models for the following paper.

DC-ShadowNet Introduction This is an implementation of the following paper DC-ShadowNet: Single-Image Hard and Soft Shadow Removal Using Unsupervised

AuAgCu 65 Dec 27, 2022