Feature Engineering - feature preprocessing (normalization, standardization)

Why do we normalize / Standardization ？

Unification of feature weights
Dimensionless

The units or sizes of features vary greatly , Or the variance of a feature is several orders of magnitude larger than that of other features , Easy to influence （ control ） Target result , Some algorithms cannot learn other features

We need to use some methods to dimensionless , Convert data of different specifications to the same specification
Include content ( Dimensionless of numerical data )
normalization
Standardization

Feature preprocessing API
sklearn.preprocessing

sklearn.preprocessing.MinMaxScaler (feature_range=(0,1)… )
MinMaxScalar.fit_transform(X)
X:numpy array Formatted data [n_samples,n_features]
Return value ： The transformed shape is the same array

Normalization demonstration

import pandas as pd
from sklearn.preprocessing import MinMaxScaler

def minmax_demo():
    """  Normalization demonstration  :return: None """
    data = pd.read_csv("./dating.txt")
    print(data)
    # 1、 Instantiate a converter class 
    transfer = MinMaxScaler(feature_range=(2, 3))
    # 2、 call fit_transform
    data = transfer.fit_transform(data[['milage','Liters','Consumtime']])
    print(" The result of normalization of minimum and maximum values ：\n", data)

    return None


if __name__ == '__main__':
    minmax_demo()

Robustness is poor

Standardization ：
Particularly vulnerable to maximum , The minimum value affects , That is, the influence of special points and outliers , such as , The age is counted into hundreds
（ Robustness ）
So we need to standardize

API
sklearn.preprocessing.StandardScaler( )
After processing, all data in each column is clustered in the mean value 0 The standard deviation is 1
StandardScaler.fit_transform(X)
X:numpy array Formatted data [n_samples,n_features]
Return value ： The transformed shape is the same array

Standardized demonstration ：

import pandas as pd
from sklearn.preprocessing import StandardScaler

def stand_demo():
    """  Standardized demonstration  :return: None """
    data = pd.read_csv("dating.txt")
    print(data)
    # 1、 Instantiate a converter class 
    transfer = StandardScaler()
    # 2、 call fit_transform
    data = transfer.fit_transform(data[['milage','Liters','Consumtime']])
    print(" The result of Standardization :\n", data)
    print(" The average value of each column of features ：\n", transfer.mean_)
    print(" The variance of each column characteristic ：\n", transfer.var_)

    return None




if __name__ == '__main__':
    # minmax_demo()
    stand_demo()

Standardization is generally used , big data , Noisy scene