数据科学导论——数据预处理

第1关：引言-根深之树不怯风折，泉深之水不会涸竭

第2关：数据清理-查漏补缺

本关任务：模型的基线取决于数据的好坏，本关将学习数据处理的常用处理

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
def student():
    train = pd.read_csv('Task1/diabetes_null.csv', na_values=['#NAME?'])
    train['Insulin'] = train['Insulin'].fillna(100)
    train['SkinThickness'] = train['SkinThickness'].fillna(train['SkinThickness'].median())
    train['BloodPressure'] = train['BloodPressure'].fillna(train['BloodPressure'].median())
    train['BMI'] = train['BMI'].fillna(train['BMI'].mean())
    train['Glucose'] = train['Glucose'].fillna(train['Glucose'].mean())
    #********* Begin *********#
    train.sort_values(by='Age', ascending=False)[:1]
    train = train.drop((train[train['Age'] >= 80]).index)
    plt.figure(figsize=(10, 10))
    plt.scatter(x=train['Age'], y=train['Pregnancies'])
    plt.savefig("Task1/img/T1.png")
    plt.show()
    #********* End *********# 

第3关：数据集成-海纳百川

本关任务：编写一个能够将两个数据集合并的程序。

import numpy as np
import pandas as pd

def student():
    #********* Begin *********#
    train = pd.read_csv('Task2/diabetes_null.csv', na_values=['#NAME?'])
    another_train = pd.read_csv('Task2/diabetes_zero.csv', na_values=['#NAME?'])
    merge_data=pd.concat([train,another_train])
    print(merge_data.shape)


    #********* End *********# 

第4关：数据变换-同源共流

本关任务：学习掌握常见的数据变换，完成给定数据的规范化的输出。

import numpy as np
import pandas as pd
from sklearn.preprocessing import normalize,MinMaxScaler

def student():
    train = pd.read_csv('Task3/diabetes_null.csv', na_values=['#NAME?'])
    train['Insulin'] = train['Insulin'].fillna(100)
    train['SkinThickness'] = train['SkinThickness'].fillna(train['SkinThickness'].median())
    train['BloodPressure'] = train['BloodPressure'].fillna(train['BloodPressure'].median())
    train['BMI'] = train['BMI'].fillna(train['BMI'].mean())
    train['Glucose'] = train['Glucose'].fillna(train['Glucose'].mean())
    #********* Begin *********#
    data_normalized=normalize(train,axis=0)
    print("z-score规范化：\n",data_normalized)
    data_scaler=MinMaxScaler()
    data_scaled=data_scaler.fit_transform(train)
    print("\n最小－最大规范化：\n",data_scaled)
    #********* End *********# 

数据科学导论——数据预处理进阶

第1关：数据归约

本关任务：使用直方图展示不同年龄的发病次数。

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
def student():
    train = pd.read_csv('Task1/diabetes_null.csv', na_values=['#NAME?'])
    train['Insulin'] = train['Insulin'].fillna(100)
    train['SkinThickness'] = train['SkinThickness'].fillna(train['SkinThickness'].median())
    train['BloodPressure'] = train['BloodPressure'].fillna(train['BloodPressure'].median())
    train['BMI'] = train['BMI'].fillna(train['BMI'].mean())
    train['Glucose'] = train['Glucose'].fillna(train['Glucose'].mean())
    #********* Begin *********#
    plt.figure(figsize=(10,10))
    x=pd.Series(train['Age'])
    count=x.value_counts()
    count.plot(kind='bar')
    plt.savefig("Task1/img/T1.png")
    plt.show()
    #********* End *********# 

第2关：数据离散化

本关任务：对 Age 进行分区。

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
def student():
    train = pd.read_csv('Task1/diabetes_null.csv', na_values=['#NAME?']).dropna()
    #********* Begin *********#
    bin=[0,30,50,90]
    a=pd.cut(train["Age"],bin)
    print(pd.value_counts(a))
    #********* End *********#