What are the characteristics of engineering

1. Data preprocessing
   1. Handling missing values
   2. Picture data expansion
   3. Handling outliers
   4. Deal with category imbalance
2. Feature scaling
   1. normalization
   2. Regularization
3. Feature code
   1. Serial number code
   2. Hot coding alone
   3. Binary code
   4. discretization
4. feature selection
   1. Filter type （filter）: First of all, we select the features of the data set , The process has nothing to do with subsequent learners , That is to design some statistics to filter features , Do not consider the follow-up learner problem , Such as variance selection , Chi square test , Mutual information
   2. Parcel type （wrapper）： It's actually a classifier , It is the performance of subsequent learners as the evaluation standard of feature subset . Such as las vagas Algorithm
   3. The embedded （embedding）： In fact, it is the learner's autonomous selection of features . For example, based on the selection of punishment items , Tree based selection GBDT
5. feature extraction
   1. Dimension reduction
   2. Image feature extraction
   3. Text feature extraction
6. Feature building

In case of missing value , What are the ways to deal with it

7. Use features with missing values directly ： When only a small number of samples lack this feature, you can try to use ;
8. Delete features with missing values ： This method is generally applicable to most samples that lack this feature , And containing only a small number of valid values is valid
9. Interpolation to complete missing values
   mean value 、 The number of 、 Median 、 Fixed value 、 Manual 、 Nearest neighbor complement
   Modeling predictions ： Return to 、 Decision tree
   High dimensional mapping , Compress perception
   There are many ways to interpolate

Treatment of sample imbalance

10. Expand the data set
11. Try other evaluation indicators
12. Resampling the dataset

 -  Sample the data samples of the subclass to increase the number of data samples of the subclass , Oversampling （over-sampling,  The number of samples is greater than the number of such samples ）
 -  Sample a large class of data samples to reduce the number of such data samples , Under sampling （under-sampling,  The number of samples is less than the number of such samples ） 

• Try different classification algorithms ： For example, the decision tree often performs well on category unbalanced data
• Try to punish the model ： For example, your classification task is to identify those sub categories , Then you can add weights to the small class sample data of the classifier , Reduce the weight of large classes of samples ,focal loss

appear Nan Why

• Nan The meaning of is meaningless number , There are several situations ：0/0, Inf/Inf, Inf-Inf, Inf*0 etc. , Will lead to uncertain results , So you get NaN
• Data processing , In practical engineering, data is often missing or incomplete , At this point, we can set those missing to nan
• When reading data , A character is not data , Then we think of it as nan Handle

Feature screening , How to find features with high similarity and remove

feature selection — Filtration method ： May adopt Variance selection method or Correlation coefficient method

Contains millions , How to deal with hundreds of millions of characteristic data in deep learning

Many features , Less data , It is easy to cause model over fitting

• Dimension reduction ：PCA or LDA
• Using regularization ,L1 or L2
• Sample expansion
• feature selection ： Remove unimportant features

What are the methods to calculate the correlation between features ？

1. pearson coefficient , Calculate the data of constant distance continuous variables . Is between -1 and 1 Between the value of the
2. spearman Rank correlation coefficient ： It is an indicator to measure the statistical correlation between two variables , Used to evaluate how good the current monotone function is to describe the relationship between two variables
3. kendall The correlation coefficient ： Kendall coefficient is a statistical value used to measure the correlation between two random variables