Breaking News: Grepper is joining You.com. Read the official announcement!

feature reduction python

Innocent Iguana answered on January 10, 2024 Popularity 3/10 Helpfulness 7/10

answer feature reduction python

feature reduction python

Comment

# Remove feature with 0 variance
df.describe()
# Remove features with very high or very low cardinality
df['col'].value_counts()
# Remove irrelevant column 
df.drop('col', axis=1)
# Inspect features with seaborn pairplot to remove duplicate columns
sns.pairplot(df, hue="cat_col", diag_kind='hist')
# USE T-SNE
# DO RFE
# DO PCA

# Drop columns that have missing values beyond a threshold (generally 30%)
mask = pokemon_df.isna().sum() / len(pokemon_df) < 0.3
reduced_df = df.loc[:, mask]

# Generate new features (eg: average)
df["avg_col"]=( df["col1"] + df["col2"])/2

# Drop highly correlated features if you are confident that they may add bias into the model (Use heatmap on dataframe)
# Drop scaled variance up to a threshold (features with very low variance or close to 0)
from sklearn.feature_selection import VarianceThreshold
sel = VarianceThreshold(threshold=0.005)
sel.fit(df / df.mean())
mask = sel.get_support()
reduced_df = df.loc[:, mask]

# Tree based models to filter out features (You can use RFE with it)
rf = RandomForestClassifier()
rf.fit(X_train, y_train)
mask = rf.feature_importances_ > 0.1
# See surviving features
X_reduced = X.loc[:, mask]
print(X_reduced.columns)

Popularity 3/10 Helpfulness 7/10 Language python

Source: Grepper

Tags: python reduction

Link to this answer
Share Copy Link

Contributed on Jan 10 2024

Innocent Iguana

0 Answers Avg Quality 2/10

feature reduction python

Contents

More Related Answers

feature reduction python

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.