Breaking News: Grepper is joining You.com. Read the official announcement!

pyspark missing values

Innocent Iguana answered on February 27, 2024 Popularity 5/10 Helpfulness 6/10

answer pyspark missing values

pyspark missing values

Comment

# No of missing values
df.where(df['col_name'].isNull()).count()
# Visualise missing values with heatmap
pandas_df = spark_df.toPandas()
sns.heatmap(data=pandas_df.isnull())
# Drop any records with NULL values
df = df.dropna()
# drop records if both LISTPRICE and SALESCLOSEPRICE are NULL
df = df.dropna(how='all', subset['col1', 'col2 '])
# Drop records where at least two columns have NULL values
df = df.dropna(thresh=2)
# Drop columns with >30% missing values
df = df.drop(*col_list)
# Replace missing values
col_mean = df.agg({'col_name': 'mean'}).collect()[0][0]
df.fillna(col_mean, subset=['col_name'])
# Drop duplicates
df.dropDuplicates(['col_name'])

Popularity 5/10 Helpfulness 6/10 Language python

Source: Grepper

Tags: pyspark python

Link to this answer
Share Copy Link

Contributed on Feb 27 2024

Innocent Iguana

0 Answers Avg Quality 2/10

Continue with Google

By continuing, I agree that I have read and agree to Greppers's Terms of Service and Privacy Policy.

pyspark missing values

Contents

More Related Answers

pyspark missing values

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.