Breaking News: Grepper is joining You.com. Read the official announcement!

how to split data into training and testing in pyspark

KK SK answered on October 14, 2022 Popularity 7/10 Helpfulness 3/10

answer how to split data into training and testing in pyspark

related pyspark train test split

how to split data into training and testing in pyspark

Comment

# Split the data into training and testing

splits = df4.randomSplit([1.0, 2.0], seed=24)

Popularity 7/10 Helpfulness 3/10 Language whatever

Source: spark.apache.org

Tags: pyspark testing training whatever

Link to this answer
Share Copy Link

Contributed on Oct 14 2022

KK SK

0 Answers Avg Quality 2/10

Closely Related Answers

pyspark train test split

Comment

########### Splitting non-sequence data
train_df, test_df = df.randomSplit([0.8, 0.2], seed=42)

########### Splitting sequence data with inherent sequence (eg : Time Series)
# Find how many days our data spans
from pyspark.sql.functions import datediff
range_in_days = datediff(max_date, min_date) # Find the no of days beteen minimum and maximum date
# Find the date to split the dataset on
from pyspark.sql.functions import date_add
split_in_days = round(range_in_days * 0.8) # Find 80% date split point
split_date = date_add(min_date, split_in_days) # Add split point with minimum date to get the split date
# Split the data into 80% train, 20% test
train_df = df.where(df['DATE'] < split_date) # Use filtering with split date to take only training data
test_df = df.where(df['DATE'] >= split_date) # Use filtering with split date to take only testing data

Popularity 4/10 Helpfulness 8/10 Language python

Source: Grepper

Tags: python pyspark

Link to this answer
Share Copy Link

Contributed on Feb 27 2024

Innocent Iguana

0 Answers Avg Quality 2/10

how to split data into training and testing in pyspark

Contents

More Related Answers

how to split data into training and testing in pyspark

Closely Related Answers

pyspark train test split

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.