Breaking News: Grepper is joining You.com. Read the official announcement!

pyspark sql

Innocent Iguana answered on February 23, 2024 Popularity 5/10 Helpfulness 7/10

answer pyspark sql

pyspark sql

Comment

df = spark.read.csv("filename.csv", header=True)
df.createOrReplaceTempView("table_name")
result = spark.sql("SELECT * FROM table_name") # simple query, result saved as dataframe
result.show()
result = spark.sql("DESCRIBE tablename") # See table information

# Window functions
query = """
SELECT *,
ROW_NUMBER() OVER(PARTITION BY train_id ORDER BY time) AS id
FROM schedule
"""
spark.sql(query)

# equivalent dot notation
window = Window.partitionBy('train_id').orderBy('time')
dfx = df.withColumn('id', row_number().over(window))

# CASE.. WHEN
query = """
SELECT id,
    CASE
        WHEN id < 25000 THEN 'Preface'
        WHEN id < 50000 THEN 'Chapter 1'
        WHEN id < 75000 THEN 'Chapter 2'
        ELSE 'Chapter 3'
    END AS title
FROM df
"""
spark.sql(query)

# equivalent dot notation
df2 = df.withColumn('title', when(df.id < 25000, 'Preface')
.when(df.id < 50000, 'Chapter 1')
.when(df.id < 75000, 'Chapter 2')
.otherwise('Chapter 3'))

Popularity 5/10 Helpfulness 7/10 Language sql

Source: Grepper

Tags: pyspark-sql sql

Link to this answer
Share Copy Link

Contributed on Feb 23 2024

Innocent Iguana

0 Answers Avg Quality 2/10

pyspark sql

Contents

More Related Answers

pyspark sql

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.