PYSPARK BASIC CONCEPTs

PYSPARK BASIC CONCEPTS PART-2

by

6. Transformations and Actions

Copy Text 
"""
Transformations (Lazy):
- map()
- filter()
- flatMap()
- union()

Actions (Eager):
- collect()
- count()
- take()
- first()
"""

7. Reading CSV File

Copy Text 
# Basic read
df = spark.read.csv("path/to/file.csv")

# With options
df = spark.read \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .csv("path/to/file.csv")

8. Narrow vs Wide Transformations

Copy Text 
"""
Narrow Transformations:
- One-to-one partition dependency
- Examples: map(), filter()
- No shuffle required

Wide Transformations:
- Many-to-many partition dependency
- Examples: groupByKey(), reduceByKey()
- Requires shuffle
"""

9. Lazy Evaluation

Copy Text 
"""
PySpark uses lazy evaluation:
- Transformations are not executed immediately
- Actions trigger execution
- Creates DAG of operations
- Optimizes execution plan
"""

10. select() vs withColumn() vs selectExpr()

Copy Text 
# select()
df.select("name", "age")

# withColumn()
df.withColumn("age_doubled", col("age") * 2)

# selectExpr()
df.selectExpr("name", "age * 2 as age_doubled")

11. Filtering Rows

Copy Text 
# Using filter
df.filter(df.age > 25)

# Using where
df.where(df.age > 25)

# Multiple conditions
df.filter((df.age > 25) & (df.salary >= 50000))

12. Handling Missing Values

Copy Text 
# Drop nulls
df.dropna()

# Fill nulls
df.fillna(0)
df.fillna({"age": 0, "name": "Unknown"})

Leave a Comment