Core Classes

SparkSession(sparkContext[, jsparkSession, …])	The entry point to programming Spark with the Dataset and DataFrame API.
Catalog(sparkSession)	User-facing catalog API, accessible through SparkSession.catalog.
DataFrame(jdf, sql_ctx)	A distributed collection of data grouped into named columns.
Column(jc)	A column in a DataFrame.
Observation([name])	Class to observe (named) metrics on a DataFrame.
Row	A row in DataFrame.
GroupedData(jgd, df)	A set of methods for aggregations on a DataFrame, created by DataFrame.groupBy().
PandasCogroupedOps(gd1, gd2)	A logical grouping of two GroupedData, created by GroupedData.cogroup().
DataFrameNaFunctions(df)	Functionality for working with missing data in DataFrame.
DataFrameStatFunctions(df)	Functionality for statistic functions with DataFrame.
Window	Utility functions for defining window in DataFrames.
DataFrameReader(spark)	Interface used to load a DataFrame from external storage systems (e.g.
DataFrameWriter(df)	Interface used to write a DataFrame to external storage systems (e.g.

previous

Spark SQL

next

pyspark.sql.SparkSession