Databricks PySpark API Reference

This documentation is no longer maintained. For the latest PySpark API reference, see the Databricks documentation.

This page lists an overview of all public PySpark modules, classes, functions and methods.

Pandas API on Spark follows the API specifications of latest pandas release.

• Spark SQL
  • Core Classes
  • Spark Session
  • Configuration
  • Input/Output
  • DataFrame
  • Column
  • Data Types
  • Row
  • Functions
  • Window
  • Grouping
  • Catalog
  • Avro
  • Observation
• Pandas API on Spark
  • Input/Output
  • General functions
  • Series
  • DataFrame
  • Index objects
  • Window
  • GroupBy
  • Machine Learning utilities
  • Extensions
• Structured Streaming
  • Core Classes
  • Input/Output
  • Query Management
• MLlib (DataFrame-based)
  • Pipeline APIs
  • Parameters
  • Feature
  • Classification
  • Clustering
  • Functions
  • Vector and Matrix
  • Recommendation
  • Regression
  • Statistics
  • Tuning
  • Evaluation
  • Frequency Pattern Mining
  • Image
  • Utilities
• Spark Streaming
  • Core Classes
  • Streaming Management
  • Input and Output
  • Transformations and Actions
  • Kinesis
• MLlib (RDD-based)
  • Classification
  • Clustering
  • Evaluation
  • Feature
  • Frequency Pattern Mining
  • Vector and Matrix
  • Random
  • Recommendation
  • Regression
  • Statistics
  • Tree
  • Utilities
• Spark Core
  • Public Classes
  • Spark Context APIs
  • RDD APIs
  • Broadcast and Accumulator
  • Management
• Resource Management
  • Core Classes

previous

PySpark Documentation

next

Spark SQL