pyspark.sql.DataFrameReader.load¶

DataFrameReader.load(path: Union[str, List[str], None] = None, format: Optional[str] = None, schema: Union[pyspark.sql.types.StructType, str, None] = None, **options: OptionalPrimitiveType) → DataFrame¶

Loads data from a data source and returns it as a DataFrame.

Parameters

pathstr or list, optional: optional string or a list of string for file-system backed data sources.
formatstr, optional: optional string for format of the data source. Default to ‘parquet’.
schemapyspark.sql.types.StructType or str, optional: optional pyspark.sql.types.StructType for the input schema or a DDL-formatted string (For example col0 INT, col1 DOUBLE).
**optionsdict: all other string options

Examples

>>> df = spark.read.format("parquet").load('python/test_support/sql/parquet_partitioned',
...     opt1=True, opt2=1, opt3='str')
>>> df.dtypes
[('name', 'string'), ('year', 'int'), ('month', 'int'), ('day', 'int')]

>>> df = spark.read.format('json').load(['python/test_support/sql/people.json',
...     'python/test_support/sql/people1.json'])
>>> df.dtypes
[('age', 'bigint'), ('aka', 'string'), ('name', 'string')]

pyspark.sql.DataFrameReader.json

pyspark.sql.DataFrameReader.option