pyspark.pandas.read_delta

pyspark.pandas.read_delta(path: str, version: Optional[str] = None, timestamp: Optional[str] = None, index_col: Union[str, List[str], None] = None, **options: Any) → pyspark.pandas.frame.DataFrame

Read a Delta Lake table on some file system and return a DataFrame.

If the Delta Lake table is already stored in the catalog (aka the metastore), use ‘read_table’.

Parameters
pathstring

Path to the Delta Lake table.

versionstring, optional

Specifies the table version (based on Delta’s internal transaction version) to read from, using Delta’s time travel feature. This sets Delta’s ‘versionAsOf’ option. Note that this parameter and timestamp parameter cannot be used together, otherwise it will raise a ValueError.

timestampstring, optional

Specifies the table version (based on timestamp) to read from, using Delta’s time travel feature. This must be a valid date or timestamp string in Spark, and sets Delta’s ‘timestampAsOf’ option. Note that this parameter and version parameter cannot be used together, otherwise it will raise a ValueError.

index_colstr or list of str, optional, default: None

Index column of table in Spark.

options

Additional options that can be passed onto Delta.

Returns
DataFrame

Examples

>>> ps.range(1).to_delta('%s/read_delta/foo' % path)  
>>> ps.read_delta('%s/read_delta/foo' % path)  
   id
0   0
>>> ps.range(10, 15, num_partitions=1).to_delta('%s/read_delta/foo' % path,
...                                             mode='overwrite')  
>>> ps.read_delta('%s/read_delta/foo' % path)  
   id
0  10
1  11
2  12
3  13
4  14
>>> ps.read_delta('%s/read_delta/foo' % path, version=0)  
   id
0   0

You can preserve the index in the roundtrip as below.

>>> ps.range(10, 15, num_partitions=1).to_delta(
...     '%s/read_delta/bar' % path, index_col="index")  
>>> ps.read_delta('%s/read_delta/bar' % path, index_col="index")  
       id
index
0      10
1      11
2      12
3      13
4      14