pyspark.sql.streaming.DataStreamReader.schema¶
-
DataStreamReader.schema(schema: Union[pyspark.sql.types.StructType, str]) → pyspark.sql.streaming.readwriter.DataStreamReader¶ Specifies the input schema.
Some data sources (e.g. JSON) can infer the input schema automatically from data. By specifying the schema here, the underlying data source can skip the schema inference step, and thus speed up data loading.
- Parameters
- schema
pyspark.sql.types.StructTypeor str a
pyspark.sql.types.StructTypeobject or a DDL-formatted string (For examplecol0 INT, col1 DOUBLE).
- schema
Notes
This API is evolving.
Examples
>>> s = spark.readStream.schema(sdf_schema) >>> s = spark.readStream.schema("col0 INT, col1 DOUBLE")