Type Members

1.  final class DataStreamReader extends Logging

   Interface used to load a streaming Dataset from external storage systems (e.g.

   Interface used to load a streaming Dataset from external storage systems (e.g. file systems, key-value stores, etc). Use SparkSession.readStream to access this.

   Annotations

   @Evolving ()

   Since

   2.0.0

2.  final class DataStreamWriter [ T] extends AnyRef

   Interface used to write a streaming Dataset to external storage systems (e.g.

   Interface used to write a streaming Dataset to external storage systems (e.g. file systems, key-value stores, etc). Use Dataset.writeStream to access this.

   Annotations

   @Evolving ()

   Since

   2.0.0

3.  trait ExpiredTimerInfo extends Serializable

   :: Experimental ::

   :: Experimental ::

   Class used to provide access to expired timer's expiry time.

   Annotations

   @Experimental () @Evolving ()

   Since

   2.2.0

4.  trait GroupState [ S] extends LogicalGroupState [ S]

   :: Experimental ::

   :: Experimental ::

   Wrapper class for interacting with per-group state data in mapGroupsWithState and flatMapGroupsWithState operations on KeyValueGroupedDataset .

   S

   User-defined type of the state to be stored for each group. Must be encodable into Spark SQL types (see Encoder for more details).

   Annotations

   @Experimental () @Evolving ()

   Since

   2.2.0

5.  class GroupStateTimeout extends AnyRef

   Annotations

   @Experimental () @Evolving ()

6.  trait ListState extends Serializable

   :: Experimental ::

   :: Experimental ::

   Annotations

   @Experimental () @Evolving ()

   Since

   2.2.0

7.  trait MapState extends Serializable

   :: Experimental ::

   :: Experimental ::

   Annotations

   @Experimental () @Evolving ()

   Since

   2.2.0

8.  class OutputMode extends AnyRef

   Annotations

   @Evolving ()

9.  trait QueryInfo extends Serializable

   Represents the query info provided to the stateful processor used in the arbitrary state API v2 to easily identify task retries on the same partition.

   :: Experimental ::

   Represents the query info provided to the stateful processor used in the arbitrary state API v2 to easily identify task retries on the same partition.

   Annotations

   @Experimental () @Evolving ()

   Since

   2.2.0

10.  class SinkProgress extends Serializable

    Information about progress made for a sink in the execution of a StreamingQuery during a trigger.

    Information about progress made for a sink in the execution of a StreamingQuery during a trigger. See StreamingQueryProgress for more information.

    Annotations

    @Evolving ()

    Since

    2.1.0

11.  class SourceProgress extends Serializable

    Information about progress made for a source in the execution of a StreamingQuery during a trigger.

    Information about progress made for a source in the execution of a StreamingQuery during a trigger. See StreamingQueryProgress for more information.

    Annotations

    @Evolving ()

    Since

    2.1.0

12.  class StatefulProcessor extends Serializable

    Represents the arbitrary stateful logic that needs to be provided by the user to perform stateful manipulations on keyed streams.

    Represents the arbitrary stateful logic that needs to be provided by the user to perform stateful manipulations on keyed streams.

    Annotations

    @Evolving ()

    @Experimental ()

    Since

    2.1.0

13.  class StatefulProcessorHandle extends Serializable

    Represents the operation handle provided to the stateful processor used in the arbitrary state API v2.

    Represents the operation handle provided to the stateful processor used in the arbitrary state API v2.

    Annotations

    @Evolving ()

    @Experimental ()

    Since

    2.1.0

14.  class StateOperatorProgress extends Serializable

    Information about updates made to stateful operators in a StreamingQuery during a trigger.

    Information about updates made to stateful operators in a StreamingQuery during a trigger.

    Annotations

    @Evolving ()

15.  trait StreamingQuery extends AnyRef

    A handle to a query that is executing continuously in the background as new data arrives.

    A handle to a query that is executing continuously in the background as new data arrives. All these methods are thread-safe.

    Annotations

    @Evolving ()

    Since

    2.0.0

16.  class StreamingQueryException extends Exception

    Exception that stopped a StreamingQuery .

    Exception that stopped a StreamingQuery . Use cause get the actual exception that caused the failure.

    Annotations

    @Evolving ()

    Since

    2.0.0

17.  abstract class StreamingQueryListener extends AnyRef

    Interface for listening to events related to StreamingQueries .

    Interface for listening to events related to StreamingQueries .

    Annotations

    @Evolving ()

    Since

    2.0.0

    Note

    The methods are not thread-safe as they may be called from different threads.

18.  class StreamingQueryManager extends Logging

    A class to manage all the StreamingQuery active in a SparkSession .

    A class to manage all the StreamingQuery active in a SparkSession .

    Annotations

    @Evolving ()

    Since

    2.0.0

19.  class StreamingQueryProgress extends Serializable

    Information about progress made in the execution of a StreamingQuery during a trigger.

    Information about progress made in the execution of a StreamingQuery during a trigger. Each event relates to processing done for a single trigger of the streaming query. Events are emitted even when no new data is available to be processed.

    Annotations

    @Evolving ()

    Since

    2.1.0

20.  class StreamingQueryStatus extends Serializable

    Reports information about the instantaneous status of a streaming query.

    Reports information about the instantaneous status of a streaming query.

    Annotations

    @Evolving ()

    Since

    2.1.0

21.  trait TestGroupState [ S] extends GroupState [ S]

    :: Experimental ::

    :: Experimental ::

    The extended version of GroupState interface with extra getters of state machine fields to improve testability of the GroupState implementations which inherit from the extended interface.

    S

    User-defined type of the state to be stored for each group. Must be encodable intopark SQL types (see Encoder for more details).

    Annotations

    @Experimental () @Evolving ()

    Since

    3.2.0

22.  trait TimerValues extends Serializable

    :: Experimental ::

    Class used to provide access to timer values for processing and event time populated before method invocations using the arbitrary state API v2.

    Annotations

    @Experimental () @Evolving ()

    Since

    2.2.0

23.  class Trigger extends AnyRef

    Annotations

    @Evolving ()

24.  class TTLConfig extends Serializable

    TTL Configuration for state variable. State values will not be returned past , and will be eventually removed from the state store.

    TTL Configuration for state variable. State values will not be returned past , and will be eventually removed from the state store. Any state update resets the ttl to current processing time plus ttlDuration .
    
    Passing a TTL duration of zero will disable the TTL for the state variable. Users can also use the helper method TTLConfig.NONE in Scala or TTLConfig.NONE() in Java to disable TTL for the state variable.

    Annotations

    @Evolving ()

    @Experimental ()

    Since

    2.1.0

25.  trait ValueState extends Serializable

    :: Experimental ::

    Annotations

    @Experimental () @Evolving ()

    Since

    2.2.0