package streaming
- Alphabetic
- Public
- All
Type Members
-
final
class
DataStreamReader
extends
Logging
Interface used to load a streaming
Dataset
from external storage systems (e.g.Interface used to load a streaming
Dataset
from external storage systems (e.g. file systems, key-value stores, etc). UseSparkSession.readStream
to access this.- Annotations
- @Evolving ()
- Since
-
2.0.0
-
final
class
DataStreamWriter
[
T]
extends
AnyRef
Interface used to write a streaming
Dataset
to external storage systems (e.g.Interface used to write a streaming
Dataset
to external storage systems (e.g. file systems, key-value stores, etc). UseDataset.writeStream
to access this.- Annotations
- @Evolving ()
- Since
-
2.0.0
-
trait
ExpiredTimerInfo
extends
Serializable
:: Experimental ::
:: Experimental ::
Class used to provide access to expired timer's expiry time.
- Annotations
- @Experimental () @Evolving ()
- Since
-
2.2.0
-
trait
GroupState
[
S]
extends
LogicalGroupState
[
S]
:: Experimental ::
:: Experimental ::
Wrapper class for interacting with per-group state data in
mapGroupsWithState
andflatMapGroupsWithState
operations onKeyValueGroupedDataset
.- S
-
User-defined type of the state to be stored for each group. Must be encodable into Spark SQL types (see
Encoder
for more details).
- Annotations
- @Experimental () @Evolving ()
- Since
-
2.2.0
-
class
GroupStateTimeout
extends
AnyRef
- Annotations
- @Experimental () @Evolving ()
-
trait
ListState
extends
Serializable
:: Experimental ::
:: Experimental ::
- Annotations
- @Experimental () @Evolving ()
- Since
-
2.2.0
-
trait
MapState
extends
Serializable
:: Experimental ::
:: Experimental ::
- Annotations
- @Experimental () @Evolving ()
- Since
-
2.2.0
-
class
OutputMode
extends
AnyRef
- Annotations
- @Evolving ()
-
trait
QueryInfo
extends
Serializable
Represents the query info provided to the stateful processor used in the arbitrary state API v2 to easily identify task retries on the same partition.
:: Experimental ::
Represents the query info provided to the stateful processor used in the arbitrary state API v2 to easily identify task retries on the same partition.
- Annotations
- @Experimental () @Evolving ()
- Since
-
2.2.0
-
class
SinkProgress
extends
Serializable
Information about progress made for a sink in the execution of a StreamingQuery during a trigger.
Information about progress made for a sink in the execution of a StreamingQuery during a trigger. See StreamingQueryProgress for more information.
- Annotations
- @Evolving ()
- Since
-
2.1.0
-
class
SourceProgress
extends
Serializable
Information about progress made for a source in the execution of a StreamingQuery during a trigger.
Information about progress made for a source in the execution of a StreamingQuery during a trigger. See StreamingQueryProgress for more information.
- Annotations
- @Evolving ()
- Since
-
2.1.0
-
class
StatefulProcessor
extends
Serializable
Represents the arbitrary stateful logic that needs to be provided by the user to perform stateful manipulations on keyed streams.
Represents the arbitrary stateful logic that needs to be provided by the user to perform stateful manipulations on keyed streams.
- Annotations
- @Evolving ()
- @Experimental ()
- Since
-
2.1.0
-
class
StatefulProcessorHandle
extends
Serializable
Represents the operation handle provided to the stateful processor used in the arbitrary state API v2.
Represents the operation handle provided to the stateful processor used in the arbitrary state API v2.
- Annotations
- @Evolving ()
- @Experimental ()
- Since
-
2.1.0
-
class
StateOperatorProgress
extends
Serializable
Information about updates made to stateful operators in a StreamingQuery during a trigger.
Information about updates made to stateful operators in a StreamingQuery during a trigger.
- Annotations
- @Evolving ()
-
trait
StreamingQuery
extends
AnyRef
A handle to a query that is executing continuously in the background as new data arrives.
A handle to a query that is executing continuously in the background as new data arrives. All these methods are thread-safe.
- Annotations
- @Evolving ()
- Since
-
2.0.0
-
class
StreamingQueryException
extends
Exception
Exception that stopped a StreamingQuery .
Exception that stopped a StreamingQuery . Use
cause
get the actual exception that caused the failure.- Annotations
- @Evolving ()
- Since
-
2.0.0
-
abstract
class
StreamingQueryListener
extends
AnyRef
Interface for listening to events related to StreamingQueries .
Interface for listening to events related to StreamingQueries .
- Annotations
- @Evolving ()
- Since
-
2.0.0
- Note
-
The methods are not thread-safe as they may be called from different threads.
-
class
StreamingQueryManager
extends
Logging
A class to manage all the StreamingQuery active in a
SparkSession
.A class to manage all the StreamingQuery active in a
SparkSession
.- Annotations
- @Evolving ()
- Since
-
2.0.0
-
class
StreamingQueryProgress
extends
Serializable
Information about progress made in the execution of a StreamingQuery during a trigger.
Information about progress made in the execution of a StreamingQuery during a trigger. Each event relates to processing done for a single trigger of the streaming query. Events are emitted even when no new data is available to be processed.
- Annotations
- @Evolving ()
- Since
-
2.1.0
-
class
StreamingQueryStatus
extends
Serializable
Reports information about the instantaneous status of a streaming query.
Reports information about the instantaneous status of a streaming query.
- Annotations
- @Evolving ()
- Since
-
2.1.0
-
trait
TestGroupState
[
S]
extends
GroupState
[
S]
:: Experimental ::
:: Experimental ::
The extended version of GroupState interface with extra getters of state machine fields to improve testability of the GroupState implementations which inherit from the extended interface.
- S
-
User-defined type of the state to be stored for each group. Must be encodable intopark SQL types (see
Encoder
for more details).
- Annotations
- @Experimental () @Evolving ()
- Since
-
3.2.0
-
trait
TimerValues
extends
Serializable
:: Experimental ::
Class used to provide access to timer values for processing and event time populated before method invocations using the arbitrary state API v2.
- Annotations
- @Experimental () @Evolving ()
- Since
-
2.2.0
-
class
Trigger
extends
AnyRef
- Annotations
- @Evolving ()
-
class
TTLConfig
extends
Serializable
TTL Configuration for state variable. State values will not be returned past
, and will be eventually removed from the state store. TTL Configuration for state variable. State values will not be returned past
, and will be eventually removed from the state store. Any state update resets the ttl to current processing time plus ttlDuration
.
Passing a TTL duration of zero will disable the TTL for the state variable. Users can also use the helper methodTTLConfig.NONE
in Scala orTTLConfig.NONE()
in Java to disable TTL for the state variable.- Annotations
- @Evolving ()
- @Experimental ()
- Since
-
2.1.0
-
trait
ValueState
extends
Serializable
:: Experimental ::
- Annotations
- @Experimental () @Evolving ()
- Since
-
2.2.0
Value Members
- object DataStreamWriter
-
object
StreamingQueryListener
Companion object of StreamingQueryListener that defines the listener events.
Companion object of StreamingQueryListener that defines the listener events.
- Annotations
- @Evolving ()
- Since
-
2.0.0
- object TestGroupState