StreamingKMeans

class pyspark.mllib.clustering.StreamingKMeans(k: int = 2, decayFactor: float = 1.0, timeUnit: str = 'batches')

Provides methods to set k, decayFactor, timeUnit to configure the KMeans algorithm for fitting and predicting on incoming dstreams. More details on how the centroids are updated are provided under the docs of StreamingKMeansModel.

Parameters

kint, optional

Number of clusters. (default: 2)

decayFactorfloat, optional

Forgetfulness of the previous centroids. (default: 1.0)

timeUnitstr, optional

Can be “batches” or “points”. If points, then the decay factor is raised to the power of number of new points and if batches, then decay factor will be used as is. (default: “batches”)

Methods

latestModel()	Return the latest model
predictOn(dstream)	Make predictions on a dstream.
predictOnValues(dstream)	Make predictions on a keyed dstream.
setDecayFactor(decayFactor)	Set decay factor.
setHalfLife(halfLife, timeUnit)	Set number of batches after which the centroids of that particular batch has half the weightage.
setInitialCenters(centers, weights)	Set initial centers.
setK(k)	Set number of clusters.
setRandomCenters(dim, weight, seed)	Set the initial centers to be random samples from a gaussian population with constant weights.
trainOn(dstream)	Train the model on the incoming dstream.

Methods Documentation

latestModel() → Optional[pyspark.mllib.clustering.StreamingKMeansModel]

Return the latest model

predictOn(dstream: DStream[VectorLike]) → DStream[int]

Make predictions on a dstream. Returns a transformed dstream object

predictOnValues(dstream: DStream[Tuple[T, VectorLike]]) → DStream[Tuple[T, int]]

Make predictions on a keyed dstream. Returns a transformed dstream object.

setDecayFactor(decayFactor: float) → pyspark.mllib.clustering.StreamingKMeans

Set decay factor.

setHalfLife(halfLife: float, timeUnit: str) → pyspark.mllib.clustering.StreamingKMeans

Set number of batches after which the centroids of that particular batch has half the weightage.

setInitialCenters(centers: List[VectorLike], weights: List[float]) → StreamingKMeans

Set initial centers. Should be set before calling trainOn.

setK(k: int) → pyspark.mllib.clustering.StreamingKMeans

Set number of clusters.

setRandomCenters(dim: int, weight: float, seed: int) → pyspark.mllib.clustering.StreamingKMeans

Set the initial centers to be random samples from a gaussian population with constant weights.

trainOn(dstream: DStream[VectorLike]) → None

Train the model on the incoming dstream.

previous

PowerIterationClustering

next

StreamingKMeansModel