LDA¶

class pyspark.ml.clustering.LDA(*, featuresCol: str = 'features', maxIter: int = 20, seed: Optional[int] = None, checkpointInterval: int = 10, k: int = 10, optimizer: str = 'online', learningOffset: float = 1024.0, learningDecay: float = 0.51, subsamplingRate: float = 0.05, optimizeDocConcentration: bool = True, docConcentration: Optional[List[float]] = None, topicConcentration: Optional[float] = None, topicDistributionCol: str = 'topicDistribution', keepLastCheckpoint: bool = True)¶

Latent Dirichlet Allocation (LDA), a topic model designed for text documents.

Terminology:

“term” = “word”: an element of the vocabulary

“token”: instance of a term appearing in a document

“topic”: multinomial distribution over terms representing some concept

“document”: one piece of text, corresponding to one row in the input data

Original LDA paper (journal version):: Blei, Ng, and Jordan. “Latent Dirichlet Allocation.” JMLR, 2003.

Input data (featuresCol): LDA is given a collection of documents as input data, via the featuresCol parameter. Each document is specified as a Vector of length vocabSize, where each entry is the count for the corresponding term (word) in the document. Feature transformers such as pyspark.ml.feature.Tokenizer and pyspark.ml.feature.CountVectorizer can be useful for converting text to word count vectors.

Examples

>>> from pyspark.ml.linalg import Vectors, SparseVector
>>> from pyspark.ml.clustering import LDA
>>> df = spark.createDataFrame([[1, Vectors.dense([0.0, 1.0])],
...      [2, SparseVector(2, {0: 1.0})],], ["id", "features"])
>>> lda = LDA(k=2, seed=1, optimizer="em")
>>> lda.setMaxIter(10)
LDA...
>>> lda.getMaxIter()
10
>>> lda.clear(lda.maxIter)
>>> model = lda.fit(df)
>>> model.transform(df).show()
+---+-------------+--------------------+
| id|     features|   topicDistribution|
+---+-------------+--------------------+
|  1|    [0.0,1.0]|...
|  2|(2,[0],[1.0])|...
+---+-------------+--------------------+
...
>>> model.setSeed(1)
DistributedLDAModel...
>>> model.getTopicDistributionCol()
'topicDistribution'
>>> model.isDistributed()
True
>>> localModel = model.toLocal()
>>> localModel.isDistributed()
False
>>> model.vocabSize()
2
>>> model.describeTopics().show()
+-----+-----------+--------------------+
|topic|termIndices|         termWeights|
+-----+-----------+--------------------+
|    0|     [1, 0]|[0.50401530077160...|
|    1|     [0, 1]|[0.50401530077160...|
+-----+-----------+--------------------+
...
>>> model.topicsMatrix()
DenseMatrix(2, 2, [0.496, 0.504, 0.504, 0.496], 0)
>>> lda_path = temp_path + "/lda"
>>> lda.save(lda_path)
>>> sameLDA = LDA.load(lda_path)
>>> distributed_model_path = temp_path + "/lda_distributed_model"
>>> model.save(distributed_model_path)
>>> sameModel = DistributedLDAModel.load(distributed_model_path)
>>> local_model_path = temp_path + "/lda_local_model"
>>> localModel.save(local_model_path)
>>> sameLocalModel = LocalLDAModel.load(local_model_path)
>>> model.transform(df).take(1) == sameLocalModel.transform(df).take(1)
True

Methods

`clear`(param)	Clears a param from the param map if it has been explicitly set.
`copy`([extra])	Creates a copy of this instance with the same uid and some extra params.
`explainParam`(param)	Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
`explainParams`()	Returns the documentation of all params with their optionally default values and user-supplied values.
`extractParamMap`([extra])	Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
`fit`(dataset[, params])	Fits a model to the input dataset with optional parameters.
`fitMultiple`(dataset, paramMaps)	Fits a model to the input dataset for each param map in paramMaps.
`getCheckpointInterval`()	Gets the value of checkpointInterval or its default value.
`getDocConcentration`()	Gets the value of `docConcentration` or its default value.
`getFeaturesCol`()	Gets the value of featuresCol or its default value.
`getK`()	Gets the value of `k` or its default value.
`getKeepLastCheckpoint`()	Gets the value of `keepLastCheckpoint` or its default value.
`getLearningDecay`()	Gets the value of `learningDecay` or its default value.
`getLearningOffset`()	Gets the value of `learningOffset` or its default value.
`getMaxIter`()	Gets the value of maxIter or its default value.
`getOptimizeDocConcentration`()	Gets the value of `optimizeDocConcentration` or its default value.
`getOptimizer`()	Gets the value of `optimizer` or its default value.
`getOrDefault`(param)	Gets the value of a param in the user-supplied param map or its default value.
`getParam`(paramName)	Gets a param by its name.
`getSeed`()	Gets the value of seed or its default value.
`getSubsamplingRate`()	Gets the value of `subsamplingRate` or its default value.
`getTopicConcentration`()	Gets the value of `topicConcentration` or its default value.
`getTopicDistributionCol`()	Gets the value of `topicDistributionCol` or its default value.
`hasDefault`(param)	Checks whether a param has a default value.
`hasParam`(paramName)	Tests whether this instance contains a param with a given (string) name.
`isDefined`(param)	Checks whether a param is explicitly set by user or has a default value.
`isSet`(param)	Checks whether a param is explicitly set by user.
`load`(path)	Reads an ML instance from the input path, a shortcut of read().load(path).
`read`()	Returns an MLReader instance for this class.
`save`(path)	Save this ML instance to the given path, a shortcut of ‘write().save(path)’.
`set`(param, value)	Sets a parameter in the embedded param map.
`setCheckpointInterval`(value)	Sets the value of `checkpointInterval`.
`setDocConcentration`(value)	Sets the value of `docConcentration`.
`setFeaturesCol`(value)	Sets the value of `featuresCol`.
`setK`(value)	Sets the value of `k`.
`setKeepLastCheckpoint`(value)	Sets the value of `keepLastCheckpoint`.
`setLearningDecay`(value)	Sets the value of `learningDecay`.
`setLearningOffset`(value)	Sets the value of `learningOffset`.
`setMaxIter`(value)	Sets the value of `maxIter`.
`setOptimizeDocConcentration`(value)	Sets the value of `optimizeDocConcentration`.
`setOptimizer`(value)	Sets the value of `optimizer`.
`setParams`(self, \*[, featuresCol, maxIter, …])	Sets params for LDA.
`setSeed`(value)	Sets the value of `seed`.
`setSubsamplingRate`(value)	Sets the value of `subsamplingRate`.
`setTopicConcentration`(value)	Sets the value of `topicConcentration`.
`setTopicDistributionCol`(value)	Sets the value of `topicDistributionCol`.
`write`()	Returns an MLWriter instance for this ML instance.

Attributes

`checkpointInterval`
`docConcentration`
`featuresCol`
`k`
`keepLastCheckpoint`
`learningDecay`
`learningOffset`
`maxIter`
`optimizeDocConcentration`
`optimizer`
`params`	Returns all params ordered by name.
`seed`
`subsamplingRate`
`topicConcentration`
`topicDistributionCol`

Methods Documentation

clear(param: pyspark.ml.param.Param) → None¶: Clears a param from the param map if it has been explicitly set.

copy(extra: Optional[ParamMap] = None) → JP¶

Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java pipeline component with extra params. So both the Python wrapper and the Java pipeline component get copied.

Parameters

extradict, optional: Extra parameters to copy to the new instance

Returns

JavaParams: Copy of this instance

explainParam(param: Union[str, pyspark.ml.param.Param]) → str¶: Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.

explainParams() → str¶: Returns the documentation of all params with their optionally default values and user-supplied values.

extractParamMap(extra: Optional[ParamMap] = None) → ParamMap¶

Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.

Parameters

extradict, optional: extra param values

Returns

dict: merged param map

fit(dataset: pyspark.sql.dataframe.DataFrame, params: Union[ParamMap, List[ParamMap], Tuple[ParamMap], None] = None) → Union[M, List[M]]¶

Fits a model to the input dataset with optional parameters.

Parameters

datasetpyspark.sql.DataFrame: input dataset.
paramsdict or list or tuple, optional: an optional param map that overrides embedded params. If a list/tuple of param maps is given, this calls fit on each param map and returns a list of models.

Returns

Transformer or a list of Transformer: fitted model(s)

fitMultiple(dataset: pyspark.sql.dataframe.DataFrame, paramMaps: Sequence[ParamMap]) → Iterator[Tuple[int, M]]¶

Fits a model to the input dataset for each param map in paramMaps.

Parameters

datasetpyspark.sql.DataFrame: input dataset.
paramMapscollections.abc.Sequence: A Sequence of param maps.

Returns

_FitMultipleIterator: A thread safe iterable which contains one model for each param map. Each call to next(modelIterator) will return (index, model) where model was fit using paramMaps[index]. index values may not be sequential.

getCheckpointInterval() → int¶: Gets the value of checkpointInterval or its default value.

getDocConcentration() → List[float]¶: Gets the value of docConcentration or its default value.

getFeaturesCol() → str¶: Gets the value of featuresCol or its default value.

getK() → int¶: Gets the value of k or its default value.

getKeepLastCheckpoint() → bool¶: Gets the value of keepLastCheckpoint or its default value.

getLearningDecay() → float¶: Gets the value of learningDecay or its default value.

getLearningOffset() → float¶: Gets the value of learningOffset or its default value.

getMaxIter() → int¶: Gets the value of maxIter or its default value.

getOptimizeDocConcentration() → bool¶: Gets the value of optimizeDocConcentration or its default value.

getOptimizer() → str¶: Gets the value of optimizer or its default value.

getOrDefault(param: Union[str, pyspark.ml.param.Param[T]]) → Union[Any, T]¶: Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.

getParam(paramName: str) → pyspark.ml.param.Param ¶: Gets a param by its name.

getSeed() → int¶: Gets the value of seed or its default value.

getSubsamplingRate() → float¶: Gets the value of subsamplingRate or its default value.

getTopicConcentration() → float¶: Gets the value of topicConcentration or its default value.

getTopicDistributionCol() → str¶: Gets the value of topicDistributionCol or its default value.

hasDefault(param: Union[str, pyspark.ml.param.Param[Any]]) → bool¶: Checks whether a param has a default value.

hasParam(paramName: str) → bool¶: Tests whether this instance contains a param with a given (string) name.

isDefined(param: Union[str, pyspark.ml.param.Param[Any]]) → bool¶: Checks whether a param is explicitly set by user or has a default value.

isSet(param: Union[str, pyspark.ml.param.Param[Any]]) → bool¶: Checks whether a param is explicitly set by user.

classmethod load(path: str) → RL¶: Reads an ML instance from the input path, a shortcut of read().load(path).

classmethod read() → pyspark.ml.util.JavaMLReader[RL]¶: Returns an MLReader instance for this class.

save(path: str) → None¶: Save this ML instance to the given path, a shortcut of ‘write().save(path)’.

set(param: pyspark.ml.param.Param, value: Any) → None¶: Sets a parameter in the embedded param map.

setCheckpointInterval(value: int) → pyspark.ml.clustering.LDA ¶: Sets the value of checkpointInterval.

setDocConcentration(value: List[float]) → pyspark.ml.clustering.LDA ¶

Sets the value of docConcentration.

Examples

>>> algo = LDA().setDocConcentration([0.1, 0.2])
>>> algo.getDocConcentration()
[0.1..., 0.2...]

setFeaturesCol(value: str) → pyspark.ml.clustering.LDA ¶: Sets the value of featuresCol.

setK(value: int) → pyspark.ml.clustering.LDA ¶

Sets the value of k.

>>> algo = LDA().setK(10)
>>> algo.getK()
10

setKeepLastCheckpoint(value: bool) → pyspark.ml.clustering.LDA ¶

Sets the value of keepLastCheckpoint.

Examples

>>> algo = LDA().setKeepLastCheckpoint(False)
>>> algo.getKeepLastCheckpoint()
False

setLearningDecay(value: float) → pyspark.ml.clustering.LDA ¶

Sets the value of learningDecay.

Examples

>>> algo = LDA().setLearningDecay(0.1)
>>> algo.getLearningDecay()
0.1...

setLearningOffset(value: float) → pyspark.ml.clustering.LDA ¶

Sets the value of learningOffset.

Examples

>>> algo = LDA().setLearningOffset(100)
>>> algo.getLearningOffset()
100.0

setMaxIter(value: int) → pyspark.ml.clustering.LDA ¶: Sets the value of maxIter.

setOptimizeDocConcentration(value: bool) → pyspark.ml.clustering.LDA ¶

Sets the value of optimizeDocConcentration.

Examples

>>> algo = LDA().setOptimizeDocConcentration(True)
>>> algo.getOptimizeDocConcentration()
True

setOptimizer(value: str) → pyspark.ml.clustering.LDA ¶

Sets the value of optimizer. Currently only support ‘em’ and ‘online’.

Examples

>>> algo = LDA().setOptimizer("em")
>>> algo.getOptimizer()
'em'

setParams(self, \*, featuresCol="features", maxIter=20, seed=None, checkpointInterval=10, k=10, optimizer="online", learningOffset=1024.0, learningDecay=0.51, subsamplingRate=0.05, optimizeDocConcentration=True, docConcentration=None, topicConcentration=None, topicDistributionCol="topicDistribution", keepLastCheckpoint=True)¶: Sets params for LDA.

setSeed(value: int) → pyspark.ml.clustering.LDA ¶: Sets the value of seed.

setSubsamplingRate(value: float) → pyspark.ml.clustering.LDA ¶

Sets the value of subsamplingRate.

Examples

>>> algo = LDA().setSubsamplingRate(0.1)
>>> algo.getSubsamplingRate()
0.1...

setTopicConcentration(value: float) → pyspark.ml.clustering.LDA ¶

Sets the value of topicConcentration.

Examples

>>> algo = LDA().setTopicConcentration(0.5)
>>> algo.getTopicConcentration()
0.5...

setTopicDistributionCol(value: str) → pyspark.ml.clustering.LDA ¶

Sets the value of topicDistributionCol.

Examples

>>> algo = LDA().setTopicDistributionCol("topicDistributionCol")
>>> algo.getTopicDistributionCol()
'topicDistributionCol'

write() → pyspark.ml.util.JavaMLWriter¶: Returns an MLWriter instance for this ML instance.

Attributes Documentation

checkpointInterval = Param(parent='undefined', name='checkpointInterval', doc='set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations. Note: this setting will be ignored if the checkpoint directory is not set in the SparkContext.')¶

docConcentration = Param(parent='undefined', name='docConcentration', doc='Concentration parameter (commonly named "alpha") for the prior placed on documents\' distributions over topics ("theta").')¶

featuresCol = Param(parent='undefined', name='featuresCol', doc='features column name.')¶

k = Param(parent='undefined', name='k', doc='The number of topics (clusters) to infer. Must be > 1.')¶

keepLastCheckpoint = Param(parent='undefined', name='keepLastCheckpoint', doc='(For EM optimizer) If using checkpointing, this indicates whether to keep the last checkpoint. If false, then the checkpoint will be deleted. Deleting the checkpoint can cause failures if a data partition is lost, so set this bit with care.')¶

learningDecay = Param(parent='undefined', name='learningDecay', doc='Learning rate, set as anexponential decay rate. This should be between (0.5, 1.0] to guarantee asymptotic convergence.')¶

learningOffset = Param(parent='undefined', name='learningOffset', doc='A (positive) learning parameter that downweights early iterations. Larger values make early iterations count less')¶

maxIter = Param(parent='undefined', name='maxIter', doc='max number of iterations (>= 0).')¶

optimizeDocConcentration = Param(parent='undefined', name='optimizeDocConcentration', doc='Indicates whether the docConcentration (Dirichlet parameter for document-topic distribution) will be optimized during training.')¶

optimizer = Param(parent='undefined', name='optimizer', doc='Optimizer or inference algorithm used to estimate the LDA model. Supported: online, em')¶

params¶: Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.

seed = Param(parent='undefined', name='seed', doc='random seed.')¶

subsamplingRate = Param(parent='undefined', name='subsamplingRate', doc='Fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent, in range (0, 1].')¶

topicConcentration = Param(parent='undefined', name='topicConcentration', doc='Concentration parameter (commonly named "beta" or "eta") for the prior placed on topic\' distributions over terms.')¶

topicDistributionCol = Param(parent='undefined', name='topicDistributionCol', doc='Output column with estimates of the topic mixture distribution for each document (often called "theta" in the literature). Returns a vector of zeros for an empty document.')¶

GaussianMixtureSummary

LDAModel