DistributedLDAModel¶

class pyspark.ml.clustering.DistributedLDAModel(java_model: Optional[JavaObject] = None)¶

Distributed model fitted by LDA. This type of model is currently only produced by Expectation-Maximization (EM).

This model stores the inferred topics, the full training dataset, and the topic distribution for each training document.

Methods

`clear`(param)	Clears a param from the param map if it has been explicitly set.
`copy`([extra])	Creates a copy of this instance with the same uid and some extra params.
`describeTopics`([maxTermsPerTopic])	Return the topics described by their top-weighted terms.
`estimatedDocConcentration`()	Value for `LDA.docConcentration` estimated from data.
`explainParam`(param)	Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
`explainParams`()	Returns the documentation of all params with their optionally default values and user-supplied values.
`extractParamMap`([extra])	Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
`getCheckpointFiles`()	If using checkpointing and `LDA.keepLastCheckpoint` is set to true, then there may be saved checkpoint files.
`getCheckpointInterval`()	Gets the value of checkpointInterval or its default value.
`getDocConcentration`()	Gets the value of `docConcentration` or its default value.
`getFeaturesCol`()	Gets the value of featuresCol or its default value.
`getK`()	Gets the value of `k` or its default value.
`getKeepLastCheckpoint`()	Gets the value of `keepLastCheckpoint` or its default value.
`getLearningDecay`()	Gets the value of `learningDecay` or its default value.
`getLearningOffset`()	Gets the value of `learningOffset` or its default value.
`getMaxIter`()	Gets the value of maxIter or its default value.
`getOptimizeDocConcentration`()	Gets the value of `optimizeDocConcentration` or its default value.
`getOptimizer`()	Gets the value of `optimizer` or its default value.
`getOrDefault`(param)	Gets the value of a param in the user-supplied param map or its default value.
`getParam`(paramName)	Gets a param by its name.
`getSeed`()	Gets the value of seed or its default value.
`getSubsamplingRate`()	Gets the value of `subsamplingRate` or its default value.
`getTopicConcentration`()	Gets the value of `topicConcentration` or its default value.
`getTopicDistributionCol`()	Gets the value of `topicDistributionCol` or its default value.
`hasDefault`(param)	Checks whether a param has a default value.
`hasParam`(paramName)	Tests whether this instance contains a param with a given (string) name.
`isDefined`(param)	Checks whether a param is explicitly set by user or has a default value.
`isDistributed`()	Indicates whether this instance is of type DistributedLDAModel
`isSet`(param)	Checks whether a param is explicitly set by user.
`load`(path)	Reads an ML instance from the input path, a shortcut of read().load(path).
`logLikelihood`(dataset)	Calculates a lower bound on the log likelihood of the entire corpus.
`logPerplexity`(dataset)	Calculate an upper bound on perplexity.
`logPrior`()	Log probability of the current parameter estimate: log P(topics, topic distributions for docs \| alpha, eta)
`read`()	Returns an MLReader instance for this class.
`save`(path)	Save this ML instance to the given path, a shortcut of ‘write().save(path)’.
`set`(param, value)	Sets a parameter in the embedded param map.
`setFeaturesCol`(value)	Sets the value of `featuresCol`.
`setSeed`(value)	Sets the value of `seed`.
`setTopicDistributionCol`(value)	Sets the value of `topicDistributionCol`.
`toLocal`()	Convert this distributed model to a local representation.
`topicsMatrix`()	Inferred topics, where each topic is represented by a distribution over terms.
`trainingLogLikelihood`()	Log likelihood of the observed tokens in the training set, given the current parameter estimates: log P(docs \| topics, topic distributions for docs, Dirichlet hyperparameters)
`transform`(dataset[, params])	Transforms the input dataset with optional parameters.
`vocabSize`()	Vocabulary size (number of terms or words in the vocabulary)
`write`()	Returns an MLWriter instance for this ML instance.

Attributes

`checkpointInterval`
`docConcentration`
`featuresCol`
`k`
`keepLastCheckpoint`
`learningDecay`
`learningOffset`
`maxIter`
`optimizeDocConcentration`
`optimizer`
`params`	Returns all params ordered by name.
`seed`
`subsamplingRate`
`topicConcentration`
`topicDistributionCol`

Methods Documentation

clear(param: pyspark.ml.param.Param) → None¶: Clears a param from the param map if it has been explicitly set.

copy(extra: Optional[ParamMap] = None) → JP¶

Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java pipeline component with extra params. So both the Python wrapper and the Java pipeline component get copied.

Parameters

extradict, optional: Extra parameters to copy to the new instance

Returns

JavaParams: Copy of this instance

describeTopics(maxTermsPerTopic: int = 10) → pyspark.sql.dataframe.DataFrame¶: Return the topics described by their top-weighted terms.

estimatedDocConcentration() → pyspark.ml.linalg.Vector ¶: Value for LDA.docConcentration estimated from data. If Online LDA was used and LDA.optimizeDocConcentration was set to false, then this returns the fixed (given) value for the LDA.docConcentration parameter.

explainParam(param: Union[str, pyspark.ml.param.Param]) → str¶: Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.

explainParams() → str¶: Returns the documentation of all params with their optionally default values and user-supplied values.

extractParamMap(extra: Optional[ParamMap] = None) → ParamMap¶

Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.

Parameters

extradict, optional: extra param values

Returns

dict: merged param map

getCheckpointFiles() → List[str]¶

If using checkpointing and LDA.keepLastCheckpoint is set to true, then there may be saved checkpoint files. This method is provided so that users can manage those files.

Returns

list: List of checkpoint files from training

Notes

Removing the checkpoints can cause failures if a partition is lost and is needed by certain DistributedLDAModel methods. Reference counting will clean up the checkpoints when this model and derivative data go out of scope.

getCheckpointInterval() → int¶: Gets the value of checkpointInterval or its default value.

getDocConcentration() → List[float]¶: Gets the value of docConcentration or its default value.

getFeaturesCol() → str¶: Gets the value of featuresCol or its default value.

getK() → int¶: Gets the value of k or its default value.

getKeepLastCheckpoint() → bool¶: Gets the value of keepLastCheckpoint or its default value.

getLearningDecay() → float¶: Gets the value of learningDecay or its default value.

getLearningOffset() → float¶: Gets the value of learningOffset or its default value.

getMaxIter() → int¶: Gets the value of maxIter or its default value.

getOptimizeDocConcentration() → bool¶: Gets the value of optimizeDocConcentration or its default value.

getOptimizer() → str¶: Gets the value of optimizer or its default value.

getOrDefault(param: Union[str, pyspark.ml.param.Param[T]]) → Union[Any, T]¶: Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.

getParam(paramName: str) → pyspark.ml.param.Param ¶: Gets a param by its name.

getSeed() → int¶: Gets the value of seed or its default value.

getSubsamplingRate() → float¶: Gets the value of subsamplingRate or its default value.

getTopicConcentration() → float¶: Gets the value of topicConcentration or its default value.

getTopicDistributionCol() → str¶: Gets the value of topicDistributionCol or its default value.

hasDefault(param: Union[str, pyspark.ml.param.Param[Any]]) → bool¶: Checks whether a param has a default value.

hasParam(paramName: str) → bool¶: Tests whether this instance contains a param with a given (string) name.

isDefined(param: Union[str, pyspark.ml.param.Param[Any]]) → bool¶: Checks whether a param is explicitly set by user or has a default value.

isDistributed() → bool¶: Indicates whether this instance is of type DistributedLDAModel

isSet(param: Union[str, pyspark.ml.param.Param[Any]]) → bool¶: Checks whether a param is explicitly set by user.

classmethod load(path: str) → RL¶: Reads an ML instance from the input path, a shortcut of read().load(path).

logLikelihood(dataset: pyspark.sql.dataframe.DataFrame) → float¶: Calculates a lower bound on the log likelihood of the entire corpus. See Equation (16) in the Online LDA paper (Hoffman et al., 2010).

Warning

If this model is an instance of DistributedLDAModel (produced when optimizer is set to “em”), this involves collecting a large topicsMatrix() to the driver. This implementation may be changed in the future.

logPerplexity(dataset: pyspark.sql.dataframe.DataFrame) → float¶: Calculate an upper bound on perplexity. (Lower is better.) See Equation (16) in the Online LDA paper (Hoffman et al., 2010).

Warning

If this model is an instance of DistributedLDAModel (produced when optimizer is set to “em”), this involves collecting a large topicsMatrix() to the driver. This implementation may be changed in the future.

logPrior() → float¶: Log probability of the current parameter estimate: log P(topics, topic distributions for docs | alpha, eta)

classmethod read() → pyspark.ml.util.JavaMLReader[RL]¶: Returns an MLReader instance for this class.

save(path: str) → None¶: Save this ML instance to the given path, a shortcut of ‘write().save(path)’.

set(param: pyspark.ml.param.Param, value: Any) → None¶: Sets a parameter in the embedded param map.

setFeaturesCol(value: str) → M¶: Sets the value of featuresCol.

setSeed(value: int) → M¶: Sets the value of seed.

setTopicDistributionCol(value: str) → M¶: Sets the value of topicDistributionCol.

toLocal() → pyspark.ml.clustering.LocalLDAModel ¶: Convert this distributed model to a local representation. This discards info about the training dataset.

Warning

This involves collecting a large topicsMatrix() to the driver.

topicsMatrix() → pyspark.ml.linalg.Matrix ¶: Inferred topics, where each topic is represented by a distribution over terms. This is a matrix of size vocabSize x k, where each column is a topic. No guarantees are given about the ordering of the topics.

Warning

If this model is actually a DistributedLDAModel instance produced by the Expectation-Maximization (“em”) optimizer, then this method could involve collecting a large amount of data to the driver (on the order of vocabSize x k).

trainingLogLikelihood() → float¶

Log likelihood of the observed tokens in the training set, given the current parameter estimates: log P(docs | topics, topic distributions for docs, Dirichlet hyperparameters)

Notes

This excludes the prior; for that, use logPrior().
Even with logPrior(), this is NOT the same as the data log likelihood given
the hyperparameters.
This is computed from the topic distributions computed during training. If you call
logLikelihood() on the same training dataset, the topic distributions will be computed again, possibly giving different results.

transform(dataset: pyspark.sql.dataframe.DataFrame, params: Optional[ParamMap] = None) → pyspark.sql.dataframe.DataFrame¶

Transforms the input dataset with optional parameters.

Parameters

datasetpyspark.sql.DataFrame: input dataset
paramsdict, optional: an optional param map that overrides embedded params.

Returns

pyspark.sql.DataFrame: transformed dataset

vocabSize() → int¶: Vocabulary size (number of terms or words in the vocabulary)

write() → pyspark.ml.util.JavaMLWriter¶: Returns an MLWriter instance for this ML instance.

Attributes Documentation

checkpointInterval = Param(parent='undefined', name='checkpointInterval', doc='set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations. Note: this setting will be ignored if the checkpoint directory is not set in the SparkContext.')¶

docConcentration: pyspark.ml.param.Param[List[float]] = Param(parent='undefined', name='docConcentration', doc='Concentration parameter (commonly named "alpha") for the prior placed on documents\' distributions over topics ("theta").')¶

featuresCol = Param(parent='undefined', name='featuresCol', doc='features column name.')¶

k: pyspark.ml.param.Param[int] = Param(parent='undefined', name='k', doc='The number of topics (clusters) to infer. Must be > 1.')¶

keepLastCheckpoint: pyspark.ml.param.Param[bool] = Param(parent='undefined', name='keepLastCheckpoint', doc='(For EM optimizer) If using checkpointing, this indicates whether to keep the last checkpoint. If false, then the checkpoint will be deleted. Deleting the checkpoint can cause failures if a data partition is lost, so set this bit with care.')¶

learningDecay: pyspark.ml.param.Param[float] = Param(parent='undefined', name='learningDecay', doc='Learning rate, set as anexponential decay rate. This should be between (0.5, 1.0] to guarantee asymptotic convergence.')¶

learningOffset: pyspark.ml.param.Param[float] = Param(parent='undefined', name='learningOffset', doc='A (positive) learning parameter that downweights early iterations. Larger values make early iterations count less')¶

maxIter = Param(parent='undefined', name='maxIter', doc='max number of iterations (>= 0).')¶

optimizeDocConcentration: pyspark.ml.param.Param[bool] = Param(parent='undefined', name='optimizeDocConcentration', doc='Indicates whether the docConcentration (Dirichlet parameter for document-topic distribution) will be optimized during training.')¶

optimizer: pyspark.ml.param.Param[str] = Param(parent='undefined', name='optimizer', doc='Optimizer or inference algorithm used to estimate the LDA model. Supported: online, em')¶

params¶: Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.

seed = Param(parent='undefined', name='seed', doc='random seed.')¶

subsamplingRate: pyspark.ml.param.Param[float] = Param(parent='undefined', name='subsamplingRate', doc='Fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent, in range (0, 1].')¶

topicConcentration: pyspark.ml.param.Param[float] = Param(parent='undefined', name='topicConcentration', doc='Concentration parameter (commonly named "beta" or "eta") for the prior placed on topic\' distributions over terms.')¶

topicDistributionCol: pyspark.ml.param.Param[str] = Param(parent='undefined', name='topicDistributionCol', doc='Output column with estimates of the topic mixture distribution for each document (often called "theta" in the literature). Returns a vector of zeros for an empty document.')¶

LocalLDAModel

PowerIterationClustering