StandardScaler¶

class pyspark.mllib.feature.StandardScaler(withMean: bool = False, withStd: bool = True)¶

Standardizes features by removing the mean and scaling to unit variance using column summary statistics on the samples in the training set.

Parameters

withMeanbool, optional: False by default. Centers the data with mean before scaling. It will build a dense output, so take care when applying to sparse input.
withStdbool, optional: True by default. Scales the data to unit standard deviation.

Examples

>>> vs = [Vectors.dense([-2.0, 2.3, 0]), Vectors.dense([3.8, 0.0, 1.9])]
>>> dataset = sc.parallelize(vs)
>>> standardizer = StandardScaler(True, True)
>>> model = standardizer.fit(dataset)
>>> result = model.transform(dataset)
>>> for r in result.collect(): r
DenseVector([-0.7071, 0.7071, -0.7071])
DenseVector([0.7071, -0.7071, 0.7071])
>>> int(model.std[0])
4
>>> int(model.mean[0]*10)
9
>>> model.withStd
True
>>> model.withMean
True

Methods

fit(dataset)

Computes the mean and variance and stores as a model to be used for later scaling.

Methods Documentation

fit(dataset: pyspark.rdd.RDD[VectorLike]) → StandardScalerModel¶

Computes the mean and variance and stores as a model to be used for later scaling.

Parameters

datasetpyspark.RDD: The data used to compute the mean and variance to build the transformation model.

Returns

StandardScalerModel

StandardScalerModel

HashingTF