pyspark.pandas.DataFrame.shift¶
-
DataFrame.shift(periods: int = 1, fill_value: Optional[Any] = None) → pyspark.pandas.frame.DataFrame¶ Shift DataFrame by desired number of periods.
Note
the current implementation of shift uses Spark’s Window without specifying partition specification. This leads to move all data into single partition in single machine and could cause serious performance degradation. Avoid this method against very large dataset.
- Parameters
- periodsint
Number of periods to shift. Can be positive or negative.
- fill_valueobject, optional
The scalar value to use for newly introduced missing values. The default depends on the dtype of self. For numeric data, np.nan is used.
- Returns
- Copy of input DataFrame, shifted.
Examples
>>> df = ps.DataFrame({'Col1': [10, 20, 15, 30, 45], ... 'Col2': [13, 23, 18, 33, 48], ... 'Col3': [17, 27, 22, 37, 52]}, ... columns=['Col1', 'Col2', 'Col3'])
>>> df.shift(periods=3) Col1 Col2 Col3 0 NaN NaN NaN 1 NaN NaN NaN 2 NaN NaN NaN 3 10.0 13.0 17.0 4 20.0 23.0 27.0
>>> df.shift(periods=3, fill_value=0) Col1 Col2 Col3 0 0 0 0 1 0 0 0 2 0 0 0 3 10 13 17 4 20 23 27