pyspark.pandas.Series.compare¶
-
Series.compare(other: pyspark.pandas.series.Series, keep_shape: bool = False, keep_equal: bool = False) → pyspark.pandas.frame.DataFrame¶ Compare to another Series and show the differences.
Note
This API is slightly different from pandas when indexes from both Series are not identical and config ‘compute.eager_check’ is False. pandas raises an exception; however, pandas-on-Spark just proceeds and performs by ignoring mismatches.
>>> psser1 = ps.Series([1, 2, 3, 4, 5], index=pd.Index([1, 2, 3, 4, 5])) >>> psser2 = ps.Series([1, 2, 3, 4, 5], index=pd.Index([1, 2, 4, 3, 6])) >>> psser1.compare(psser2) ... ValueError: Can only compare identically-labeled Series objects
>>> with ps.option_context("compute.eager_check", False): ... psser1.compare(psser2) ... self other 3 3.0 4.0 4 4.0 3.0 5 5.0 NaN 6 NaN 5.0
- Parameters
- otherSeries
Object to compare with.
- keep_shapebool, default False
If true, all rows and columns are kept. Otherwise, only the ones with different values are kept.
- keep_equalbool, default False
If true, the result keeps values that are equal. Otherwise, equal values are shown as NaNs.
- Returns
- DataFrame
Notes
Matching NaNs will not appear as a difference.
Examples
>>> from pyspark.pandas.config import set_option, reset_option >>> set_option("compute.ops_on_diff_frames", True) >>> s1 = ps.Series(["a", "b", "c", "d", "e"]) >>> s2 = ps.Series(["a", "a", "c", "b", "e"])
Align the differences on columns
>>> s1.compare(s2).sort_index() self other 1 b a 3 d b
Keep all original rows
>>> s1.compare(s2, keep_shape=True).sort_index() self other 0 None None 1 b a 2 None None 3 d b 4 None None
Keep all original rows and also all original values
>>> s1.compare(s2, keep_shape=True, keep_equal=True).sort_index() self other 0 a a 1 b a 2 c c 3 d b 4 e e
>>> reset_option("compute.ops_on_diff_frames")