pyspark.RDD.mapPartitionsWithSplit¶
-
RDD.mapPartitionsWithSplit(f: Callable[[int, Iterable[T]], Iterable[U]], preservesPartitioning: bool = False) → pyspark.rdd.RDD[U]¶ Return a new RDD by applying a function to each partition of this RDD, while tracking the index of the original partition.
use
RDD.mapPartitionsWithIndex()instead.Examples
>>> rdd = sc.parallelize([1, 2, 3, 4], 4) >>> def f(splitIndex, iterator): yield splitIndex >>> rdd.mapPartitionsWithSplit(f).sum() 6