pyspark.BarrierTaskContext¶
-
class
pyspark.BarrierTaskContext¶ A
TaskContextwith extra contextual info and tooling for tasks in a barrier stage. UseBarrierTaskContext.get()to obtain the barrier context for a running barrier task.Notes
This API is experimental
Methods
allGather([message])This function blocks until all tasks in the same stage have reached this routine.
”
barrier()Sets a global barrier and waits until all tasks in this stage hit this barrier.
get()Return the currently active
BarrierTaskContext.getLocalProperty(key)Get a local property set upstream in the driver, or None if it is missing.
Returns
BarrierTaskInfofor all tasks in this barrier stage, ordered by partition ID.The ID of the RDD partition that is computed by this task.
Resources allocated to the task.
stageId()The ID of the stage that this task belong to.
An ID that is unique to this task attempt (within the same SparkContext, no two task attempts will share the same attempt ID).