package util
Spark utilities.
- Alphabetic
- By Inheritance
- util
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Type Members
-
abstract
class
AccumulatorV2[IN, OUT] extends Serializable
The base class for accumulators, that can accumulate inputs of type
IN
, and produce output of typeOUT
.The base class for accumulators, that can accumulate inputs of type
IN
, and produce output of typeOUT
.OUT
should be a type that can be read atomically (e.g., Int, Long), or thread-safely (e.g., synchronized collections) because it will be read from other threads. - class ChildFirstURLClassLoader extends MutableURLClassLoader
-
class
CollectionAccumulator[T] extends AccumulatorV2[T, List[T]]
An accumulator for collecting a list of elements.
An accumulator for collecting a list of elements.
- Since
2.0.0
-
class
DoubleAccumulator extends AccumulatorV2[Double, Double]
An accumulator for computing sum, count, and averages for double precision floating numbers.
An accumulator for computing sum, count, and averages for double precision floating numbers.
- Since
2.0.0
-
class
EnumUtil extends AnyRef
- Annotations
- @Private()
-
class
LongAccumulator extends AccumulatorV2[Long, Long]
An accumulator for computing sum, count, and average of 64-bit integers.
An accumulator for computing sum, count, and average of 64-bit integers.
- Since
2.0.0
-
case class
MutablePair[T1, T2](_1: T1, _2: T2) extends Product2[T1, T2] with Product with Serializable
:: DeveloperApi :: A tuple of 2 elements.
:: DeveloperApi :: A tuple of 2 elements. This can be used as an alternative to Scala's Tuple2 when we want to minimize object allocation.
- _1
Element 1 of this MutablePair
- _2
Element 2 of this MutablePair
- Annotations
- @DeveloperApi()
- class MutableURLClassLoader extends URLClassLoader
- class ParentClassLoader extends ClassLoader
-
class
SerializableConfiguration extends Serializable
Hadoop configuration but serializable.
Hadoop configuration but serializable. Use
value
to access the Hadoop configuration.- Annotations
- @DeveloperApi() @Unstable()
-
class
StatCounter extends Serializable
A class for tracking the statistics of a set of numbers (count, mean and variance) in a numerically robust way.
A class for tracking the statistics of a set of numbers (count, mean and variance) in a numerically robust way. Includes support for merging two StatCounters. Based on Welford and Chan's algorithms for running variance.
-
trait
TaskCompletionListener extends EventListener
:: DeveloperApi ::
:: DeveloperApi ::
Listener providing a callback function to invoke when a task's execution completes.
- Annotations
- @DeveloperApi()
-
trait
TaskFailureListener extends EventListener
:: DeveloperApi ::
:: DeveloperApi ::
Listener providing a callback function to invoke when a task's execution encounters an error. Operations defined here must be idempotent, as
onTaskFailure
can be called multiple times.- Annotations
- @DeveloperApi()
Value Members
-
object
SizeEstimator extends Logging
:: DeveloperApi :: Estimates the sizes of Java objects (number of bytes of memory they occupy), for use in memory-aware caches.
:: DeveloperApi :: Estimates the sizes of Java objects (number of bytes of memory they occupy), for use in memory-aware caches.
Based on the following JavaWorld article: http://www.javaworld.com/javaworld/javaqa/2003-12/02-qa-1226-sizeof.html
- Annotations
- @DeveloperApi()
- object StatCounter extends Serializable