Class: RDD

eclairjs/rdd.RDD

A Resilient Distributed Dataset (RDD), the basic abstraction in Spark. Represents an immutable, partitioned collection of elements that can be operated on in parallel.

Constructor

new RDD()

Source:

Methods

aggregate(zeroValue, func1, func2, bindArgs1opt, bindArgs2opt) → {object}

Aggregate the elements of each partition, and then the results for all the partitions, using given combine functions and a neutral "zero value". This function can return a different result type, U, than the type of this RDD, T. Thus, we need one operation for merging a T into an U and one operation for merging two U's, as in scala.TraversableOnce. Both of these functions are allowed to modify and return their first argument instead of creating a new U to avoid memory allocation.
Parameters:
Name Type Attributes Description
zeroValue module:eclairjs/rdd.RDD (undocumented)
func1 function seqOp - (undocumented) Function with two parameters
func2 function combOp - (undocumented) Function with two parameters
bindArgs1 Array.<Object> <optional>
 array whose values will be added to func1's argument list.
bindArgs2 Array.<Object> <optional>
 array whose values will be added to func2's argument list.
Source:
Returns:
Type
object

cache() → {module:eclairjs/rdd.RDD}

Persist this RDD with the default storage level (`MEMORY_ONLY`).
Source:
Returns:
Type
module:eclairjs/rdd.RDD

collect() → {Promise.<Array>}

Asynchronously returns all elements of the RDD.
Source:
Returns:
A Promise that resolves to an array containing all elements in the RDD.
Type
Promise.<Array>

count() → {Promise.<Number>}

Asynchronously returns the number of elements in the RDD.
Source:
Returns:
A Promise that resolves to the number of elements in the RDD.
Type
Promise.<Number>

filter(func, bindArgsopt) → {module:eclairjs/rdd.RDD}

Return a new RDD containing only the elements that satisfy a predicate.
Parameters:
Name Type Attributes Description
func function (undocumented) Function with one parameter
bindArgs Array.<Object> <optional>
 array whose values will be added to func's argument list.
Source:
Returns:
Type
module:eclairjs/rdd.RDD

flatMap(func, bindArgsopt) → {module:eclairjs/rdd.RDD}

Return a new RDD by first applying a function to all elements of this RDD, and then flattening the results.
Parameters:
Name Type Attributes Description
func function (undocumented) - Function with one parameter
bindArgs Array.<Object> <optional>
 array whose values will be added to func's argument list.
Source:
Returns:
Type
module:eclairjs/rdd.RDD

foreach(func, bindArgsopt) → {void}

Applies a function to all elements of this RDD.
Parameters:
Name Type Attributes Description
func function Function with one parameter that returns void
bindArgs Array.<Object> <optional>
 array whose values will be added to func's argument list.
Source:
Returns:
Type
void
Example
rdd3.foreach(function(record) {
   var connection = createNewConnection()
   connection.send(record);
   connection.close()
});

groupBy(func, numPartitionsopt, partitioneropt, bindArgsopt) → {module:eclairjs/rdd.RDD}

Return an RDD of grouped items. Each group consists of a key and a sequence of elements mapping to that key. The ordering of elements within each group is not guaranteed, and may even differ each time the resulting RDD is evaluated. Note: This operation may be very expensive. If you are grouping in order to perform an aggregation (such as a sum or average) over each key, using aggregateByKey or reduceByKey will provide much better performance.
Parameters:
Name Type Attributes Description
func function (undocumented) Function with one parameter
numPartitions number <optional>
 How many partitions to use in the resulting RDD (if non-zero partitioner is ignored)
partitioner Partitioner <optional>
 Partitioner to use for the resulting RDD
bindArgs Array.<Object> <optional>
 array whose values will be added to func's argument list.
Source:
Returns:
Type
module:eclairjs/rdd.RDD

join(other, optionanl) → {module:eclairjs/rdd.RDD}

Return an RDD containing all pairs of elements with matching keys in `this` and `other`. Each pair of elements will be returned as a (k, (v1, v2)) tuple, where (k, v1) is in `this` and (k, v2) is in `other`. Performs a hash join across the cluster.
Parameters:
Name Type Description
other module:eclairjs/rdd.RDD
optionanl number
Source:
Returns:
Type
module:eclairjs/rdd.RDD

map(func, bindArgsopt) → {module:eclairjs/rdd.RDD}

Return a new RDD by applying a function to all elements of this RDD.
Parameters:
Name Type Attributes Description
func function (undocumented) Function with one parameter
bindArgs Array.<Object> <optional>
 array whose values will be added to func's argument list.
Source:
Returns:
Type
module:eclairjs/rdd.RDD

mapToPair((function), bindArgsopt) → {module:eclairjs/rdd.PairRDD}

Return a new RDD by applying a function to all elements of this RDD.
Parameters:
Name Type Attributes Description
(function) func - (undocumented) Function with one parameter that returns tuple
bindArgs Array.<Object> <optional>
 array whose values will be added to func's argument list.
Source:
Returns:
Type
module:eclairjs/rdd.PairRDD

mapValues(func, bindArgsopt) → {module:eclairjs/rdd.RDD}

Pass each value in the key-value pair RDD through a map function without changing the keys; this also retains the original RDD's partitioning.
Parameters:
Name Type Attributes Description
func func
bindArgs Array.<Object> <optional>
 array whose values will be added to func's argument list.
Source:
Returns:
Type
module:eclairjs/rdd.RDD

reduce(bindArgsopt) → {object}

Reduces the elements of this RDD using the specified commutative and associative binary operator. {function} func - (undocumented) Function with two parameters
Parameters:
Name Type Attributes Description
bindArgs Array.<Object> <optional>
 array whose values will be added to func's argument list.
Source:
Returns:
Type
object

reduceByKey(func, bindArgsopt) → {module:eclairjs/rdd.RDD}

Reduces the elements of this RDD using the specified function.
Parameters:
Name Type Attributes Description
func function
bindArgs Array.<Object> <optional>
 array whose values will be added to func's argument list.
Source:
Returns:
Type
module:eclairjs/rdd.RDD

sample(withReplacement, fraction, seed) → {module:eclairjs/rdd.RDD}

Returns a new DataFrame by sampling a fraction of rows, using a random seed.
Parameters:
Name Type Description
withReplacement boolean
fraction float
seed integer Optional
Source:
Returns:
Type
module:eclairjs/rdd.RDD

saveAsObjectFile(path, overwriteopt) → {Promise.<void>}

Save this RDD as a SequenceFile of serialized objects.
Parameters:
Name Type Attributes Description
path string
overwrite boolean <optional>
 defaults to false, if true overwrites file if it exists
Source:
Returns:
Type
Promise.<void>

saveAsTextFile(path, overwriteopt) → {Promise.<void>}

Save this RDD as a text file, using string representations of elements.
Parameters:
Name Type Attributes Description
path string
overwrite boolean <optional>
 defaults to false, if true overwrites file if it exists
Source:
Returns:
Type
Promise.<void>

sortByKey(ascending) → {module:eclairjs/rdd.RDD}

Return this RDD sorted by the given key function.
Parameters:
Name Type Description
ascending Boolean
Source:
Returns:
Type
module:eclairjs/rdd.RDD

subtract(other, numPartitionsopt, popt) → {module:eclairjs/rdd.RDD}

Return an RDD with the elements from `this` that are not in `other`.
Parameters:
Name Type Attributes Description
other module:eclairjs/rdd.RDD
numPartitions int <optional>
p Partition <optional>
 ignored if numPartitions is non-zero)
Source:
Returns:
Type
module:eclairjs/rdd.RDD

take(num) → {Promise.<Array>}

Asynchronously returns the first num elements in this RDD.
Parameters:
Name Type Description
num Number
Source:
Returns:
A Promise that resolves to an array containing the first num elements in this RDD.
Type
Promise.<Array>

takeOrdered(num, func, bindArgsopt) → {Promise.<Array>}

Asynchronously returns the first k (smallest) elements from this RDD as defined by the specified implicit Ordering[T] and maintains the ordering. This does the opposite of top.
Parameters:
Name Type Attributes Description
num Number
func function (undocumented) Function with one parameter
bindArgs Array.<Object> <optional>
 array whose values will be added to func's argument list.
Source:
Returns:
A Promise that resolves to an array containing the first num elements in this RDD.
Type
Promise.<Array>

takeSample(withReplacement, num, seed) → {Promise.<Array>}

Return a fixed-size sampled subset of this RDD in an array
Parameters:
Name Type Description
withReplacement boolean whether sampling is done with replacement
num number size of the returned sample
seed number seed for the random number generator
Source:
Returns:
A Promise that resolves to an array containing the specified number of elements in this RDD.
Type
Promise.<Array>

toArray() → {Promise.<Array>}

Return an array that contains all of the elements in this RDD.
Source:
Returns:
A Promise that resolves to an array containing all elements in this RDD.
Type
Promise.<Array>

union(other) → {module:eclairjs/rdd.RDD}

Return the union of this RDD and another one. Any identical elements will appear multiple times (use `.distinct()` to eliminate them).
Parameters:
Name Type Description
other module:eclairjs/rdd.RDD
Source:
Returns:
Type
module:eclairjs/rdd.RDD

values() → {module:eclairjs/rdd.RDD}

Return an RDD with the values of each tuple.
Source:
Returns:
Type
module:eclairjs/rdd.RDD

zipWithIndex() → {module:eclairjs/rdd.RDD}

Zips this RDD with its element indices. The ordering is first based on the partition index and then the ordering of items within each partition. So the first item in the first partition gets index 0, and the last item in the last partition receives the largest index. This is similar to Scala's zipWithIndex but it uses Long instead of Int as the index type. This method needs to trigger a spark job when this RDD contains more than one partitions. Note that some RDDs, such as those returned by groupBy(), do not guarantee order of elements in a partition. The index assigned to each element is therefore not guaranteed, and may even change if the RDD is reevaluated. If a fixed ordering is required to guarantee the same index assignments, you should sort the RDD with sortByKey() or save it to a file.
Source:
Returns:
Type
module:eclairjs/rdd.RDD