Class: StreamingContext

eclairjs/streaming. StreamingContext

Main entry point for Spark Streaming functionality. It provides methods used to create DStreams from various input sources. It can be either created by providing a Spark master URL and an appName, or from a org.apache.spark.SparkConf configuration (see core Spark documentation), or from an existing org.apache.spark.SparkContext. The associated SparkContext can be accessed using context.sparkContext. After creating and transforming DStreams, the streaming computation can be started and stopped using context.start() and context.stop(), respectively. context.awaitTermination() allows the current thread to wait for the termination of the context by stop() or by an exception.

Constructor

new StreamingContext(sparkContext, duration)

Parameters:
Name Type Description
sparkContext SparkContex
duration module:eclairjs/streaming.Duration
Source:

Methods

(static) getOrCreate(checkpointPath, creatingFunc) → {JavaStreamingContext}

Either recreate a StreamingContext from checkpoint data or create a new StreamingContext. If checkpoint data exists in the provided `checkpointPath`, then StreamingContext will be recreated from the checkpoint data. If the data does not exist, then the provided factory will be used to create a JavaStreamingContext.
Parameters:
Name Type Description
checkpointPath string Checkpoint directory used in an earlier JavaStreamingContext program
creatingFunc func Function to create a new JavaStreamingContext
Source:
Returns:
Type
JavaStreamingContext

awaitTermination()

Wait for the execution to stop
Source:

awaitTerminationOrTimeout(millis) → {boolean}

Wait for the execution to stop, or timeout
Parameters:
Name Type Description
millis long
Source:
Returns:
Type
boolean

checkpoint(directory)

Sets the context to periodically checkpoint the DStream operations for master fault-tolerance. The graph will be checkpointed every batch interval.
Parameters:
Name Type Description
directory string HDFS-compatible directory where the checkpoint data will be reliably stored
Source:

queueStream(queue, oneAtATimeopt, defaultRDDopt) → {module:eclairjs/streaming/dstream.DStream}

Create an input stream from an queue of RDDs. In each batch, it will process either one or all of the RDDs returned by the queue. NOTE: 1. Changes to the queue after the stream is created will not be recognized. 2. Arbitrary RDDs can be added to `queueStream`, there is no way to recover data of those RDDs, so `queueStream` doesn't support checkpointing.
Parameters:
Name Type Attributes Default Description
queue Array.<module:eclairjs.RDD> Queue of RDDs
oneAtATime boolean <optional>
 true Whether only one RDD should be consumed from the queue in every interval
defaultRDD module:eclairjs.RDD <optional>
 Default RDD is returned by the DStream when the queue is empty
Source:
Returns:
Type
module:eclairjs/streaming/dstream.DStream

socketTextStream(host, port) → {module:eclairjs/streaming/dstream.DStream}

Create a input stream from TCP source hostname:port.
Parameters:
Name Type Description
host string
port string
Source:
Returns:
Type
module:eclairjs/streaming/dstream.DStream

sparkContext() → {module:eclairjs.SparkContext}

The underlying SparkContext
Source:
Returns:
Type
module:eclairjs.SparkContext

start()

Start the execution of the streams.
Source:

stop()

Stops the execution of the streams.
Source: