Class: StreamingContext

eclairjs/streaming.StreamingContext

Main entry point for Spark Streaming functionality. It provides methods used to create DStreams from various input sources. It can be either created by providing a Spark master URL and an appName, or from a org.apache.spark.SparkConf configuration (see core Spark documentation), or from an existing org.apache.spark.SparkContext. The associated SparkContext can be accessed using context.sparkContext. After creating and transforming DStreams, the streaming computation can be started and stopped using context.start() and context.stop(), respectively. context.awaitTermination() allows the current thread to wait for the termination of the context by stop() or by an exception.

Constructor

new StreamingContext(sparkContext, duration)

Parameters:
Name Type Description
sparkContext SparkContex
duration module:eclairjs/streaming.Duration
Source:

Methods

awaitTermination()

Wait for the execution to stop
Source:

awaitTerminationOrTimeout(millis) → {boolean}

Wait for the execution to stop, or timeout
Parameters:
Name Type Description
millis long
Source:
Returns:
Type
boolean

queueStream(queue, oneAtATimeopt, defaultRDDopt) → {module:eclairjs/streaming/dstream.DStream}

Create an input stream from an queue of RDDs. In each batch, it will process either one or all of the RDDs returned by the queue. NOTE: 1. Changes to the queue after the stream is created will not be recognized. 2. Arbitrary RDDs can be added to `queueStream`, there is no way to recover data of those RDDs, so `queueStream` doesn't support checkpointing.
Parameters:
Name Type Attributes Default Description
queue Array.<module:eclairjs.RDD> Queue of RDDs
oneAtATime boolean <optional>
 true Whether only one RDD should be consumed from the queue in every interval
defaultRDD module:eclairjs.RDD <optional>
 Default RDD is returned by the DStream when the queue is empty
Source:
Returns:
Type
module:eclairjs/streaming/dstream.DStream

socketTextStream(host, port) → {module:eclairjs/streaming/dstream.DStream}

Create a input stream from TCP source hostname:port.
Parameters:
Name Type Description
host string
port string
Source:
Returns:
Type
module:eclairjs/streaming/dstream.DStream

sparkContext() → {module:eclairjs.SparkContext}

The underlying SparkContext
Source:
Returns:
Type
module:eclairjs.SparkContext

start()

Start the execution of the streams.
Source:

stop()

Stops the execution of the streams.
Source: