Class: SparkConf

eclairjs.SparkConf

Configuration for a Spark application. Used to set various Spark parameters as key-value pairs. Most of the time, you would create a SparkConf object with new SparkConf(), parameters you set directly on the SparkConf object take priority over system properties. All setter methods in this class support chaining. For example, you can write new SparkConf().setMaster("local").setAppName("My app"). Note that once a SparkConf object is passed to Spark, it is cloned and can no longer be modified by the user. Spark does not support modifying the configuration at runtime.

Constructor

new SparkConf(loadDefaults)

Parameters:
Name Type Description
loadDefaults boolean
Source:

Methods

clone() → {module:eclairjs.SparkConf}

Source:
Returns:
Type
module:eclairjs.SparkConf

contains(key) → {Promise.<boolean>}

Parameters:
Name Type Description
key string
Source:
Returns:
Type
Promise.<boolean>

get(key, defaultValueopt) → {Promise.<string>}

Parameters:
Name Type Attributes Description
key string
defaultValue string <optional>
Source:
Returns:
Type
Promise.<string>

getAkkaConf() → {Array.<Tuple2>}

Source:
Returns:
Type
Array.<Tuple2>

getAll() → {Array.<Tuple2>}

Source:
Returns:
Type
Array.<Tuple2>

getAppId() → {Promise.<string>}

Returns the Spark application id, valid in the Driver after TaskScheduler registration and from the start in the Executor.
Source:
Returns:
Type
Promise.<string>

getAvroSchema() → {Map}

Source:
Returns:
Type
Map

getBoolean(key, defaultValue) → {Promise.<boolean>}

Parameters:
Name Type Description
key string
defaultValue boolean
Source:
Returns:
Type
Promise.<boolean>

getDouble(key, defaultValue) → {Promise.<number>}

Parameters:
Name Type Description
key string
defaultValue number
Source:
Returns:
Type
Promise.<number>

getExecutorEnv() → {Array.<Tuple2>}

Source:
Returns:
Type
Array.<Tuple2>

getInt(key, defaultValue) → {Promise.<number>}

Parameters:
Name Type Description
key string
defaultValue number
Source:
Returns:
Type
Promise.<number>

getLong(key, defaultValue) → {Promise.<number>}

Parameters:
Name Type Description
key string
defaultValue number
Source:
Returns:
Type
Promise.<number>

getOption(key) → {Promise.<string>}

Parameters:
Name Type Description
key string
Source:
Returns:
Type
Promise.<string>

getSizeAsBytes0(key) → {Promise.<number>}

Get a size parameter as bytes; throws a NoSuchElementException if it's not set. If no suffix is provided then bytes are assumed.
Parameters:
Name Type Description
key string
Source:
Throws:
NoSuchElementException
Returns:
Type
Promise.<number>

getSizeAsBytes1(key, defaultValue) → {Promise.<number>}

Get a size parameter as bytes, falling back to a default if not set. If no suffix is provided then bytes are assumed.
Parameters:
Name Type Description
key string
defaultValue string
Source:
Returns:
Type
Promise.<number>

getSizeAsBytes2(key, defaultValue) → {Promise.<number>}

Get a size parameter as bytes, falling back to a default if not set.
Parameters:
Name Type Description
key string
defaultValue number
Source:
Returns:
Type
Promise.<number>

getSizeAsGb(key, defaultValueopt) → {Promise.<number>}

Get a size parameter as Gibibytes, falling back to a default if not set. If no suffix is provided then Gibibytes are assumed.
Parameters:
Name Type Attributes Description
key string
defaultValue string <optional>
Source:
Returns:
Type
Promise.<number>

getSizeAsKb(key, defaultValueopt) → {Promise.<number>}

Get a size parameter as Kibibytes, falling back to a default if not set. If no suffix is provided then Kibibytes are assumed.
Parameters:
Name Type Attributes Description
key string
defaultValue string <optional>
Source:
Returns:
Type
Promise.<number>

getSizeAsMb(key, defaultValueopt) → {Promise.<number>}

Get a size parameter as Mebibytes, falling back to a default if not set. If no suffix is provided then Mebibytes are assumed.
Parameters:
Name Type Attributes Description
key string
defaultValue string <optional>
Source:
Returns:
Type
Promise.<number>

getTimeAsMs(key, defaultValueopt) → {Promise.<number>}

Get a time parameter as milliseconds, falling back to a default if not set. If no suffix is provided then milliseconds are assumed.
Parameters:
Name Type Attributes Description
key string
defaultValue string <optional>
Source:
Returns:
Type
Promise.<number>

getTimeAsSeconds(key, defaultValueopt) → {Promise.<number>}

Get a time parameter as seconds, falling back to a default if not set. If no suffix is provided then seconds are assumed.
Parameters:
Name Type Attributes Description
key string
defaultValue string <optional>
Source:
Returns:
Type
Promise.<number>

registerAvroSchemas(…schemas) → {module:eclairjs.SparkConf}

Use Kryo serialization and register the given set of Avro schemas so that the generic record serializer can decrease network IO
Parameters:
Name Type Attributes Description
schemas Schema <repeatable>
Source:
Returns:
Type
module:eclairjs.SparkConf

registerKryoClasses(classes) → {module:eclairjs.SparkConf}

Use Kryo serialization and register the given set of classes with Kryo. If called multiple times, this will append the classes from all calls together.
Parameters:
Name Type Description
classes Array.<Class>
Source:
Returns:
Type
module:eclairjs.SparkConf

remove(key) → {module:eclairjs.SparkConf}

Parameters:
Name Type Description
key string
Source:
Returns:
Type
module:eclairjs.SparkConf

set(key, value) → {module:eclairjs.SparkConf}

Parameters:
Name Type Description
key string
value string
Source:
Returns:
Type
module:eclairjs.SparkConf

setAll(settings) → {module:eclairjs.SparkConf}

Parameters:
Name Type Description
settings Traversable
Source:
Returns:
Type
module:eclairjs.SparkConf

setAppName(appName) → {module:eclairjs.SparkConf}

Set a name for your application.
Parameters:
Name Type Description
appName string
Source:
Returns:
Type
module:eclairjs.SparkConf

setExecutorEnv0(variable, value) → {module:eclairjs.SparkConf}

Set an environment variable to be used when launching executors for this application. These variables are stored as properties of the form spark.executorEnv.VAR_NAME (for example spark.executorEnv.PATH) but this method makes them easier to set.
Parameters:
Name Type Description
variable string
value string
Source:
Returns:
Type
module:eclairjs.SparkConf

setExecutorEnv1(variables) → {module:eclairjs.SparkConf}

Set multiple environment variables to be used when launching executors. These variables are stored as properties of the form spark.executorEnv.VAR_NAME (for example spark.executorEnv.PATH) but this method makes them easier to set.
Parameters:
Name Type Description
variables Array.<Tuple2>
Source:
Returns:
Type
module:eclairjs.SparkConf

setExecutorEnv2(variables) → {module:eclairjs.SparkConf}

Set multiple environment variables to be used when launching executors. (Java-friendly version.)
Parameters:
Name Type Description
variables Array.<Tuple2>
Source:
Returns:
Type
module:eclairjs.SparkConf

setIfMissing(key, value) → {module:eclairjs.SparkConf}

Parameters:
Name Type Description
key string
value string
Source:
Returns:
Type
module:eclairjs.SparkConf

setJars(jars) → {module:eclairjs.SparkConf}

Parameters:
Name Type Description
jars Array.<string>
Source:
Returns:
Type
module:eclairjs.SparkConf

setJars(jars) → {module:eclairjs.SparkConf}

Parameters:
Name Type Description
jars Array.<string>
Source:
Returns:
Type
module:eclairjs.SparkConf

setMaster(master) → {module:eclairjs.SparkConf}

The master URL to connect to, such as "local" to run locally with one thread, "local[4]" to run locally with 4 cores, or "spark://master:7077" to run on a Spark standalone cluster.
Parameters:
Name Type Description
master string
Source:
Returns:
Type
module:eclairjs.SparkConf

setSparkHome(home) → {module:eclairjs.SparkConf}

Set the location where Spark is installed on worker nodes.
Parameters:
Name Type Description
home string
Source:
Returns:
Type
module:eclairjs.SparkConf

toDebugString() → {Promise.<string>}

Return a string listing all keys and values, one per line. This is useful to print the configuration out for debugging.
Source:
Returns:
Type
Promise.<string>