Constructor
new SparkConf(loadDefaults)
Parameters:
Name |
Type |
Description |
loadDefaults |
boolean
|
|
- Source:
Methods
- Source:
Returns:
-
Type
-
module:eclairjs.SparkConf
contains(key) → {Promise.<boolean>}
Parameters:
Name |
Type |
Description |
key |
string
|
|
- Source:
Returns:
-
Type
-
Promise.<boolean>
get(key, defaultValueopt) → {Promise.<string>}
Parameters:
Name |
Type |
Attributes |
Description |
key |
string
|
|
|
defaultValue |
string
|
<optional>
|
|
- Source:
Returns:
-
Type
-
Promise.<string>
getAkkaConf() → {Array.<Tuple2>}
- Source:
Returns:
-
Type
-
Array.<Tuple2>
getAll() → {Array.<Tuple2>}
- Source:
Returns:
-
Type
-
Array.<Tuple2>
getAppId() → {Promise.<string>}
Returns the Spark application id, valid in the Driver after TaskScheduler registration and
from the start in the Executor.
- Source:
Returns:
-
Type
-
Promise.<string>
getAvroSchema() → {Map}
- Source:
Returns:
-
Type
-
Map
getBoolean(key, defaultValue) → {Promise.<boolean>}
Parameters:
Name |
Type |
Description |
key |
string
|
|
defaultValue |
boolean
|
|
- Source:
Returns:
-
Type
-
Promise.<boolean>
getDouble(key, defaultValue) → {Promise.<number>}
Parameters:
Name |
Type |
Description |
key |
string
|
|
defaultValue |
number
|
|
- Source:
Returns:
-
Type
-
Promise.<number>
getExecutorEnv() → {Array.<Tuple2>}
- Source:
Returns:
-
Type
-
Array.<Tuple2>
getInt(key, defaultValue) → {Promise.<number>}
Parameters:
Name |
Type |
Description |
key |
string
|
|
defaultValue |
number
|
|
- Source:
Returns:
-
Type
-
Promise.<number>
getLong(key, defaultValue) → {Promise.<number>}
Parameters:
Name |
Type |
Description |
key |
string
|
|
defaultValue |
number
|
|
- Source:
Returns:
-
Type
-
Promise.<number>
getOption(key) → {Promise.<string>}
Parameters:
Name |
Type |
Description |
key |
string
|
|
- Source:
Returns:
-
Type
-
Promise.<string>
getSizeAsBytes0(key) → {Promise.<number>}
Get a size parameter as bytes; throws a NoSuchElementException if it's not set. If no
suffix is provided then bytes are assumed.
Parameters:
Name |
Type |
Description |
key |
string
|
|
- Source:
Throws:
NoSuchElementException
Returns:
-
Type
-
Promise.<number>
getSizeAsBytes1(key, defaultValue) → {Promise.<number>}
Get a size parameter as bytes, falling back to a default if not set. If no
suffix is provided then bytes are assumed.
Parameters:
Name |
Type |
Description |
key |
string
|
|
defaultValue |
string
|
|
- Source:
Returns:
-
Type
-
Promise.<number>
getSizeAsBytes2(key, defaultValue) → {Promise.<number>}
Get a size parameter as bytes, falling back to a default if not set.
Parameters:
Name |
Type |
Description |
key |
string
|
|
defaultValue |
number
|
|
- Source:
Returns:
-
Type
-
Promise.<number>
getSizeAsGb(key, defaultValueopt) → {Promise.<number>}
Get a size parameter as Gibibytes, falling back to a default if not set. If no
suffix is provided then Gibibytes are assumed.
Parameters:
Name |
Type |
Attributes |
Description |
key |
string
|
|
|
defaultValue |
string
|
<optional>
|
|
- Source:
Returns:
-
Type
-
Promise.<number>
getSizeAsKb(key, defaultValueopt) → {Promise.<number>}
Get a size parameter as Kibibytes, falling back to a default if not set. If no
suffix is provided then Kibibytes are assumed.
Parameters:
Name |
Type |
Attributes |
Description |
key |
string
|
|
|
defaultValue |
string
|
<optional>
|
|
- Source:
Returns:
-
Type
-
Promise.<number>
getSizeAsMb(key, defaultValueopt) → {Promise.<number>}
Get a size parameter as Mebibytes, falling back to a default if not set. If no
suffix is provided then Mebibytes are assumed.
Parameters:
Name |
Type |
Attributes |
Description |
key |
string
|
|
|
defaultValue |
string
|
<optional>
|
|
- Source:
Returns:
-
Type
-
Promise.<number>
getTimeAsMs(key, defaultValueopt) → {Promise.<number>}
Get a time parameter as milliseconds, falling back to a default if not set. If no
suffix is provided then milliseconds are assumed.
Parameters:
Name |
Type |
Attributes |
Description |
key |
string
|
|
|
defaultValue |
string
|
<optional>
|
|
- Source:
Returns:
-
Type
-
Promise.<number>
getTimeAsSeconds(key, defaultValueopt) → {Promise.<number>}
Get a time parameter as seconds, falling back to a default if not set. If no
suffix is provided then seconds are assumed.
Parameters:
Name |
Type |
Attributes |
Description |
key |
string
|
|
|
defaultValue |
string
|
<optional>
|
|
- Source:
Returns:
-
Type
-
Promise.<number>
Use Kryo serialization and register the given set of Avro schemas so that the generic
record serializer can decrease network IO
Parameters:
Name |
Type |
Attributes |
Description |
schemas |
Schema
|
<repeatable>
|
|
- Source:
Returns:
-
Type
-
module:eclairjs.SparkConf
Use Kryo serialization and register the given set of classes with Kryo.
If called multiple times, this will append the classes from all calls together.
Parameters:
Name |
Type |
Description |
classes |
Array.<Class>
|
|
- Source:
Returns:
-
Type
-
module:eclairjs.SparkConf
Parameters:
Name |
Type |
Description |
key |
string
|
|
- Source:
Returns:
-
Type
-
module:eclairjs.SparkConf
Parameters:
Name |
Type |
Description |
key |
string
|
|
value |
string
|
|
- Source:
Returns:
-
Type
-
module:eclairjs.SparkConf
Parameters:
Name |
Type |
Description |
settings |
Traversable
|
|
- Source:
Returns:
-
Type
-
module:eclairjs.SparkConf
Set a name for your application.
Parameters:
Name |
Type |
Description |
appName |
string
|
|
- Source:
Returns:
-
Type
-
module:eclairjs.SparkConf
Set an environment variable to be used when launching executors for this application.
These variables are stored as properties of the form spark.executorEnv.VAR_NAME
(for example spark.executorEnv.PATH) but this method makes them easier to set.
Parameters:
Name |
Type |
Description |
variable |
string
|
|
value |
string
|
|
- Source:
Returns:
-
Type
-
module:eclairjs.SparkConf
Set multiple environment variables to be used when launching executors.
These variables are stored as properties of the form spark.executorEnv.VAR_NAME
(for example spark.executorEnv.PATH) but this method makes them easier to set.
Parameters:
Name |
Type |
Description |
variables |
Array.<Tuple2>
|
|
- Source:
Returns:
-
Type
-
module:eclairjs.SparkConf
Set multiple environment variables to be used when launching executors.
(Java-friendly version.)
Parameters:
Name |
Type |
Description |
variables |
Array.<Tuple2>
|
|
- Source:
Returns:
-
Type
-
module:eclairjs.SparkConf
Parameters:
Name |
Type |
Description |
key |
string
|
|
value |
string
|
|
- Source:
Returns:
-
Type
-
module:eclairjs.SparkConf
Parameters:
Name |
Type |
Description |
jars |
Array.<string>
|
|
- Source:
Returns:
-
Type
-
module:eclairjs.SparkConf
Parameters:
Name |
Type |
Description |
jars |
Array.<string>
|
|
- Source:
Returns:
-
Type
-
module:eclairjs.SparkConf
The master URL to connect to, such as "local" to run locally with one thread, "local[4]" to
run locally with 4 cores, or "spark://master:7077" to run on a Spark standalone cluster.
Parameters:
Name |
Type |
Description |
master |
string
|
|
- Source:
Returns:
-
Type
-
module:eclairjs.SparkConf
Set the location where Spark is installed on worker nodes.
Parameters:
Name |
Type |
Description |
home |
string
|
|
- Source:
Returns:
-
Type
-
module:eclairjs.SparkConf
toDebugString() → {Promise.<string>}
Return a string listing all keys and values, one per line. This is useful to print the
configuration out for debugging.
- Source:
Returns:
-
Type
-
Promise.<string>