new DataFrameWriter()
:: Experimental ::
Interface used to write a DataFrame to external storage systems (e.g. file systems,
key-value stores, etc). Use write to access this.
- Since:
- EclairJS 0.1 Spark 1.4.0
- Source:
Methods
format(source) → {module:eclairjs/sql.DataFrameWriter}
Specifies the underlying output data source. Built-in options include "parquet", "json", etc.
Parameters:
Name | Type | Description |
---|---|---|
source |
string |
- Since:
- EclairJS 0.1 Spark 1.4.0
- Source:
Returns:
insertInto(tableName) → {Promise.<Void>}
Inserts the content of the DataFrame to the specified table. It requires that
the schema of the DataFrame is the same as the schema of the table.
Because it inserts data to an existing table, format or options will be ignored.
Parameters:
Name | Type | Description |
---|---|---|
tableName |
string |
- Since:
- EclairJS 0.1 Spark 1.4.0
- Source:
Returns:
A Promise that resolves to nothing.
- Type
- Promise.<Void>
jdbc(url, table, connectionProperties) → {Promise.<Void>}
Saves the content of the DataFrame to a external database table via JDBC. In the case the
table already exists in the external database, behavior of this function depends on the
save mode, specified by the `mode` function (default to throwing an exception).
Don't create too many partitions in parallel on a large cluster; otherwise Spark might crash
your external database systems.
Parameters:
Name | Type | Description |
---|---|---|
url |
string | JDBC database url of the form `jdbc:subprotocol:subname` |
table |
string | Name of the table in the external database. |
connectionProperties |
object | JDBC database connection arguments, a list of arbitrary string tag/value. Normally at least a "user" and "password" property should be included. |
- Source:
Returns:
A Promise that resolves to nothing.
- Type
- Promise.<Void>
json(path) → {Promise.<Void>}
Saves the content of the DataFrame in JSON format at the specified path.
This is equivalent to:
Parameters:
Name | Type | Description |
---|---|---|
path |
string |
- Since:
- EclairJS 0.1 Spark 1.4.0
- Source:
Returns:
A Promise that resolves to nothing.
- Type
- Promise.<Void>
Example
format("json").save(path)
mode(saveMode) → {module:eclairjs/sql.DataFrameWriter}
Specifies the behavior when data or table already exists. Options include:
- `overwrite`: overwrite the existing data.
- `append`: append the data.
- `ignore`: ignore the operation (i.e. no-op).
- `error`: default option, throw an exception at runtime.
Parameters:
Name | Type | Description |
---|---|---|
saveMode |
string |
- Since:
- EclairJS 0.1 Spark 1.4.0
- Source:
Returns:
option(key, value) → {module:eclairjs/sql.DataFrameWriter}
Adds an input option for the underlying data source.
Parameters:
Name | Type | Description |
---|---|---|
key |
string | |
value |
string |
- Since:
- EclairJS 0.1 Spark 1.4.0
- Source:
Returns:
options(options) → {module:eclairjs/sql.DataFrameWriter}
Adds input options for the underlying data source.
Parameters:
Name | Type | Description |
---|---|---|
options |
Map |
- Since:
- EclairJS 0.1 Spark 1.4.0
- Source:
Returns:
orc(path) → {Promise.<Void>}
Saves the content of the DataFrame in ORC format at the specified path.
This is equivalent to:
Parameters:
Name | Type | Description |
---|---|---|
path |
string |
- Since:
- EclairJS 0.1 Spark 1.5.0
- Source:
Returns:
A Promise that resolves to nothing.
- Type
- Promise.<Void>
Example
format("orc").save(path)
parquet(path) → {Promise.<Void>}
Saves the content of the DataFrame in Parquet format at the specified path.
This is equivalent to:
Parameters:
Name | Type | Description |
---|---|---|
path |
string |
- Since:
- EclairJS 0.1 Spark 1.4.0
- Source:
Returns:
A Promise that resolves to nothing.
- Type
- Promise.<Void>
Example
format("parquet").save(path)
partitionBy() → {module:eclairjs/sql.DataFrameWriter}
Partitions the output by the given columns on the file system. If specified, the output is
laid out on the file system similar to Hive's partitioning scheme.
This is only applicable for Parquet at the moment.
Parameters:
Name | Type | Description |
---|---|---|
colName,...colName |
string |
- Since:
- EclairJS 0.1 Spark 1.4.0
- Source:
Returns:
save(pathopt) → {Promise.<Void>}
Saves the content of the DataFrame as the specified table.
Parameters:
Name | Type | Attributes | Description |
---|---|---|---|
path |
string |
<optional> |
Saves the content of the DataFrame at the specified path. |
- Since:
- EclairJS 0.1 Spark 1.4.0
- Source:
Returns:
A Promise that resolves to nothing.
- Type
- Promise.<Void>
saveAsTable(tableName) → {Promise.<Void>}
Saves the content of the DataFrame as the specified table.
In the case the table already exists, behavior of this function depends on the
save mode, specified by the `mode` function (default to throwing an exception).
When `mode` is `Overwrite`, the schema of the DataFrame does not need to be
the same as that of the existing table.
When `mode` is `Append`, the schema of the DataFrame need to be
the same as that of the existing table, and format or options will be ignored.
When the DataFrame is created from a non-partitioned HadoopFsRelation with a single input
path, and the data source provider can be mapped to an existing Hive builtin SerDe (i.e. ORC
and Parquet), the table is persisted in a Hive compatible format, which means other systems
like Hive will be able to read this table. Otherwise, the table is persisted in a Spark SQL
specific format.
Parameters:
Name | Type | Description |
---|---|---|
tableName |
string |
- Since:
- EclairJS 0.1 Spark 1.4.0
- Source:
Returns:
A Promise that resolves to nothing.
- Type
- Promise.<Void>