Class: DataFrameNaFunctions

eclairjs/sql. DataFrameNaFunctions

Functionality for working with missing data in DataFrame.

Constructor

new DataFrameNaFunctions()

Since:
  • EclairJS 0.1 Spark 1.3.1
Source:

Methods

drop(arg1opt, arg2opt) → {module:eclairjs/sql.Dataset}

Returns a new DataFrame that drops rows containing any null or NaN values.
Parameters:
Name Type Attributes Description
arg1 string | Array.<string> <optional>
 If "any", then drop rows containing any null or NaN values If "all", then drop rows only if every column is null or NaN for that row. If integer Returns a new DataFrame that drops rows containing less than arg1 non-null and non-NaN values. If array of column names
arg2 string | Array.<string> <optional>
 array of column names, only valid if arg1 is string or integer value
Since:
  • EclairJS 0.1 Spark 1.3.1
Source:
Returns:
Type
module:eclairjs/sql.Dataset

fill(value, colsopt) → {module:eclairjs/sql.Dataset}

Returns a new DataFrame that replaces null or NaN values.
Parameters:
Name Type Attributes Description
value number | string | object If number replaces null or NaN values in numeric columns with `value`. If string replaces null values in string columns with `value`. If object, the object is expected to be a HashMap, the key of the map is the column name, and the value of the map is the replacement value. The value must be of the following type: `number`or `String`.
cols Array.<string> <optional>
 replaces null or NaN values in specified columns. Not valid when value is a map. If a specified column type does not match the values type, it is ignored.
Since:
  • EclairJS 0.1 Spark 1.3.1
Source:
Returns:
Type
module:eclairjs/sql.Dataset
Example
var hash = {"name": "missing", "age": "99"};
var result = naFunc.fill(hash);

replace(col, replacement) → {module:eclairjs/sql.Dataset}

Replaces values matching keys in `replacement` map with the corresponding values. Key and value of `replacement` map must have the same type, and can only be numbers or strings. If `col` is "*", then the replacement is applied on all string columns or numeric columns.
Parameters:
Name Type Description
col string name of the column to apply the value replacement
replacement object value replacement map, as explained above
Since:
  • EclairJS 0.1 Spark 1.3.1
Source:
Returns:
Type
module:eclairjs/sql.Dataset
Example
// Replace Michael with MichaelReplace and Andy with AndyReplace in the name column
  var hash = {"Michael": "MichaelReplace", "Andy": "AndyReplace"};
  var result = naFunc.replace("name", hash);
  // Replaces 1600.00 with 99.99, 500000000.11 with 11.11 and 29 with 0 in the age, income and networth columns
  var hash = {"1600.00": 99.99, "500000000.11": 11.11, "29": 0};
  var result = naFunc.replace(["age", "income", "networth"], hash);