Write and read SparkDataFrame in delta format

Write and read SparkDataFrame in delta format

Usage

dlt_write(df, path, ...)

write.delta(df, path, ...)

dlt_read(path, ...)

read.delta(path, ...)

# S4 method for SparkDataFrame,character
dlt_write(df, path, ...)

# S4 method for SparkDataFrame,character
write.delta(df, path, ...)

Arguments

df

SparkDataFrame

path

character path to write the data.

...

additional arguments passed to writer

Functions

• dlt_write(): Generic

• write.delta(): Generic

• dlt_read(): Load data stored as delta into SparkDataFrame

• read.delta(): Load data stored as delta into SparkDataFrame (alias of dlt_read)

• dlt_write(df = SparkDataFrame, path = character): Write SparkDataFrame` in delta format

• write.delta(df = SparkDataFrame, path = character): Write SparkDataFrame in delta format (alias of dlt_write)

Note

dlt_read since 1.0.0

read.delta since 1.0.0

dlt_write since 1.0.0

write.delta since 1.0.0

Examples

if (FALSE) {
set.seed(323)
path <- tempfile()

df <- data.frame(
  id = 1:12,
  key = rep(c("a", "b", "c"), each = 4),
  value = rnorm(12)
) %>%
  createDataFrame()

dlt_write(df, path)

dlt_read(path) %>%
  schema()

write.delta(df, path, mode = "overwrite")

read.delta(path) %>%
  schema()
}