Write and read streaming SparkDataFrame in delta format

Write and read streaming SparkDataFrame in delta format

Usage

dlt_write_stream(df, path, ...)

dlt_read_stream(path, ...)

# S4 method for SparkDataFrame,character
dlt_write_stream(df, path, ...)

Arguments

df

streaming SparkDataFrame

path

character path to write the data.

...

additional arguments passed to writer

Functions

• dlt_write_stream(): Generic

• dlt_read_stream(): Loads data stored in delta format, returning the result as a streaming SparkDataFrame

• dlt_write_stream(df = SparkDataFrame, path = character): Writes streaming SparkDataFrame using delta format

Note

dlt_read_stream since 1.0.0

dlt_write_stream since 1.0.0

Examples

if (FALSE) {
set.seed(323)
input_path <- tempfile()
output_path <- tempfile()

# Dummy input
data.frame(
  id = 1:12,
  key = rep(c("a", "b", "c"), each = 4),
  value = rnorm(12)
) %>%
  createDataFrame() %>%
  dlt_write(input_path)

# Read data as stream and write back to output location
query <- dlt_read_stream(path = input_path) %>%
  dlt_write_stream(
    path = output_path, queryName = "test", trigger.once = TRUE,
    checkpointLocation = file.path(output_path, "_checkpoints", "test")
  )

awaitTermination(query)
}