Skip to main content

Change Files

from duckstring import landing

A Spout with --mode changes lands each delivery as files: for every table that changed, a file of its changes or of the whole table, then a manifest naming those files. Change files describes the layout. These functions read a destination's deliveries back, from a local path, a /Volumes/... path or an object-store URI.

Each delivery has a position: its freshness as an ISO timestamp, with #n added for the n-th delivery again at a freshness already delivered (a resync). Positions order the deliveries. Keep the position of the last delivery you applied, and pass it back as after.

deliveries​

landing.deliveries(path, *, after=None) -> list[dict]

The committed deliveries at path, oldest first, as their manifests.

ParameterTypeDefaultDescription
pathstrThe Spout's destination.
afterstr or datetimeNoneOnly deliveries after this position, given as a position string or a freshness. None lists them all.

A manifest that's still being written ends the list, so the list never skips a delivery.

Each manifest is a dict:

KeyDescription
formatThe layout's version, 1.
pond, majorThe Pond line that published the data.
deliveryThe delivery's position.
fThe delivery's freshness, as an ISO timestamp.
previous_fThe freshness of the delivery before it, or None for the first.
tablesOne entry per table the delivery landed, below.
Table keyDescription
kind"changes" for the changes since the previous delivery, "full" for the whole table.
pkThe table's primary key, or None.
column"_duckstring_op" or "_duckstring_d" for changes, None for a whole table.
rowsThe number of rows landed.
filesThe table's files, relative to path.

changes​

landing.changes(con, path, table, *, after=None) -> Changes | None

table's changes in the deliveries after after, combined into one relation on con. Returns None when no delivery since touched the table.

ParameterTypeDefaultDescription
conduckdb.DuckDBPyConnectionThe connection the relation is made on.
pathstrThe Spout's destination.
tablestrThe table's name.
afterstr or datetimeNoneOnly deliveries after this position.

Deliveries combine as follows:

  • With a primary key, each changed key's row from the newest delivery that has it, with _duckstring_op: 1 with the key's row (insert or update), -1 with its old row (delete).
  • Without one, the weights of each row summed, with _duckstring_d holding the sum. Rows whose weights cancel out are left out.
  • When a whole-table delivery is among them, from the newest such delivery on. reset is then true, and with a primary key only the 1 rows are returned.

Every row also carries _duckstring_f, the freshness of the delivery it came from (without a key, the newest of the deliveries it was combined from). On a Volume reached through the Files API, the files are fetched and loaded into a temporary table on con.

Changes​

AttributeTypeDescription
relationduckdb.DuckDBPyRelationThe combined changes, with the table's columns plus _duckstring_f and _duckstring_op or _duckstring_d.
resetboolA whole-table delivery is included, so relation holds the whole table: replace what you hold with it.
pklist of str or NoneThe table's primary key.
fdatetimeThe newest delivery's freshness.
afterstrThe newest delivery's position. Pass it as after next time.

This applies a merge table's changes to a copy in DuckDB:

import duckdb

from duckstring import landing

con = duckdb.connect("replica.duckdb")
got = landing.changes(con, "/srv/landing/priced", "priced_line", after=last_applied)
if got is not None:
rows = got.relation.sql_query()
cols = "order_id, product_id, quantity, unit_price, revenue"
if got.reset:
con.execute("DELETE FROM priced_line")
else:
con.execute(f"DELETE FROM priced_line WHERE order_id IN (SELECT order_id FROM ({rows}))")
con.execute(f"INSERT INTO priced_line SELECT {cols} FROM ({rows}) WHERE _duckstring_op = 1")
last_applied = got.after