Data workflow examples

pyxsd makes XML schema-valid and schema-typed. It does not do your analysis. These examples show the boundary in three end-to-end paths, each with runnable code in this repository:

  • XML to typed application records — validate the document, snapshot generated Pydantic models, and apply business rules with exact arithmetic.

  • XML to nested or contextual Arrow to analysis — prepare a nested record projection, or named scalar columns with parent context, and query them with DuckDB or Polars.

  • Parquet to analysis — repeat the same queries from a Parquet export without re-reading the XML.

Schema interpretation, exact decimals, nil/presence semantics, and document validation stay in the library. Aggregations, joins, business policy, and consumer-specific type compatibility stay in application code.

Install

The library is unchanged by these examples. Install only the extras you need:

pip install 'pyxsd[arrow]' duckdb
pip install 'pyxsd[arrow]' polars
pip install 'pyxsd[pydantic]'

From a repository checkout, the examples-* dependency groups provide the same consumer environments without adding DuckDB or Polars to the published package. Each line is an alternative: uv sync makes the environment match exactly the groups you name, so a second sync without the first group removes it.

uv sync --extra arrow --group examples-duckdb                    # DuckDB workflow
uv sync --extra arrow --group examples-polars                    # Polars workflow
uv sync --extra arrow --group examples-duckdb --group examples-polars  # both
uv sync --extra pydantic                                         # assessment app

Tested with DuckDB 1.5.6 and Polars 1.44.2 against PyArrow 25.0.1; the Arrow extra supports PyArrow >=22, and the compatible versions are verified at implementation time rather than treated as universal.

Shared preparation

examples/analytics/common.py prepares the same four observation records two ways, from the unchanged examples/arrow/observations.xsd and observations.xml fixtures:

  • a nested projection for struct and list handling, where reading is a struct with $nil, @unit, and $ fields and tag/replicates are lists;

  • a contextual projection of scalar columns — station, sequence, accession, collected, analyte, reading, unit, quality — suited to aggregation and joins.

The two representations are complementary. The analytical examples use the scalar table for totals and keep the nested table for child expansion, so expanding tags and replicates never multiplies the original reading totals.

Two representations follow library policy rather than consumer convenience:

  • accession (an arbitrary-size xs:integer) and collected (xs:dateTime, including a nanosecond lexical timestamp) stay strings. There is no silent numeric narrowing or timezone conversion. Convert explicitly in your application if you need numbers or instants.

  • reading is an exact decimal128(8, 3) column (XSD totalDigits + fractionDigits), not a float. Decimal sums are compared exactly before any formatting.

A checked-in examples/analytics/stations.json maps three station identifiers to plain region labels. The join enforces one catalog entry per key; duplicate keys fail before joining, and an unknown station still produces a row with an explicit unknown region.

DuckDB workflow

From the repository root:

python examples/analytics/duckdb_observations.py

--- nested struct and list demonstration (XML path only) ---
Reading units: [(1, 'mg/L'), (2, 'mg/L'), (3, 'mg/L'), (4, 'mg/L')]
Child row counts: {'tags': 4, 'replicates': 8}
Station regions: [(1, 'river'), (2, 'river'), (3, 'river'), (4, 'coastal')]
Passing quality sequences: [1, 4]
--- summary ---
dissolved oxygen [mg/L] readings=2 populated=2 total=12.000
nitrate [mg/L] readings=2 populated=1 total=0.015

What the output shows:

  • The nil nitrate reading keeps its mg/L unit even though its value is null (reading['$nil'] is true, reading['@unit'] is still mg/L).

  • Four tag values and eight replicate values expand from the separate nested relation; the analyte totals still come from the unexpanded four-record table, so 12.000 and 0.015 are not multiplied.

  • quality filtering selects sequences [1, 4], excluding the nil reading.

  • Grouping is by analyte and unit; units are never mixed.

Export a Parquet file and query it back without re-reading the XML:

python examples/analytics/duckdb_observations.py --write-parquet observations.parquet
python examples/analytics/duckdb_observations.py --parquet observations.parquet

The Parquet path uses DuckDB’s parameterized file access (read_parquet($path)); paths are never interpolated into SQL, so directories with spaces or quotes are safe. The --parquet run prints the same summary and reads no XML.

Polars workflow

python examples/analytics/polars_observations.py

The eager path imports the same Arrow table with pl.from_arrow and produces the same summary lines. The Polars module also checks the imported dtypes (accession/collected strings, sequence unsigned integer, quality boolean, reading Decimal(8, 3)) and representative exact values, because Polars documents fallback casting for some unsupported Arrow types: verify the types your version actually delivers instead of assuming zero-copy fidelity.

The lazy Parquet companion scans without materializing the XML again:

python examples/analytics/polars_observations.py --write-parquet observations.parquet
python examples/analytics/polars_observations.py --parquet observations.parquet

List expansion normalizes empty and null lists to zero child rows explicitly (explode(empty_as_null=False).drop_nulls()), so both consumers agree on the documented counts. Lazy Parquet is not lazy XML parsing — the XML is fully parsed when the Arrow table is prepared.

Both consumers agree on independently enumerated expected values, not merely with each other: the tests assert the literal expected results, not cross-tool equality.

Order assessment

examples/pydantic/assess_orders.py parses and validates orders.xml, snapshots generated Pydantic models, and applies synthetic policy in one order currency:

python examples/pydantic/assess_orders.py

Each order reports an order number, a disposition (accepted, approval_required, or rejected), its currency, an exact subtotal as a JSON string, and deterministic reason codes. With the synthetic catalog fully recognized:

  • PO-2026-1041 — USD 681.90, approval_required (above the 600.00 threshold);

  • PO-2026-1042 — EUR 450.00, accepted;

  • PO-2026-1043 — USD 100.00, accepted (exactly at the boundary is accepted; only strictly greater requires approval).

The rejection and boundary cases live in orders_review.xml:

python examples/pydantic/assess_orders.py examples/pydantic/orders_review.xml

  • PO-2026-2001 — unknown account NWL-99, rejected, subtotal still reported;

  • PO-2026-2002 — unknown SKU GADGET-X, rejected, subtotal still reported;

  • PO-2026-2003 — a USD line and a EUR line, rejected as mixed_currencies with no currency and no aggregate subtotal (no exchange rate is invented);

  • PO-2026-2004 — USD 600.00, accepted at the exact threshold.

The account list, SKU list, and 600.00 approval threshold are synthetic policy fixtures for this example. They are not XSD constraints, not a complete catalog, and not financial advice. XSD validity and business acceptance are separate: an XSD-valid order can be rejected by policy, and invalid XML or model input never reaches policy evaluation at all.

Absent and explicitly nil notes are different states. model_fields_set distinguishes a nil <note/> from an omitted element, and the example never mutates the source XML.

Caveats and failure modes

Out-of-profile decimals are rejected, not cast. The shared assert_consumer_profile check accepts exact decimals with precision <= 38 and rejects wider fields (for example decimal256(76, 6)) with an error naming the field, its type, and the rule. There is no silent float or string fallback, and no claim that every PyArrow type is DuckDB- or Polars-compatible.

Duplicate catalog keys fail before joining. load_stations raises a ValueError naming the duplicate station. One catalog entry per key is a precondition, not a dedupe step.

Malformed XML and policy rejection are different outcomes. A not-well-formed document raises pyxsd.exceptions.PyXSDError during parse; an XSD-invalid document (for example quantity of zero) parses but raises pyxsd.exceptions.ValidationError when the snapshot revalidates. Both happen before any business rule runs, so invalid input never appears as a rejected-but-valid order.

Temporal and identifier conversion is explicit. Strings stay strings. Parse timestamps or convert large integers in your own code, where the precision and timezone decisions belong to your application.

Why analysis stays outside the library

pyxsd deliberately does not embed SQL execution, dataframe engines, joins, business rules, exchange rates, or user-supplied expressions in the library or its integrations. The library delivers schema-typed values with honest validation; consumers decide what the values mean. That boundary keeps the base dependency surface at elementpath, keeps Arrow and Pydantic independently optional, and lets DuckDB, Polars, and application policy evolve on their own schedules.