Sequential multi-document parsing

Compile an XSD once and parse an explicit sequence of local XML files:

from pyxsd import Schema
from pyxsd.batch import DocumentSource

schema = Schema.compile("records.xsd")
sources = [DocumentSource("first", "a.xml"), DocumentSource("second", "b.xml")]
for outcome in schema.iter_parse(sources, errors="report"):
    print(outcome.source.id, outcome.status)
    for issue in outcome.issues:
        print(issue.phase, issue.format())
    if outcome.error is not None:
        print(outcome.error.kind, outcome.error.message)

This API adds no dependencies. An ordinary loop over Schema.parse remains supported, including its broader stream and parsed-element inputs.

Sources and outcomes

DocumentSource(id, path) requires a nonempty string ID and a text path (str or os.PathLike[str]). It anchors relative paths to the current working directory at construction; the file need not exist yet. Changing directories during iteration does not change those paths. Inline XML, URLs, streams, directory discovery, and parsed elements are not batch inputs.

IDs are caller labels, not filenames, hashes, or XSD identities. Duplicate source IDs stop iteration with ValueError when encountered, before opening the duplicate. Different IDs may refer to the same path: it is parsed twice. XSD keys/keyrefs are checked independently within each document. A repeated business ID in another document is legal; a keyref cannot resolve across files.

Each frozen ParseOutcome carries its source and one of these statuses:

Status	Document	Issues	Error
valid	Parsed Document	All issues, including warnings	None
invalid	Parsed Document (possibly without a bound root)	All validation issues	None
input_error	None	Empty tuple	InputFailure

Issues are tuple snapshots of the native report. Their phase identifies schema-compilation or instance-binding findings; see Validation for codes and severities. The record is frozen, not its contained Document. Later mutations need explicit Document.revalidate() under the existing API. Earlier Documents remain usable after later parses, with the same native query/export/serialization limitations as single-document parsing.

InputFailure(kind, message) stores text, not an exception or traceback. Its kind is io for expected file I/O failures or malformed_xml for XML syntax errors. Source paths are absolute runtime inputs; avoid exposing them in exported records by default. Native I/O error messages may also include paths, so redact them before publishing diagnostics.

Error timing and policies

Schema.iter_parse(sources, *, errors="raise") returns a lazy iterator. Policy validation and schema.require_valid() run on first iteration, before pulling any source, even for an empty iterable. An invalid schema is a global ValidationError, not a repeated per-file failure. Compilation always precedes the batch; instance schema-location hints do not select a different schema.

The default raise policy stops at the first invalid document or expected input failure. It raises pyxsd.batch.BatchParseError, a PyXSDError subclass, with the failed result in .outcome. Its native validation/input exception is chained as __cause__. The failed outcome is not yielded, and later inputs are not requested.

The report policy yields those failed outcomes and continues. Neither policy suppresses internal binding defects, unclassified PyXSDErrors, input-iterator exceptions, interrupts, or cancellation. Invalid policies, non-DocumentSource inputs, and duplicate IDs are configuration errors, not document outcomes.

Resources and limits

Processing is sequential in caller order, with no prefetch. Do not parse on the same Schema concurrently or reentrantly while its iterator is active. The API offers no scheduler, shared-schema concurrency, or XML streaming: each file still becomes a materialized tree.

Files are closed before results are yielded. The iterator keeps no list of Documents or outcomes, only current input state and an O(number of inputs) ID set. The native compiler host may keep bounded state from its latest parse. Retaining outcomes intentionally retains their Documents; an exception caught in raise mode can also retain trees through its traceback. Report-mode failure records contain no live tracebacks. Close a partially consumed iterator to release its own references:

from contextlib import closing

with closing(schema.iter_parse(sources, errors="report")) as outcomes:
    first = next(outcomes)

No constant-RSS guarantee is made: tree size and Python allocator behavior still matter.

Runnable validation example

From a repository checkout:

uv run python examples/batch/validate_documents.py records.xsd inputs b.xml a.xml

The example accepts explicit filenames relative to the input root, sorts their relative IDs, and uses namespaced binding. It prints each observed result and exits 0 for all valid inputs, 1 for invalid/unreadable/ malformed inputs, or 2 for configuration, schema, or internal failures. Unexpected defects include a traceback. Sorting is application behavior; the library never discovers or sorts files.

Optional Parquet dataset export

Install pyxsd[arrow], prepare one projection, and give it explicit sources:

from pyxsd import ParseModes, Schema
from pyxsd.batch import DocumentSource
from pyxsd.integrations.arrow import records

schema = Schema.compile("examples/arrow/observations.xsd", mode=ParseModes.NAMESPACED)
projection = records(schema, element="observations", path=("observation",))
result = projection.write_dataset(
    [DocumentSource("day-1", "examples/arrow/observations.xml")],
    "new-observation-dataset",
    selector="observation",
    batch_size=10000,
    errors="report",
)
print(result.status, result.rows, result.manifest)

DatasetResult is a frozen, dependency-free record from pyxsd.integrations.dataset: destination, status, inputs, succeeded, failed, rows, and manifest. A returned result is not a claim that every source succeeded:

Overall status	Meaning
complete	No failures, including zero inputs or successful empty selections
partial	At least one successful input and at least one expected failure
failed	Attempted inputs all failed; report mode still publishes schema and manifest

The default errors="raise" aborts at the first expected input/validation/ projection failure, does not pull later inputs, removes its staging output, and publishes nothing. errors="report" records those failures and continues. Input I/O and malformed XML remain distinct from invalid documents and projection errors. A late projection failure discards that source’s entire part, even after earlier batches were written; its row count is zero. Output write/close/publication failures, unexpected backend/binding defects, iterator errors and duplicate IDs abort both modes. DatasetExportError extends IntegrationError, carries source_id (possibly None) and stage, and chains the cause. Interrupts/cancellation propagate after cleanup.

Layout, provenance and manifest version 1

new-observation-dataset/
  manifest.json
  _schema.parquet
  parts/
    part-000001.parquet
    part-000003.parquet

One closed part is admitted per successful input, including a typed zero-row part for an empty selection. Filenames use 1-based input ordinals, never source IDs or source paths; failed inputs leave gaps. _schema.parquet is always a zero-row reference with the complete schema, not a source part.

Original fields, nullability and metadata are preserved, followed by two reserved non-null columns: __pyxsd_source_id (string) and __pyxsd_row_index (int64). Row indexes start at 1 per source and continue across batches. Their pair identifies rows within this run/selection, not a stable XML node key after edits or selector changes. Storage/query order is not guaranteed: order by provenance explicitly. Repeated business keys across files are allowed; related-table joins/deduplication remain application work.

Manifest JSON includes format_version=1, overall status, versions (pyxsd, pyarrow), xsd_version, parse_policy, projection metadata and row_route, selector, namespaces, schema_file, counts and ordered entries. Schema/field byte metadata is encoded as lists of key_base64/value_base64 pairs. The schema reference supplies exact Arrow types; version/route metadata is not a transitive-XSD fingerprint. Dataset writers preserve prepared list-child names (typically item) instead of PyArrow’s default canonical element rename; the Parquet list layout stays three-level. Consumers must support that legacy child-name convention.

Each entry has source_id, ordinal, status, rows, part, copied issues (severity, code, message, element, phase), and failure (null or kind/message). Entry statuses are written, empty, invalid, input_error or projection_error. Failed entries always have rows=0 and part=null. No Documents, live exceptions, raw XML or dedicated absolute source paths are serialized. Diagnostic text can include paths and sensitive values: manifests are local artifacts, not sanitized telemetry.

The manifest uses O(inputs + diagnostic text) memory; the writer retains no list of Documents or batches. Batching bounds projected output row count, not XML memory: each input still becomes a tree and selection can materialize nodes. Keep source files unchanged throughout the run and do not share the Schema with concurrent/reentrant parses. Instance schema hints do not choose or independently fetch another schema; the prepared schema/resolver policy governs all inputs.

Publication and recovery limits

Only a new local destination is supported. Its parent must already exist. Existing files, empty directories, and symlinks (including dangling ones) are rejected and never removed. There is no append, overwrite, resume, cloud publication, partition inference or schema merging.

The writer creates an owned sibling .DESTINATION.pyxsd-* staging directory on the same filesystem, closes parts/schema/manifest, rechecks absence, then renames it. macOS uses renamex_np(RENAME_EXCL) and Linux libc with renameat2 uses RENAME_NOREPLACE; unsupported native/filesystem operations abort rather than weaken that primitive. Windows rename rejects existing targets. Other POSIX hosts (or Linux libc without renameat2) have only checked rename. One writer and a stable destination parent are required on every platform; the checked fallback is not race-proof. Concurrent/adversarial filesystem races are outside this API’s contract.

Rename provides atomic local visibility, not fsync/power-loss durability or a distributed transaction. Handled failures remove only owned staging output. A process kill/crash can leave staging directories: identify a known run’s directory manually, verify that no writer is active, and remove it or inspect it outside the API. Incomplete staging is not a published dataset and has no automatic recovery/resume; never delete arbitrary lookalike directories.

Export and consumption recipes

The observation recipe compiles once and sorts explicit relative IDs. From a checkout, with no preexisting observation-dataset:

uv run --extra arrow python examples/batch/export_dataset.py examples/arrow observation-dataset observations.xml

It exits 0 for complete, 1 for partial/failed, 2 for fatal export errors (with chained traceback). --errors raise stops instead of reporting; --batch-size controls rows per output batch.

Read only the successful part paths listed in a trusted local manifest, never recursively glob the dataset. If there are no parts, read _schema.parquet for a typed empty table. The recipe’s read_dataset function does this and orders by source/row index. This consumer materializes the whole dataset; the incremental writer does not.

Reuse the existing analytical observation queries, with consumer libraries installed only in the example environment:

uv run --extra arrow --group examples-duckdb python examples/batch/summarize_dataset.py observation-dataset --consumer duckdb
uv run --extra arrow --group examples-polars python examples/batch/summarize_dataset.py observation-dataset --consumer polars

These reuse the precision-limited consumer profile in Data workflow examples; they add no DuckDB/Polars/pandas library dependency. Generalize the projection for your schema rather than inferring types from sample XML.