Gaia DR3 Flux Analyzer — Embedded Python Edition

A solution to InterSystems Employee Programming Challenge #1. It scans 20 gzipped
Gaia DR3 epoch-photometry files and reports every astronomical object whose BP or RP
flux varied by more than 100 % across the observation period.

Implementation language: Python 3 (standard library only), run through IRIS
Embedded Python. do ^RunScript calls a Python module via %SYS.Python; there is
no ObjectScript computation on the execution path. A companion ObjectScript Edition
exists as a separate variant; this repository is the Embedded Python variant and is
fully self-contained.

What it computes

For each source_id, over the valid (finite, non-NaN) values of each band:

bp_percentage_change = ((bp_max - bp_min) / bp_min) * 100
rp_percentage_change = ((rp_max - rp_min) / rp_min) * 100
percentage_change    = max(bp_percentage_change, rp_percentage_change)

A source is written only if percentage_change > 100 (strict; exactly 100 is
excluded). Output columns, in order:

source_id,bp_min_flux,bp_max_flux,rp_min_flux,rp_max_flux,percentage_change

Prerequisites

• Docker Desktop
• git

Build & run (the moderator workflow)

git clone https://github.com/bostonbrad/gaia-dr3-flux-embedded-python.git
cd gaia-dr3-flux-embedded-python
docker compose up --build -d
docker compose exec iris iris session iris

Then in the IRIS terminal (namespace USER):

USER>do ^RunScript

This prints the qualifying row count and elapsed time, and writes the result to
data/out/result.csv.

How it works

• src/RunScript.mac — the benchmark entry point. Run() times the work with
  $ZHOROLOG, adds src/ to the Python path, imports gaia_production via
  ##class(%SYS.Python).Import, and calls gaia.run(inDir, outFile).
• src/gaia_production.py — the solution (standard library only: gzip, math,
  os):
  • streams each data/in/*.csv.gz line by line in sorted filename order;
  • skips the ECSV comment header and column header;
  • splits rows with a quote-aware positional parser that stops at the rp_flux
    column (17);
  • folds min/max per band in a single pass (ignoring NaN/null/±Inf/invalid),
    computes the per-band percentage, takes the larger, and emits rows above 100 %;
  • writes data/out/result.csv (UTF-8, LF, one trailing newline).
• It fails loudly (raises, non-normal exit, no partial output) on malformed,
  missing, or corrupt input, naming the file, row, and reason.

No third-party packages, no network access.

Correctness methodology

Correctness is defined by an independent Python reference oracle
(tools/reference_solution.py), kept structurally separate from the production module
(different I/O strategy — the oracle reads whole files with the csv module and builds
value lists; production streams and folds min/max in one pass). A tolerance comparator
(tools/compare_results.py) checks equivalence by parsed value (abs/rel 1e-9).

• 16/16 test cases pass — run python tests/test_superset.py (14 computed
  fixtures + malformed_row + missing_input), for both the oracle and production.
• Production output equals the oracle over the full 20-file dataset: 57,099
  rows, identical source_id set and order, all fields within 1e-9.
• A clean docker compose build --no-cache reproduces the output byte-identically
  (SHA-256 5a2a4cdc…).

Measured performance

Median ~51 s inside Run() on the development machine (three runs:
50.3 / 54.5 / 51.1 s, re-measured 2026-07-24) — roughly 2× faster than the
ObjectScript edition, because the workload is gzip-I/O-bound and CPython handles that
in C. These are local measurements; runtime is hardware-dependent and official
moderator results may differ. Only the code inside Run() is timed.

Output

• File: data/out/result.csv
• Rows (this dataset): 57,099
• SHA-256: 5a2a4cdc008dbd0eee078714d418887c386fa8a1fab20ca36dbfc4a6eb566d53

Project notes

Design decisions, data profiling, the Python-strategy review, and the full benchmark
report are in docs/. The data/in/ inputs are the fixed challenge benchmark and are
never modified.

License

See LICENSE.