Gaia DR3 Flux Analyzer — ObjectScript Edition

A solution to InterSystems Employee Programming Challenge #1. It scans 20 gzipped
Gaia DR3 epoch-photometry files and reports every astronomical object whose BP or RP
flux varied by more than 100 % across the observation period.

Implementation language: InterSystems ObjectScript (pure — no Python on the
execution path). A companion Python Embedded Python Edition exists as a separate
variant; this repository is the ObjectScript variant and is fully self-contained.

Parallel ObjectScript update

The ObjectScript edition now processes the 20 independent Gaia input files in
parallel using IRIS %SYSTEM.WorkMgr. Each worker writes indexed intermediate
results, and the parent routine merges them in deterministic input order. The
result remains byte-identical to the previous verified serial implementation.

What it computes

For each source_id, over the valid (finite, non-NaN) values of each band:

bp_percentage_change = ((bp_max - bp_min) / bp_min) * 100
rp_percentage_change = ((rp_max - rp_min) / rp_min) * 100
percentage_change    = max(bp_percentage_change, rp_percentage_change)

A source is written only if percentage_change > 100 (strict; exactly 100 is
excluded). Output columns, in order:

source_id,bp_min_flux,bp_max_flux,rp_min_flux,rp_max_flux,percentage_change

Prerequisites

• Docker Desktop
• git

Build & run (the moderator workflow)

git clone https://github.com/bostonbrad/gaia-dr3-flux-objectscript.git
cd gaia-dr3-flux-objectscript
docker compose up --build -d
docker compose exec iris iris session iris

Then in the IRIS terminal (namespace USER):

USER>do ^RunScript

This prints the qualifying row count and elapsed time, and writes the result to
data/out/result.csv.

How it works

The production pipeline is pure ObjectScript. It runs the 20 independent input
files in parallel using IRIS %SYSTEM.WorkMgr (a shipped IRIS feature — no
third-party dependency), then merges deterministically.

• src/RunScript.mac — the benchmark entry point. Run() times the work with
  $ZHOROLOG and calls $$Process^GaiaChallenge.
• src/GaiaChallenge.mac — the solution:
  • Process assigns each data/in/*.csv.gz a stable index in sorted filename
    order, initializes a %SYSTEM.WorkMgr queue, and dispatches one independent
    work unit per file. It then blocks on WaitForComplete.
  • Worker (runs in a worker process) parses its one file and writes that
    file’s qualifying records to the shared global ^GaiaWork(fileIdx, seq)
    (seq = row order within the file).
  • Each file is read with the native %Stream.FileCharacterGzip (no external
    decompression), the 365-line ECSV comment header and column header are skipped,
    rows are split with a quote-aware parser that stops at the rp_flux column (17),
    valid $DOUBLE flux values per band are collected (ignoring
    NaN/null/±Inf/invalid), min/max and the per-band percentage are computed,
    the larger is taken, and rows above 100 % are emitted.
  • WriteOutput merges ^GaiaWork to data/out/result.csv by iterating
    fileIdx in sorted-filename order, then seq in within-file order — so the
    output is deterministic and byte-identical to the serial single-pass output
    (UTF-8, LF, one trailing newline).
• Fail-loud, no partial output. A worker that hits malformed, missing, or corrupt
  input returns an error status; WaitForComplete surfaces it to the parent, which
  re-throws — so a worker failure fails the whole run (it is never swallowed) and
  no output file is written.

Correctness methodology

Correctness is defined by an independent Python reference oracle
(tools/reference_solution.py), kept structurally separate from the production code.
A tolerance comparator (tools/compare_results.py) checks equivalence by parsed value
(abs/rel 1e-9), not text.

• 16/16 test cases pass (14 computed fixtures + 2 loud-failure cases:
  tests/, run python tests/test_reference.py).
• Production output equals the oracle over the full 20-file dataset: 57,099
  rows, identical source_id set and order, all fields within 1e-9.
• A clean docker compose build --no-cache reproduces the output byte-identically
  (SHA-256 776a0e3e…).

Measured performance

Median ~33.6 s inside Run() on the development machine, processing the 20 files
in parallel via %SYSTEM.WorkMgr (one warm-up run, then three measured runs:
32.84 / 35.79 / 33.65 s → min 32.84 s, median 33.65 s, spread 2.95 s). Environment:
22 cores / 32 GB, worker pool sized to the 20 files. Every run produced 57,099 rows and
the identical output SHA-256. These are local measurements; runtime is
hardware-dependent and official moderator results may differ. Only the code inside
Run() is timed. (The previous serial implementation measured a ~101 s median on the
same machine; that figure is retained for comparison in docs/performance-report.md.)

Output

• File: data/out/result.csv
• Rows (this dataset): 57,099
• SHA-256: 776a0e3ea95c5befa6663cdf3cfe176608b5745e5bb701d8ab0ea794aff6ed7e

Project notes

Design decisions, data profiling, and engineering reports are in docs/. The data/in/
inputs are the fixed challenge benchmark and are never modified.

License

See LICENSE.