Gaia DR3 Variable-Source Finder — NumPy Tensor Approach

A submission to the 1st InterSystems Employee Programming Challenge
(contest #47).

The Idea

Most solutions parse CSV at runtime — spending 80%+ of their time on gzip
decompression and string-to-float conversion. This solution takes a different
approach: transform the problem into pre-computed vectors at build time.

At Docker build time, all flux arrays are parsed, and per-source min/max
values are computed and stored as compact NumPy binary vectors (~3MB).
At runtime, the benchmark measures only vectorized percentage calculation,
filtering, and CSV output — no CSV parsing, no decompression, no string work.

BUILD TIME (once, not benchmarked):
  *.csv.gz → parse → nanmin/nanmax → data/temp/*.npy (5 vectors, ~3MB)
RUNTIME (benchmarked):
np.load → percentage calc → filter → write CSV

How It Works

Build Phase (build_tensors.py)

Reads all 20 .csv.gz files, extracts source_id, bp_flux, and rp_flux,
pads flux arrays into matrices, computes nanmin/nanmax per source, and
saves five vectors:

Runtime Phase (challenge.py)

1. Load: np.load() five small vectors (~3MB total)
2. Compute: percentage_change = ((max - min) / min) * 100 — vectorized
3. Filter: boolean mask pct_change > 100
4. Write: Polars write_csv (Rust-native serialization)

Installation & Running

Prerequisites: git
and Docker Desktop.

git clone 
cd intersystems-challenge1
docker-compose up --build -d

Run the benchmark:

docker compose exec iris iris session iris -U USER

USER> do ^RunScript
Found 57099 objects with >100% flux change
Results written to /home/irisowner/dev/data/out/results.csv
Matched sources: 57099
Elapsed time: .180 seconds

Output is written to data/out/results.csv.

Project Structure

src/
  challenge.py       Runtime: load vectors, percentage calc, filter, write CSV
  build_tensors.py   Build time: CSV.gz → pre-computed min/max vectors
  RunScript.mac      IRIS entry point (do ^RunScript)
data/
  in/                20 input .csv.gz files
  temp/              Pre-computed .npy vector files (created at build)
  out/               Output results.csv (created at runtime)
Dockerfile           Installs deps, builds tensors, loads IRIS routines
docker-compose.yml

Technology

• InterSystems IRIS Community + Embedded Python — platform and runtime
• NumPy — vectorized percentage calculation and filtering
• Polars — fast CSV output (Rust-native writer)
• Docker — reproducible build with docker compose up --build

AI Hub Integration

The solution includes an AI Hub GaiaAnalyzer ToolSet (see challenge.py) that
wraps the analysis as a @tool callable by an %AI.Agent. When iris_llm is
installed, an LLM agent can invoke the photometry analysis via natural language.

Data & Attribution

Input data: ESA Gaia DR3 epoch photometry.
Column definitions: https://gea.esac.esa.int/archive/documentation/GDR3/Gaia_archive/chap_datamodel/sec_dm_photometry/ssec_dm_epoch_photometry.html