Gaia DR3 Variable Star Detection – Speed Entry

InterSystems Employee Programming Challenge #1

Detects variable stars in Gaia DR3 epoch photometry using parallel binary scanning.
Under 1 second end-to-end on the 20 benchmark files (360 MB compressed,
1.54 GB uncompressed), producing a CSV of 57,099 variable sources sorted by flux
variability.

How It Works

1. do ^RunScript in IRIS triggers irispython run_embedded.py
2. isal (SIMD-accelerated gzip) inflates all 20 files in parallel via
   ProcessPoolExecutor – decompression is ~70% of the workload
3. Inflate is streamed in 4 MB chunks, so no worker ever holds a whole
   decompressed file in memory
4. Per-file: binary scan finds BP flux (9th [) and RP flux (14th [) arrays per
   source; NaN values are dropped
5. Computes pct_change = (max - min) / |min| x 100 – keeps sources >100%
6. Writes result.csv sorted by pct_change DESC

Prerequisites

• Docker + Docker Compose v2 installed (docker compose up --wait needs v2.17+)
• wget – used by the download step below (curl -O works if you prefer;
  substitute it in the loop)
• 4 GB RAM available to Docker
• ~7 GB free disk space. Measured on a clean build: 3.77 GB image (2.6 GB of
  that is the IRIS base image) + 0.78 GB container writable layer once IRIS has
  started + 0.37 GB downloaded data = 4.9 GB steady state. Allow ~2 GB more
  for BuildKit’s cached copy of the pulled base layers during
  docker compose build.
• Internet access for the image pull and the 360 MB data download

Uses the public intersystems/iris-community:latest-em image from Docker Hub –
no registry login or license key required.

Get the Data

Do this first – the pipeline reads data/in/ and has nothing to do if it is
empty. Download the 20 benchmark Gaia DR3 EpochPhotometry files from the ESA
archive:

mkdir -p data/in
cd data/in
BASE=https://cdn.gea.esac.esa.int/Gaia/gdr3/Photometry/epoch_photometry
for f in \
  EpochPhotometry_000000-003111 EpochPhotometry_003112-005263 \
  EpochPhotometry_005264-006601 EpochPhotometry_006602-007952 \
  EpochPhotometry_007953-010234 EpochPhotometry_010235-012597 \
  EpochPhotometry_012598-014045 EpochPhotometry_014046-015369 \
  EpochPhotometry_015370-016240 EpochPhotometry_016241-017018 \
  EpochPhotometry_017019-017658 EpochPhotometry_017659-018028 \
  EpochPhotometry_018029-018472 EpochPhotometry_018473-019161 \
  EpochPhotometry_019162-019657 EpochPhotometry_019658-020091 \
  EpochPhotometry_020092-020493 EpochPhotometry_020494-020747 \
  EpochPhotometry_020748-020984 EpochPhotometry_020985-021233 ; do
  wget "$BASE/${f}.csv.gz"
done
cd ../..

Expected: 20 files, 360 MB total compressed (11-28 MB each), 1.54 GB
uncompressed. These are the first 20 files of the archive.

Quick Start

With the 20 .csv.gz files in data/in/ (see Get the Data):

docker compose up --build -d --wait
docker compose exec iris iris session IRIS -U USER ^RunScript
head -5 data/out/result.csv

--wait blocks until the container’s healthcheck reports IRIS as running, so the
exec on the next line cannot race the startup. Without it the exec may fail
with “IRIS is not running” on the first try.

Output Format

source_id,bp_min_flux,bp_max_flux,rp_min_flux,rp_max_flux,percentage_change
94494163890456704,0.000000,153710.892478,22.176222,171140.910128,1258654236473444007936.0000
...

Tests

tests/e2e.sh          # or: tests/e2e.sh 

Deletes data/out/result.csv, runs ^RunScript in the container, and checks
the output the way a judge would: 20 inputs present, header, 57,099 rows,
descending sort, no row at or below the 100% threshold, no duplicate
source_id, min <= max on both bands, and a wall-clock ceiling read from the
routine’s own printed elapsed time.

One assertion is a literal checksum of the source_id set. The same checksum
is asserted by gaia-iml and
gaia-terse, which reach the answer
by three unrelated routes – so if the three ever disagree, at most one of them
is right.

Performance

Measured on 16 cores. Scales with core count – inflate and parse are both
CPU-bound, so one worker per core.

Related Entries

• gaia-iml – IntegratedML PREDICT()
  plus an AI Hub agent (+6 bonus pts)
• gaia-terse – minimal-code entry, a
  single Python expression (+3 bonus pts)