Gaia DR3 Variable Star Detection – Minimal-Code Entry

InterSystems Employee Programming Challenge #1

The whole solution is one $SYSTEM.Python.Run() call holding a 580-character
Python expression. No .py file, no pip install, stdlib only. 57,099 variable
sources in ~16s on the 20-file benchmark.

How It Works

src/RunScript.mac is a comment header and one call:

NaN=0;(lambda rows: open('/o/result.csv','w').write(
    'source_id,bp_min_flux,...\n' + ''.join(...)
))(sorted([
    (s, B, R, p)
    for l in __import__('os').popen('zcat /i/*gz')
    if '/' < l[0] < ':' and (q := l.split('[')) and len(q) > 14
    and (B := list(filter(None, eval('[' + q[9].split(']')[0] + ']'))))
    and (R := list(filter(None, eval('[' + q[14].split(']')[0] + ']'))))
    and (s := l.split(',')[1])
    and (p := (max(max(B)/min(B), max(R)/min(R)) - 1) * 100) > 100
], key=lambda r: r[3], reverse=1))

Key techniques:

• NaN=0 lets eval() handle Gaia’s literal NaN values in the array strings,
  and filter(None, ...) then drops them along with any zero flux
• '/' < l[0] < ':' keeps digit-starting rows, skipping headers and comments
• q[9] / q[14] – BP flux is the 9th [-split segment, RP the 14th
• Walrus operators assign mid-comprehension, so no statements are needed
• max(B)/min(B) - 1 is the same ratio as (max-min)/min in fewer characters
• zcat does the decompression, so no gzip import and no chunking logic

Quick Start

# Place the 20 Gaia EpochPhotometry .csv.gz files in data/in/ (see Get the Data)
docker compose up --build -d --wait
docker exec gaia-terse-iris bash -c \
  'printf "do ^RunScript\nhalt\n" | iris session IRIS'
head -5 data/out/result.csv

--wait is required, not cosmetic. IRIS needs several seconds to start after
the container does, and without it the docker exec on the next line runs
against an IRIS that is not yet accepting logins. The healthcheck in
docker-compose.yml probes with a real session, so --wait returns only once
^RunScript can actually be invoked – no manual sleep.

docker-compose.yml mounts data/in at /i and data/out at /o. The short
names are worth 46 characters of the source budget.

Output Format

source_id,bp_min_flux,bp_max_flux,rp_min_flux,rp_max_flux,percentage_change
94494163890456704,0.000000,153710.892478,22.176222,171140.910128,1258654236473444007936.0000
...

Expected on the 20-file benchmark: 57,099 qualifying sources, ~16s.

The same 57,099 source_ids that gaia-iml
produces. A handful of percentage_change values differ in their last digits,
all of them sources whose minimum flux is 0 and whose ratio therefore lands past
1e17, where float printing is no longer exact.

Tests

tests/e2e.sh          # or: tests/e2e.sh 

Deletes data/out/result.csv, runs ^RunScript, and checks 20 inputs, the
header, 57,099 rows, descending sort, the 100% threshold, duplicate
source_ids, min <= max on both bands, and the shared source_id checksum
also asserted by the other two entries.

Two checks are specific to this entry. Success is asserted as no error having
been raised, rather than as a success message, because the routine prints
nothing on purpose – a write would cost source length. And the length itself
is a gate: the test extracts the Python expression out of src/RunScript.mac
and fails if it grows past 600 characters, currently 580.

Prerequisites

• Docker + Docker Compose installed
• 4 GB RAM available to Docker
• ~5 GB free disk space (image + data)
• Internet access for the image pull and the 360 MB data download

Uses the public intersystems/iris-community:latest-em image from Docker Hub –
no registry login or license key required.

Get the Data

The challenge benchmark is the first 20 files of the Gaia DR3 epoch
photometry archive, EpochPhotometry_000000-003111.csv.gz through
EpochPhotometry_020985-021233.csv.gz:

mkdir -p data/in
cd data/in
PREFIX=Gaia/gdr3/Photometry/epoch_photometry
BASE=https://cdn.gea.esac.esa.int/$PREFIX
curl -s "https://gaia.eu-1.cdn77-storage.com/?prefix=$PREFIX/&delimiter=/" \
  | grep -o '[^<]*' \
  | sed -e 's/<[^>]*>//g' -e 's#.*/##' \
  | grep -E '^EpochPhotometry_[0-9]+-[0-9]+\.csv\.gz$' \
  | sort -u | head -20 \
  | xargs -P 4 -I{} curl -sO "$BASE/{}"
cd ../..

Expected: 20 files, 360 MB total compressed (11-28 MB each), 1.54 GB
uncompressed. On a home connection the download takes a few minutes.

Two things about that listing URL are easy to get wrong. The directory URL
$BASE/ does not return a file list – it 301s to a JavaScript file browser
whose HTML contains no filenames, so scraping it yields zero matches. The
listing has to come from the storage origin
(gaia.eu-1.cdn77-storage.com) in the S3 ?prefix=...&delimiter=/ form, which
returns a ListBucketResult XML document; the <Key> elements are full paths,
hence the sed that strips the tags and the leading directories. The files
themselves are still fetched from the cdn.gea.esac.esa.int CDN.

Verify before building:

ls data/in/*.csv.gz | wc -l   # 20
du -sh data/in                # ~360M

Related Entries

• gaia-fast – isal +
  ProcessPoolExecutor, ~1s
• gaia-iml – IntegratedML PREDICT()
  plus an AI Hub agent (+6 bonus pts)