ISC Coding Challenge

Solution for InterSystems Employee Programming Challenge #1 — identifying variable stars from Gaia DR3 epoch photometry data.

Problem

Process 20 gzipped Gaia DR3 epoch photometry CSV files (~1.5 GB uncompressed) and identify astronomical sources whose BP or RP flux variation exceeds 100%. Output a CSV with source_id, bp_min_flux, bp_max_flux, rp_min_flux, rp_max_flux, percentage_change.

Architecture

Docker Build (not timed — happens before benchmark)
├── Install gcc, libc6-dev, libgomp1
├── Decompress 20 .csv.gz → /tmp/gaia_data/*.csv
├── Compile gaiascan.c with PGO (profile-generate → profile-use + LTO)
└── Initialize IRIS, load %ZSTART + RunScript.mac
Container Start (not timed — happens before benchmark)
└── ^%ZSTART reloads RunScript.mac + warms page cache (cat CSVs into RAM)
do ^RunScript (TIMED)
└── $ZF(-1) → /tmp/gaiascan: OpenMP parallel scan + parallel output

How It Works

1. Build time: Dataset is gunzipped into /tmp/gaia_data/ and the C kernel is compiled with Profile-Guided Optimization (PGO) — the compiler runs once on real data, then recompiles with that profile for optimal code layout.

2. Container start: %ZSTART reloads RunScript.mac and reads all CSV files into the OS page cache so the timed run does not pay cold I/O.

3. Runtime (timed): RunScript.mac calls /tmp/gaiascan via $ZF(-1). The binary memory-maps the already-decompressed CSVs, scans them in parallel for variable stars, then formats and writes the output in a single write() syscall.

Processing Pipeline (inside gaiascan)

1. File discovery — find .csv files, sort largest-first (prevents thread starvation)
2. Parallel scan — #pragma omp parallel for schedule(dynamic, 1) across 20 files:
   • mmap(MAP_SHARED | MAP_POPULATE) — maps decompressed data
   • Skip 366 header lines via memchr
   • For each data row: parse source_id, locate bp/rp flux arrays by counting quotes
   • Custom parse_number() — handles scientific notation without strtod overhead
   • Track min/max per source, compute percentage change, filter >100%
3. Merge + sort — combine per-file results, qsort by source_id
4. Parallel output — sprintf each row into a fixed-width slot (OpenMP static schedule), compact, single write()

Key Optimizations

File Structure

src/
├── RunScript.mac      Entry point (timed call to gaiascan)
├── ZSTART.mac         Startup: reload routine + warm page cache
└── gaiascan.c         C kernel: OpenMP parallel scan + parallel output
Dockerfile             Build-time decompress + PGO compile
docker-compose.yml     Container orchestration
iris.script            IRIS initialization
merge.cpf              IRIS config
dataset/               20 Gaia DR3 .csv.gz input files

Running

# Build (decompresses data, PGO-compiles C kernel, initializes IRIS)
docker compose build
Start container (%ZSTART warms page cache — not timed)
docker compose up -d
Run the benchmark (timed)
echo 'do ^RunScript
halt' | docker compose exec -T iris iris session IRIS -U USER
Verify output
docker compose exec iris wc -l /home/irisowner/dev/data/out/result.csv
Expected: 57100 (header + 57099 qualifying sources)

Output

Identifies 57,099 variable sources with >100% flux variation:

source_id,bp_min_flux,bp_max_flux,rp_min_flux,rp_max_flux,percentage_change
10655814178816,23.783341359526982,157841.99293824387,35.634526749900566,1566.8893953334514,663566.17941602436
...