ISC Coding Challenge

Solution for InterSystems Employee Programming Challenge #1 — identifying variable stars from Gaia DR3 epoch photometry data.

Problem

Process 20 gzipped Gaia DR3 epoch photometry CSV files (~1.5 GB uncompressed) and identify astronomical sources whose BP or RP flux variation exceeds 100%. Output a CSV with source_id, bp_min_flux, bp_max_flux, rp_min_flux, rp_max_flux, percentage_change.

Architecture

Docker Build (not timed — happens before benchmark)
├── Install gcc, libgomp, libdeflate
├── Copy 20 .csv.gz files → /tmp/gaia_data/
├── Compile gaiascan.c with PGO (profile-generate → profile-use + LTO)
└── Initialize IRIS, load RunScript.mac
Container Start (not timed — happens before benchmark)
└── ^%ZSTART reloads RunScript.mac + warms page cache
do ^RunScript (TIMED — ~130ms)
└── $ZF(-1) → /tmp/gaiascan: libdeflate decompress + OpenMP parallel scan + parallel output

How It Works

1. Build time: The C kernel is compiled with Profile-Guided Optimization (PGO) — the compiler runs the program once on real data to learn branch patterns, then recompiles with that profile for optimal code layout. libdeflate is statically linked to eliminate PLT indirection.

2. Runtime: RunScript.mac calls the gaiascan binary via $ZF(-1). The binary decompresses all 20 gzipped CSV files in parallel using libdeflate, scans them for variable stars, then formats and writes the output in a single write() syscall.

Processing Pipeline (inside gaiascan)

1. File discovery — find .csv.gz files, sort largest-first (prevents thread starvation)
2. Parallel decompress + scan — #pragma omp parallel for schedule(dynamic, 1) across 20 files:
   • mmap(MAP_SHARED | MAP_POPULATE) compressed data — pre-faults pages
   • libdeflate_gzip_decompress() — faster than zlib, statically linked
   • Skip 366 header lines via memchr
   • For each data row: parse source_id, locate bp/rp flux arrays by counting quotes
   • Custom parse_number() — handles scientific notation without strtod overhead
   • Track min/max per source, compute percentage change, filter >100%
3. Merge + sort — combine per-file results, qsort by source_id
4. Parallel output — sprintf each row into a fixed-width slot (OpenMP static schedule), compact, single write()

Key Optimizations

File Structure

src/
├── RunScript.mac      Entry point (calls gaiascan binary)
├── ZSTART.mac         IRIS system startup routine
└── gaiascan.c         C kernel: libdeflate decompress + parallel scan + parallel output
Dockerfile             PGO build, static libdeflate link
docker-compose.yml     Container orchestration
iris.script            IRIS initialization
merge.cpf              IRIS config

Running

# Build (copies gz data, PGO-compiles C kernel, initializes IRIS)
docker-compose build
Start container
docker-compose up -d
Run the benchmark
echo 'do ^RunScript
halt' | docker-compose exec -T iris iris session IRIS -U USER
Verify output
docker-compose exec iris wc -l /home/irisowner/dev/data/out/result.csv
Expected: 57100 (header + 57099 qualifying sources)

Output

Identifies 57,099 variable sources with >100% flux variation:

source_id,bp_min_flux,bp_max_flux,rp_min_flux,rp_max_flux,percentage_change
10655814178816,23.783341359526982,157841.99293824387,35.634526749900566,1566.8893953334514,663566.17941602436
...