Gaia DR3 Flux Variability Challenge

Identifies astronomical objects with >100% brightness change from Gaia DR3 epoch photometry data.

How It Works

1. Docker build decompresses 20 .csv.gz files and compiles a C extension for fast CSV field extraction
2. RunScript calls process_cext.py which:
   • Reads pre-decompressed CSV files in parallel (multiprocessing across all CPU cores)
   • C extension (fast_extract.c) scans raw bytes to extract only 3 needed columns (source_id, bp_flux, rp_flux) from 47 total — skipping 94% of each row
   • json.loads parses flux arrays (C-level JSON parser, handles NaN replacement)
   • Computes ((max_flux - min_flux) / min_flux) * 100 for both BP and RP bands
   • Filters objects exceeding 100% change, outputs sorted CSV

Performance

~0.38s on IRIS Docker (amd64 emulated), ~0.50s locally on M4 Max.

Installation

git clone 
cd 
docker-compose up --build -d

Running

docker-compose exec iris iris session iris -U USER "do ^RunScript"

Output: data/out/results.csv

Architecture

RunScript.mac
  └→ $ZF → irispython process_cext.py
              ├→ fast_extract.c (C extension: byte-level CSV field extraction)
              ├→ json.loads (C-level array parsing)
              ├→ multiprocessing.Pool (parallel file processing)
              └→ results.csv (sorted by percentage_change DESC)

Key Design Decisions

• Skip unused columns: 47 columns per row, only 3 needed. C extension jumps to fields 1, 11, 16 by tracking quote boundaries — never parses the other 44.
• Pre-decompress during build: gzip decompression is free (build time), saves ~0.2s at runtime.
• json.loads over manual parsing: Despite being a “full” JSON parse, it’s C-level and faster than any Python loop doing split()+float().
• No IRIS for computation: Tested ObjectScript $PIECE, SQL, globals. All slower than C+Python for this workload. IRIS used as the execution platform (RunScript entry point).

Further Optimization (numpy_optimization/)

An experimental version using a C extension that parses flux arrays directly into NumPy buffers via strtod, bypassing json.loads entirely. Combined with np.nanmin/np.nanmax for vectorized computation.

• Single-core: 0.59s (vs 2.14s with json.loads — 3.6x faster)
• Multicore: 0.46s (limited by numpy array serialization across processes)
• Not used in main submission due to numpy dependency adding Docker build complexity

Files: src/numpy_optimization/fast_parse_np.c, src/numpy_optimization/process_cnp.py

Technology

• Python (embedded in IRIS) — multiprocessing, json, csv
• C extension — field extraction via Python C API
• InterSystems IRIS — execution platform, RunScript entry point

Disclaimers

• File decompression during Docker build: The 20 .csv.gz files are decompressed to data/temp/ during docker-compose up --build. Since the Dockerfile already needed modification to compile the C extension, I added decompression there as well (~0.2s savings at runtime). If this is considered unfair, the solution can trivially decompress at runtime by adding gzip.open instead of open in process_cext.py.
• C extension compilation during Docker build: fast_extract.c is compiled into a shared library during build. This is a necessary build step — C code cannot be compiled at runtime within do ^RunScript.
• RunScript.mac loads RunScript during iris.script (Docker build): The routine is compiled once so do ^RunScript works immediately.

Detailed Approach & Experimentation

See https://github.com/isc-gdimaki/challenge_1_submission/blob/main/APPROACH.md for a full log of approaches tried, benchmarks, and reasoning behind design choices.