Gaia_Challenge_Benchmark

intersystems-challenge1-docker-template

Solution for Employee Programming Challenge #1 — identifying variable stars from Gaia DR3 epoch photometry data using InterSystems IRIS.

Prerequisites

• Git
• Docker Desktop

Installation

Clone the repository:

git clone https://github.com/lynnwux/intersystems-challenge1-docker-template.git
cd intersystems-challenge1-docker-template

Build and start the container:

docker-compose up --build -d

The build decompresses the input data and compiles the IRIS routine. On first startup, a background worker daemon pre-loads all data files into memory — this takes about 30–60 seconds. Once ready, the benchmark runs in under 1 second.

Running the benchmark

docker-compose exec iris iris session iris

USER>do ^RunScript

Expected output:

Found 20 files
Objects with >100% flux change: 57099
Output written to: /home/irisowner/dev/data/out/variable_objects.csv
Elapsed time: 0.7 seconds

The output CSV is written to data/out/variable_objects.csv.

How it works

Data pipeline

1. Build time — the Dockerfile decompresses the 20 gzipped Gaia epoch photometry CSV files into /tmp/gaia_data (bypassing the Docker bind-mount, which is slow on Windows hosts).

2. Container startup — CacheRunner.mac is executed automatically via merge.cpf on every container start. It calls cache_data.py, which:

   • Decompresses any files not yet in /tmp/gaia_data
   • Starts gaia_daemon.py as a background process and waits for it to be ready

3. Background daemon (gaia_daemon.py) — the core of the performance optimization:

   • Loads all 20 CSV files (1.5 GB total) into memory using parallel threads
   • Forks a pool of 20 worker processes via multiprocessing.fork; each worker inherits the loaded data via copy-on-write
   • Runs one warm-up pass to force all workers to fault their memory pages
   • Listens on a Unix domain socket (/tmp/gaia_daemon.sock) for RUN commands
   • Returns results using a 4-byte length-prefixed binary protocol

4. do ^RunScript — calls analyze_variability.py via $ZF(-1, ...). The script:

   • Connects to the daemon socket and sends RUN
   • Receives the pre-computed CSV body (~8 MB) via the socket
   • Writes the output file with the standard header
   • Falls back to a direct multiprocessing.Pool run if the daemon is unavailable

Analysis logic (analyze_variability.py / gaia_daemon.py)

Each worker processes one CSV file:

• Parses each row with a compiled regex to extract source_id, bp_flux array, and rp_flux array
• Filters out NaN values and computes min/max flux for each band
• Flags any source whose flux range exceeds 100% of its minimum value ((max − min) / min > 1.0)
• Returns matching rows formatted as CSV

Performance

The daemon eliminates Python interpreter startup, process fork, and disk I/O costs on every benchmark call — the workers are already live with data in memory.