Keep up with latest product news






Update: Consolidated to two meaningful lines
Per feedback that the Python load script (load_gaia.py, 28 lines) counted as meaningful code, the submission has been refactored:
load_gaia.py is deleted entirely — no Python file exists in the submission
All parse/load logic is now a single ObjectScript do statement in LoadGaia.mac, embedding the Python inline as a string argument to ##class(%SYS.Python).Run()
The SQL query in RunScript.mac is unchanged
The submission is now two meaningful lines:
One ObjectScript line — parses raw Gaia epoch photometry CSVs, flattens flux arrays, and loads 5,668,090 rows into table g
One SQL line — finds all 57,099 stars with >100% flux variability in either band
Verified end-to-end: ^LoadGaia → ^RunScript → variable_objects.csv with 57,099 rows.








process.py)Uses skip_rows=365 and has_header=True instead of comment_prefix="#". Polars no longer scans and discards 365 header lines per file; it seeks directly to the data. The filter(source_id != "solution_id") guard is also eliminated since the comment block never enters the dataframe.
Files are sorted by descending size before dispatch. On 8 cores (local CPU) with 20 files this follows LPT (Longest Processing Time) scheduling, minimizing the final idle-thread stall when the last batch of files completes.
max_workers = min(20, os.cpu_count()) — avoids 20 threads competing for 8 cores and reduces context-switching overhead compared to the hardcoded max_workers=len(files) in the baseline.
POLARS_MAX_THREADS=1 prevents polars' internal Rayon thread pool from spawning additional threads per file read, which would over-subscribe the CPU when multiple Python worker threads are already running concurrently.