Keep up with latest product news








process.py)Uses skip_rows=365 and has_header=True instead of comment_prefix="#". Polars no longer scans and discards 365 header lines per file; it seeks directly to the data. The filter(source_id != "solution_id") guard is also eliminated since the comment block never enters the dataframe.
Files are sorted by descending size before dispatch. On 8 cores (local CPU) with 20 files this follows LPT (Longest Processing Time) scheduling, minimizing the final idle-thread stall when the last batch of files completes.
max_workers = min(20, os.cpu_count()) — avoids 20 threads competing for 8 cores and reduces context-switching overhead compared to the hardcoded max_workers=len(files) in the baseline.
POLARS_MAX_THREADS=1 prevents polars' internal Rayon thread pool from spawning additional threads per file read, which would over-subscribe the CPU when multiple Python worker threads are already running concurrently.
Changed the runtime Docker volume layout so that only data/out is bind-mounted. Nothing is precomputed, all work stays in the timed run.
Found some more optimizations. Using zgrep now, using csv.reader, some other small changes. Down to 378 meaningful code characters.
Using docker mounts, moved file paths out of Python execution, bringing total RunScript.mac character count down to 452 characters, 433 of which are "meaningful code" (assuming "meaningful code" means "everything after ROUTINE RunScript")
