Spark Agent
An AI assistant for debugging live Spark workloads.
Built end to end
I was the sole engineer building Spark Agent: the Scala/Java driver plugin, a supervised Rust subprocess, and the React interface. The two-process architecture communicated over versioned Unix-domain-socket IPC, with multi-provider LLM orchestration and 34 tools spanning the driver and Spark History Server. The assistant was publicly launched as part of Quanton.
Designed for the running system
A debugging assistant has to coexist with the workload it investigates. I validated runtime impact with TPC-DS and listener overhead with JMH microbenchmarks: measured wall-clock impact was −0.11%, with a p99 listener cost of 233 μs.
Protecting the knowledge layer
I encrypted an 82-document knowledge base using AES-256-GCM, added per-file canary sigils and CI-gated leakage tests, and implemented citation-path validation and per-tenant prompt watermarks.