currently: building a Databricks lakehouse + multi-agent AI assistant for global trading P&L
Selected work
(05) · click a rowLTIMindtree · Specialist, Data Engineer
A Databricks lakehouse for global trade, trade-lot and ledger data that powers profit & loss reporting.
- Medallion (bronze/silver/gold) Delta Lake layers built with PySpark and SQL
- Iceberg tables queried through Trino for engine interoperability
- Ingestion + transformation pipelines with data-quality checks and trade/lot balance reconciliation against prod
- Orchestrated with Databricks Workflows, governed with Unity Catalog access controls
- Spark + Delta tuning for performance and cost: partitioning, file compaction, cluster sizing
Off-hours pipelines
dag: life_v1 · schedule: dailyOutside work I run three jobs. They depend on each other: clean water makes better coffee, and coffee fuels the training.
job 01 · house_water_etl · DIY
● running since [MONTH YEAR]
The only pipeline I've built with real pipes
I installed a whole-house 3-stage filtration system and a water softener in my townhouse myself. Turns out my B.Tech in Chemical Engineering was training for this all along. [A LINE ON THE HARDEST PART]
source
City main
stage 1
[FILTER TYPE]
stage 2
[FILTER TYPE]
stage 3
[FILTER TYPE]
transform
Softener
sink
Every tap + the espresso machine
job 02 · brew_log
upstream: house_water_etl ✓
Coffee, taken seriously
SELECT method, dose, yield, time
FROM brew_log
ORDER BY cups_per_week DESC;
| method | dose | yield | time |
|---|---|---|---|
| espresso | [__] g | [__] g | [__] s |
| flat_white | [__] g | [__] ml milk | [__] s |
| v60 | [__] g | [__] g water | [_:__] |
3 rows · current beans: [ROASTER / ORIGIN]
job 03 · training_metrics
upstream: brew_log ✓
Hybrid athlete: lift heavy, run far
bench 1RM
225 lb
5K
<25 min
longest run
15 km
Benching more than 1.4× bodyweight and still running a 15K. [NEXT GOAL, e.g. half marathon or 250 bench]
Writing
Notes on data engineering and building for the web, mostly things I had to learn the hard way.
all posts →$ whoami
arpit — specialist, data engineer @ LTIMindtree
$ tail career.log
2016 analyst · exusia
2018 senior analyst · exusia
2019 consultant · exusia
2022 specialist, data engineer · ltimindtree
$ cat interests.txt
v60, espresso, flat whites, lifting, running, DIY plumbing
$ ls ./stack
databricks/ pyspark/ sql/ iceberg/ trino/ ab-initio/ aws/
$ █
About
Ten years building enterprise data platforms, from Ab Initio ETL and real-time continuous flows to Databricks lakehouses and multi-agent AI assistants. I like turning messy, high-stakes data (think global trading P&L) into pipelines people can trust.
Chemical engineer by degree (NIT Trichy), data engineer by trade, and the person who explains both over a flat white.
résumé.pdf ↗