Skip to content

Senior Data Engineer

Adarsh Raj

I design and own the data platforms behind fintech, insurance, healthcare, and supply chain products. 6+ years building production systems end to end, from ingestion and orchestration through warehousing, modeling, and the APIs and dashboards teams rely on. I have built alongside founders and CTOs at YC and Techstars backed startups, working fully remote since 2020.

  • 6+ years of Expertise
  • Worked at YC- and Techstars-backed startups
  • Fintech, InsurTech, Healthcare, Logistics, AI-Powered SAAS
  • Remote since 2020
See my workGet in touchDownload resume
01

About

I'm a Senior Data Engineer based in Bengaluru, India with 6+ years spent designing and owning scalable data platforms, pipelines, and cloud infrastructure. I build production-grade systems end to end, from ingestion and orchestration to warehousing, modeling, APIs, and the operational workflows a business runs on. What I care about is building platforms that stay reliable and easy for teams to extend as products, headcount, and data volumes grow.

My work covers the full data engineering lifecycle, including ETL and ELT pipeline development, warehouse architecture and data modeling, orchestration with Airflow and Dagster, real-time processing with Kafka and Spark, analytics engineering with dbt, and backend and API work in Python and FastAPI. I move comfortably between batch and event-driven systems, and I have shipped on Snowflake, BigQuery, Redshift, and Databricks across AWS, GCP, and Azure.

I started out working directly with founders and CTOs at YC and Techstars backed startups, making architecture calls from day one and delivering systems other engineers could trust, operate, and extend. I am comfortable owning ambiguous problems, aligning technical decisions with what the product actually needs, and delivering with a high degree of autonomy. Fully remote since 2020, I work well across time zones and communicate proactively from design through delivery.

Adarsh Raj, Senior Data Engineer.
02

Metrics

  • 6x. Faster Data Operations.
  • 40%. Cloud Cost Reduced.
  • 50 TB. Largest Data Migration Delivered.
  • 50M+. Payment Events Processed Per Day.
  • 30+. Analytics Stakeholders Served With Self-Service.
  • 4 hours to 50 seconds. Warehouse Query Time Cut.
  • 15,000. Transactions Per Second Handled.
  • 12B. Rows Processed In A Single Project.
  • sub-50ms. API Response At Cache Hit.
  • 6 hours to 10 minutes. Reporting Runtime Cut.
03

Skills

Data Platforms and Lakehouse

  • Databricks
  • Microsoft Fabric

Pipelines and Ingestion

  • Apache Airflow
  • Dagster
  • dbt
  • Apache Spark
  • Apache Kafka
  • RabbitMQ
  • Fivetran
  • Airbyte
  • Matillion

Warehouse and Databases

  • Snowflake
  • BigQuery
  • Redshift
  • PostgreSQL
  • MySQL
  • MongoDB

Backend and APIs

  • Python
  • FastAPI
  • Node.js
  • SQL

AI and LLMs

  • OpenAI
  • Anthropic Claude
  • Google Gemini

Automation

  • Airtable
  • Make.com
  • n8n
  • Zapier

Cloud

  • AWS
  • GCP
  • Azure

DevOps

  • Docker
  • Kubernetes

BI and Analytics

  • Looker
  • Streamlit
  • Metabase
  • Tableau
  • Power BI
04

Featured work

  • Data architecture diagram for the real-time transaction monitoring and payment intelligence platform.

    Real-Time Transaction Monitoring and Payment Intelligence Platform

    50,000+ events per day, sub-second end-to-end latency, exactly-once delivery

    I architected and developed an event-driven platform that watches payments as they happen and flags fraud within sub-second latency. Card and gateway events from Stripe, payment gateways, and internal services land in Apache Kafka, and Apache Flink runs the fraud and enrichment logic with sliding-window pattern detection, idempotency keys, and a customer-metadata join against PostgreSQL. Results fan out to a live WebSocket dashboard, a Snowflake warehouse for compliance history, and banking webhook callbacks, with failed events routed to a dead-letter queue. It runs multi-AZ with exactly-once delivery, S3 checkpointing for recovery, and Prometheus and Grafana for observability.

    • Apache Kafka
    • Apache Flink
    • Snowflake
    • PostgreSQL
    • FastAPI
    • WebSocket
    • Prometheus
  • Data architecture diagram for the customer analytics data warehouse on Snowflake.

    Customer Analytics Data Warehouse with Medallion Architecture on Snowflake

    4 hours to 50 seconds query time, 30+ self-service stakeholders, daily refresh

    I created a medallion-architecture warehouse that turns raw MySQL, Shopify, and Google Analytics data into business-ready facts. Airflow orchestrates ingestion, transformation, and quality DAGs, and dbt handles standardization with SCD Type 2 dimensions in the silver layer and orders, returns, and customer-lifetime-value facts in gold. dbt tests run on every pull request, lineage is auto-documented, and Snowflake zero-copy cloning keeps development and production clean. Query times dropped from 4 hours to 50 seconds for more than thirty self-service stakeholders.

    • Snowflake
    • dbt
    • Apache Airflow
    • Medallion architecture
    • SCD Type 2
    • Tableau
    • Looker
    • Power BI
  • Data architecture diagram for the shipment data API and logistics serving platform.

    High-Performance Shipment Data API and Logistics Serving Platform

    10,000+ requests per minute, sub-50ms cache-hit latency, 100% request audit coverage

    I developed a FastAPI serving layer that delivers shipment predictions and route optimizations to mobile apps, partner APIs, and internal dispatch, holding sub-50ms responses under load. A Redis caching layer with TTL expiry sits in front of PostgreSQL, the data warehouse, and an ML route model, while JWT auth with per-consumer RBAC and token-bucket rate limiting protect every endpoint. It runs on a multi-AZ Kubernetes cluster with horizontal pod autoscaling, rolling deploys, and full request auditing through Prometheus, Grafana, and structured logs.

    • FastAPI
    • Redis
    • Kubernetes
    • PostgreSQL
    • JWT and RBAC
    • Prometheus
    • Grafana
  • Data architecture diagram for the enterprise data lakehouse for manufacturing IoT and ERP consolidation.

    Enterprise Data Lakehouse for Manufacturing IoT and ERP Consolidation

    2M+ sensor events per day, streaming and batch unified, one governed source of truth

    I designed and implemented a lakehouse that consolidates high-volume manufacturing IoT sensor streams and ERP records into one governed analytics platform. Streaming sensor data and batch ERP extracts are unified on Databricks and Microsoft Fabric with a medallion layout, so operations and finance work from the same source of truth instead of siloed exports. It is built for both real-time monitoring and historical reporting at manufacturing scale.

    • Databricks
    • Microsoft Fabric
    • Apache Spark
    • Apache Kafka
    • Delta Lake
    • ERP integration
  • Data architecture diagram for the healthcare data governance and quality framework.

    Data Governance and Quality Framework for Healthcare Compliance and Observability

    200+ automated dbt tests, column-level lineage, HIPAA-compliant audit trail

    I worked on a governance and data-quality framework for a HIPAA-sensitive healthcare environment. It layers automated dbt tests, schema and freshness checks, column-level lineage, and access controls over the warehouse, with alerting and observability so data issues surface before they reach a report. The result is auditable, compliant data that teams can trust for clinical and operational decisions.

    • dbt
    • Data quality
    • Column-level lineage
    • HIPAA
    • Observability
    • Access control
  • Data architecture diagram for the real-time telematics and underwriting platform.

    Real-Time Telematics and Underwriting Platform

    10M+ telematics events per day, data contracts and SLAs enforced, live underwriting decisions

    I architected real-time telematics and driver-behavior pipelines that feed live underwriting decisions at a commercial trucking insurance company. Dagster orchestrates the ELT, FastAPI services surface underwriting and operational data to internal tools, and clear data contracts, schema standards, and SLAs keep the numbers trustworthy across data science and product.

    • Dagster
    • FastAPI
    • Snowflake
    • Telematics
    • ELT
  • Data architecture diagram for the unified operational reporting and automation layer.

    Unified Operational Reporting and Automation Layer

    3 sources consolidated daily, dbt-tested reporting layer, 90% less manual handling

    For a lean remote team in East African FMCG and supply chain, I owned the full data and automation stack. Airflow pipelines consolidated PostgreSQL, MongoDB, and Airtable into a single reporting layer on AWS with dbt transformations, Python services and automations cut manual handling, and Looker dashboards gave operations real financial and inventory visibility.

    • Apache Airflow
    • dbt
    • AWS
    • Python
    • Looker
  • Data architecture diagram for the AI-powered compliance platform.

    AI-Powered Compliance Platform, Founding Build

    10K+ compliance checks automated, OpenAI and Gemini routing, confidence-gated review

    As founding engineer at a Techstars-backed compliance startup, I built the backend from scratch in Python and FastAPI on GCP, owned the data layer across Supabase and BigQuery with Metabase reporting, and integrated OpenAI and Gemini to give the product real AI-driven compliance and automation. I set the GCP architecture and wrote the specs that let the system scale past the founding build.

    • Python
    • FastAPI
    • GCP
    • BigQuery
    • OpenAI
    • Gemini
05

Experience

  1. February 2025 to Present

    Now

    Cover Whale

    Senior Data Engineer

    New York, United States (Remote)

    Real-time data platform work for a US commercial trucking insurance product, from telematics ingestion through underwriting analytics.

    • Architected and maintained real-time telematics and driver-behavior pipelines on Snowflake, ingesting high-volume vehicle and driver event streams into underwriting workflows.
    • Owned the Dagster orchestration repository end to end, leading pipeline architecture, deployment, and production operations across the data platform.
    • Designed dimensional data models in Snowflake with dbt for policy, claims, and operational analytics, with version-controlled transformations and automated lineage in dbt Cloud.
    • Built FastAPI services that surfaced underwriting and operational data to internal tools and downstream systems.
    • Shipped dbt CI/CD through GitHub Actions, cutting transformation deployment cycles and reducing data incidents reaching production.
    • Defined data contracts, schema standards, and pipeline SLAs with data science and product teams.

  2. January 2024 to January 2025

    Ramani.io (YC W20)

    Senior Data and Automation Engineer

    Dar es Salaam, Tanzania (Remote)

    Owned the full data and automation stack for a YC-backed East African supply chain platform, on a lean remote team.

    • Engineered scalable ETL pipelines with Apache Airflow and Apache Kafka, processing 5TB+ across MongoDB and PostgreSQL to power real-time supply chain analytics.
    • Consolidated PostgreSQL, MongoDB, and Airtable into a unified reporting layer on AWS, with dbt driving the core transformations.
    • Established SLA monitoring and pipeline observability across MongoDB and PostgreSQL workloads, reducing incidents and improving operational reliability.
    • Built Python services and Airtable automations that cut manual handling and tightened cross-functional workflows.
    • Developed Looker dashboards and reporting models used by operations and business stakeholders for financial and inventory visibility.
    • Mentored a team of 3 junior data engineers through 1-on-1 sessions and structured workshops on advanced ETL patterns and big data tooling.

  3. June 2023 to December 2023

    Complya (Techstars '24)

    Founding Senior Software Engineer

    Minnesota, United States (Remote)

    Founding engineer who built the backend, data layer, and cloud architecture from scratch for an AI-powered compliance platform.

    • Built the backend from scratch in Python and FastAPI, with production services deployed on GCP through Cloud Run and Cloud Functions.
    • Designed and owned the data layer across Supabase and BigQuery, with Metabase powering reporting and business visibility.
    • Integrated OpenAI and Gemini APIs into compliance and automation features, giving the product real AI-driven capability.
    • Led GCP architecture and deployment decisions and drove a 40% cloud cost reduction while maintaining high availability.
    • Wrote architecture documentation and handoff specs so the system could scale cleanly beyond the founding build, and mentored a team of 2 engineers on code review and deployment standards.

  4. September 2020 to June 2023

    Alopa Infotech

    Software Engineer to Senior Software Engineering Consultant

    Bengaluru, India

    Grew from junior engineering into senior backend and data-infrastructure ownership on a HIPAA-compliant healthcare product.

    • Pioneered end-to-end Snowflake architecture and pipelines with Apache Airflow and dbt, establishing the modern data stack and powering real-time analytics for product and operations teams.
    • Accelerated data operations 6x and cut data discrepancies by 50% through query optimization and pipeline re-architecture across Snowflake.
    • Led migration of legacy Oracle workloads to Snowflake with a team of 4 engineers, cutting daily report runtime from 6 hours to 10 minutes and storage costs by 25%.
    • Optimized Snowflake architecture for security and performance, reducing system downtime by 70%.
    • Built Python REST APIs with Django REST Framework and FastAPI for a HIPAA-compliant healthcare product on AWS and MySQL, scaling to handle 3x user load.

  5. January 2020 to July 2020

    Everlytics Data Science

    Data Engineer

    Bengaluru, India

    Built ingestion, transformation, and reporting foundations across AWS, Airflow, and Snowflake.

    • Designed and implemented ETL and ELT pipelines using AWS, Apache Airflow, and Snowflake for ingestion, transformation, and orchestration.
    • Developed Python data-processing scripts supporting pipeline automation and data-quality workflows.
    • Built and maintained Tableau dashboards to surface insights and reporting for business stakeholders.
06

Certifications

Earned

  • Dataiku DSS Certified Developer
  • Google Analytics (Expert)
  • Google Data Studio
  • Certified Airtable Expert

In progress

  • Snowflake SnowPro Core (COF-C03)
  • Snowflake Advanced Data Engineer (DEA-C02)
  • Snowflake Advanced Architect (ARA-C01)

Education

KLE Technological University

Bachelor of Engineering in Computer Science and Data Engineering

Hubballi, Karnataka, India

07

Let's talk

I build data platforms that stay reliable and easy to extend as teams and data grow, and I own them from first design to production. If you are hiring a data engineer you can hand the hard problems to, I am ready to make the case.