Job Title: Lead Data Engineer- DataBricks

Location: Milpitas, CA (Local Candidates only)

Work Schedule: Onsite

Job Description:

Job Summary

We are seeking a Lead Data Engineer with deep expertise in Databricks to architect, build, and lead scalable data engineering solutions on cloud-based lakehouse platforms. The role combines hands-on technical leadership with solution design, mentoring, and close collaboration with architects, BI, and AI teams.

Key Responsibilities

Technical Leadership & Architecture

• Lead the design and implementation of Databricks Lakehouse architectures
• Define medallion architecture (Bronze, Silver, Gold layers) using Delta Lake
• Drive architectural decisions for batch and streaming data pipelines
• Establish coding standards, best practices, and reusable frameworks

Data Engineering & Databricks

• Design and build scalable ETL/ELT pipelines using Databricks (PySpark/SQL/Scala)
• Optimize Spark jobs for performance, reliability, and cost
• Implement Delta Lake features (ACID, time travel, schema enforcement)
• Develop and manage Databricks workflows, jobs, and clusters

Cloud & Platform Integration

• Architect Databricks solutions on Azure (preferred) or AWS
• Integrate Databricks with cloud storage and data services
• Azure: ADLS, ADF, Synapse
• AWS: S3, Glue, Redshift
• Enable BI and analytics consumption (Power BI, Tableau)