Job Title: Data Solutions Architect – Azure Databricks

Remote

Role Summary

The Data Solutions Architect is a deeply hands-on technical leader responsible for designing and delivering the end-to-end Azure Databricks platform architecture for a critical Snowflake replatforming initiative in the legal industry. This is not a purely advisory role: the architect is expected to spend approximately 25% of their time writing production code, building reference implementations, and conducting peer code reviews alongside the engineering team. Working as a peer to the Technical Lead, this role owns the architectural vision: from metadata-driven ingestion pipeline patterns through Unity Catalog governance, BI connectivity, and enterprise integrations: and translates that vision into detailed, executable architecture diagrams and standards that guide a team of 8+ Senior Data Engineers. This is a fully remote, independent contractor engagement billed at an hourly rate.

Solution Architecture & Design

• Own end-to-end Azure Databricks platform architecture: from raw ingestion through Medallion Lakehouse layers (Bronze/Silver/Gold) to BI consumption: producing detailed, fully documented architecture diagrams that address both functional and non-functional requirements (NFRs: scalability, availability, security, performance, maintainability).
• Design the metadata-driven pipeline framework governing all ingestion patterns: on-premises SQL Server (JDBC, incremental/CDC), REST/SOAP API integrations (auth, pagination, rate-limiting, retry), and flat file ingestion (CSV, JSON, XML via ADLS Gen2/SFTP landing zones).
• Architect the migration path from legacy Snowflake and Azure Data Factory (ADF) to native Databricks tooling: including Databricks Workflows, Databricks Autoloader, and Delta Live Tables: minimizing replatforming risk while maximizing platform capability adoption.
• Define metadata store design (pipeline configuration tables, ingestion control frameworks) that enables engineers to onboard new data sources without bespoke pipeline code.
• Establish Unity Catalog architecture: workspace federation, catalog/schema/table hierarchy, data classification, column-level security, row filters, and audit log strategy.
• Design Collibra integration patterns for bidirectional lineage, data stewardship workflows, and business glossary synchronization with Unity Catalog.
• Architect Okta/Azure Active Directory integration for platform authentication, service principal management, and SCIM provisioning into Databricks and Unity Catalog.