Job Title:  Airflow Platform Engineer

Job Location:  Remote

Job Duration: Contract

Job Description: 

We are implementing Astronomer-backed Apache Airflow (Astro Private Cloud) as our enterprise orchestration platform on Azure Kubernetes Service (AKS). This platform provides a centralized, scalable scheduling and workflow orchestration layer for data pipelines and integrations across the organization. We are seeking a hands-on engineer to support the platform with responsibilities spanning DAG development enablement and platform-level support. The ideal candidate should be comfortable working across Airflow (3.x+), Kubernetes, and cloud infrastructure, while enabling developers to efficiently adopt the platform.

Experience - 4 - 6 years

Key Responsibilities

DAG Development & Developer Enablement

• Design, develop, and optimize Airflow DAGs for enterprise use cases

• Demonstrate strong experience with operators such as PythonOperator, SQL operators,API/HTTP operators, KubernetesPodOperator etc.

• Apply advanced DAG patterns including dynamic DAGs, branching, sensors, retries, and error handling

• Troubleshoot DAG failures, performance issues, and integration challenges

• Enable orchestration of modern data workflows including DBT pipelines

• Provide reusable templates, best practices, and onboarding support to drive self-service adoption Platform Operations & Kubernetes Support

• Support Airflow platform deployed on AKS, covering scheduler, webserver/API, workers, and metadata services

• Troubleshoot issues related to task execution, scheduling delays, and resource bottlenecks

• Diagnose Kubernetes-level issues involving pods, networking, storage, and RBAC

• Work with Azure services such as Key Vault, Storage, and networking

• Support CI/CD pipelines for DAG and container-based deployments

• Improve platform stability, monitoring, and alerting

Required Skills

• Strong experience with Apache Airflow (3.x or later) in production environments

• Proven experience in DAG development using operators and orchestration patterns

• Working knowledge of Kubernetes (AKS preferred) to support Airflow runtime

• Expert-level Python knowledge for building, debugging, and maintaining production- grade workflows and supporting libraries

• Expertise in writing unit, integration, and DAG validation tests for Airflow workflows, including mocking operators, validating dependencies, and ensuring reliability in CI/CD pipelines

• Experience with DBT and modern data pipelines

• Experience with CI/CD tools (Git, Jenkins, Docker)

• Familiarity with Azure or other cloud platforms

Preferred Skills

• Experience with Astronomer / Astro Private Cloud

• Exposure to data platforms such as Snowflake, and APIs

• Experience with monitoring tools such as Splunk or Prometheus

• Knowledge of secrets management using Azure Key Vault

Summary

This role is critical to ensuring reliable platform operations and scalable adoption of the Airflow-based orchestration platform. The ideal candidate combines strong Airflow expertise, Kubernetes knowledge, and hands-on development experience to support both

platform stability and developer productivity.