Role: Apache Airflow Technical SME

Location: Raleigh NC OR Phoenix, AZ || Onsite

Position Type: Contract

Years of Experience: 10+ Years

Description:

Strong Python and SQL proficiency, a deep understanding of workflow orchestration and DAG design, and experience with cloud platforms, version control systems (like Git), and monitoring tools. Other essential skills are knowledge of ETL/ELT processes, data modeling, distributed systems, and CI/CD principles.

• Airflow Expertise:
• Possessing deep knowledge of Airflow's architecture, including schedulers, executors (Celery, Kubernetes), and plugin development.
• Workflow Design and Development:
• Designing and developing complex, modular, and reusable DAGs (Directed Acyclic Graphs) to automate data pipelines.
• Performance Optimization:
• Identifying and addressing performance bottlenecks in Airflow environments and implementing best practices for orchestration and scheduling.
• Integration with Cloud Services:
• Integrating Airflow with cloud-native services like Azure Data Factory, Azure Databricks, Azure Storage, and others.
• CI/CD Pipeline Management:
• Developing and maintaining CI/CD pipelines for Airflow DAG deployment, testing, and version control, often using tools like Azure DevOps.
• Monitoring and Alerting:
• Implementing monitoring, alerting, and logging standards for Airflow jobs to ensure operational excellence and rapid incident response.
• Documentation and Knowledge Sharing:
• Creating and maintaining documentation for Airflow configurations, deployment processes, and operational procedures. Mentoring other engineers and leading knowledge-sharing sessions.
• Troubleshooting and Incident Response:
• Providing expert-level troubleshooting support for Airflow-related issues and contributing to incident response efforts.

Additional Responsibilities:

• Cloud Platform Expertise:
• May require strong knowledge of cloud platforms like Azure (including services like Azure Synapse, Azure Functions, and Azure API Management) or AWS.
• Containerization and Orchestration:
• Experience with Docker and Kubernetes for deploying and managing Airflow in containerized environments.
• Programming Skills:
• Proficiency in Python/Java for developing Airflow DAGs and other related components.