ETL / ELT Architect (Azure Data Engineering)

**Job Title: ETL / ELT Architect (Azure Data Engineering)Experience Level** **8–10 years of overall data engineering experience**, with **at least 4–5 years in cloud\-based data platforms** and **2–3 years in an architecture or lead design role**. **Role Overview** The ETL / ELT Architect will lead the design and governance of scalable, secure, and high\-performance data pipelines on **Microsoft Azure**. This role is responsible for defining enterprise\-wide **Bronze → Silver → Gold (Medallion) architecture standards**, Databricks ETL frameworks, and orchestration patterns supporting both batch and streaming workloads. The architect will act as the technical authority for ETL design decisions, performance optimization, schema evolution, and reliability across the data platform. **Key ResponsibilitiesETL / ELT Architecture \& Standards** · Define and govern **Medallion Architecture (Bronze, Silver, Gold)** standards across the program. · Establish **ELT\-first design principles** using Azure Databricks and Delta Lake. · Design reusable, metadata\-driven **ETL frameworks** supporting multiple ingestion patterns. · Define ingestion strategies for **CDC, full loads, and streaming data** from Azure Event Hub and databases. **Databricks \& Delta Lake Architecture** · Design and implement **Databricks Auto Loader** for scalable ingestion with schema drift handling. · Define **merge and upsert strategies** using Delta Lake for Silver and Gold layers. · Establish best practices for: o Schema evolution and validation o Late\-arriving data handling o Idempotent processing · Define Delta Lake maintenance strategies (OPTIMIZE, VACUUM, Z\-ORDER). **Performance \& Optimization** · Define **partitioning strategies** based on data volume, access patterns, and downstream usage. · Optimize Spark workloads for joins, aggregations, and large\-scale transformations. · Ensure efficient cluster sizing and job configuration for cost and performance balance. **Orchestration \& Workflow Design** · Define orchestration approaches using **Azure Data Factory and Databricks Workflows**. · Design dependency management across Bronze, Silver, and Gold pipelines. · Enable parameterized and reusable pipelines supporting multi\-tenant and multi\-source ingestion. **Error Handling, Monitoring \& Reliability** · Define standardized **error handling, retry, and recovery mechanisms**. · Implement data quality checks and validation at each layer. · Design observability using **Azure Monitor and Alerts**. · Ensure pipeline resilience and operational stability. **Governance \& Downstream Enablement** · Align ETL design with **Azure security, governance, and lineage standards** (Microsoft Purview). · Design Gold\-layer data models optimized for **Synapse Dedicated SQL Pool**, reporting, and analytics. · Support secure data sharing through **Azure Data Share** and external consumption platforms. **Required Skills \& ExperienceExperience** · **8–10 years** in data engineering and ETL/ELT development. · **4\+ years** designing and implementing cloud\-based data platforms (Azure preferred). · **2\+ years** in an architecture, lead, or technical design role. **Technical Skills** · Strong expertise in **Azure Databricks architecture** and Spark\-based ETL. · Deep hands\-on experience with **Delta Lake** (MERGE, schema evolution, ACID guarantees). · Experience with **Databricks Auto Loader** for streaming and incremental ingestion. · Proven experience designing **enterprise\-grade ETL frameworks**. · Strong knowledge of **schema drift handling**, CDC patterns, and incremental processing. · Hands\-on experience with **Azure Data Factory** for orchestration. · Expertise in **performance tuning and optimization** for Databricks and Spark workloads. · Experience with **real\-time and streaming data pipelines**. · Exposure to **data migration and legacy system decommissioning** programs. · Strong understanding of **error handling, retry logic, and fault\-tolerant pipeline design**. **Cloud \& Data Platform** · Strong experience with **Azure ADLS Gen2, Event Hub, Databricks, Synapse**. · Familiarity with **Microsoft Purview** or equivalent governance tools. · Experience supporting downstream analytics, reporting, and data sharing use cases. **Soft Skills** · Strong architectural thinking and decision\-making ability. · Ability to define standards and mentor engineering teams. · Excellent communication and documentation skills. · Experience collaborating with platform, security, and analytics stakeholders. **Nice to Have** · Knowledge of CI/CD and DevOps practices for Databricks and data pipelines. · Experience working in large enterprise or multi\-domain data programs. Pay: $80\.00 \- $125\.00 per hour Work Location: Remote