Senior Data Engineer

Senior Data Engineer

About the Role

The Senior Data Engineer in our AI & Data team will be responsible for designing and building

scalable data platforms, enterprise-grade data architectures, and high-performance data

ingestion frameworks across Azure, Snowflake, Databricks, and Lakebase.

This role requires a highly technical engineer capable of solving complex data platform

challenges involving large-scale API integrations, distributed data processing, cloud-native

architectures, and AI-enabled data platforms.

The ideal candidate is not only a Data Engineer but also a strong Python engineer with expertise

in data architecture, system design, API engineering, concurrent processing, and enterprise

scale platform development.

Main Responsibilities

• Design and implement scalable enterprise data architectures supporting AI, analytics,

reporting, and operational workloads.

• Build and optimize large-scale ELT/ETL pipelines using Databricks, Snowflake, Azure Data

Factory, and Azure services.

• Design and implement Medallion Architectures (Bronze/Silver/Gold), CDC frameworks,

lineage models, and master data management workflows.

• Develop high-performance Python-based ingestion frameworks supporting large-scale

extraction from internal and external data sources.

• Build and maintain REST API and GraphQL integrations including OAuth/OAuth2

authentication, token lifecycle management, pagination, retry mechanisms, rate limiting,

and automated error recovery.

• Design parallelized ingestion solutions using multi-threading, asynchronous processing, and

distributed execution patterns.

• Develop and maintain Databricks pipelines, Delta Lake architectures, and Snowflake

analytical data platforms.

• Build and support real-time and near-real-time data processing solutions using event-driven

architectures.

• Implement data quality, reconciliation, lineage, observability, and governance frameworks

across the platform.

• Design scalable data models supporting analytics, machine learning, feature engineering,

and AI workloads.

• Monitor production environments, troubleshoot pipeline failures, optimize platform

performance, and drive cost optimization initiatives.

• Collaborate closely with AI Engineers, Architects, Product Teams, and Business Stakeholders.

Required Skills & Experience

Python (Mandatory)

• Strong production-grade Python development experience.

• Object-Oriented Programming (OOP).

• Modular framework development.

• Logging, exception handling, testing, and debugging.

• Performance optimization and profiling.

• Experience building reusable ingestion and transformation frameworks.

Advanced API Engineering (Mandatory)

• REST APIs.

• GraphQL APIs.

• OAuth/OAuth2.

• JWT Authentication.

• API Pagination.

• Rate Limiting.

• Retry Logic.

• Token Refresh Handling.

• Error Handling Frameworks.

• High-volume API ingestion architecture.

Data Architecture & System Design (Mandatory)

• Medallion Architecture.

• Data Warehouse Architecture.

• Data Lakehouse Architecture.

• Master Data Management (MDM).

• Golden Record Design.

• Data Lineage.

• Change Data Capture (CDC).

• Historical Data Management.

• Enterprise Data Modeling.

Data Pipeline Design (Mandatory)

• Design and build scalable, fault-tolerant enterprise data pipelines.

• Strong experience with batch, near real-time, and event-driven processing architectures.

• Expertise in designing ingestion, transformation, validation, reconciliation, and serving

layers across modern data platforms.

• Experience implementing Medallion Architecture (Bronze, Silver, Gold) and data lakehouse

patterns.

• Strong understanding of Change Data Capture (CDC), incremental processing, watermarking

strategies, and SCD Type 1/Type 2 implementations.

• Design audit frameworks, lineage tracking, reconciliation controls, monitoring, alerting, and

observability solutions.

• Ability to build high-volume ingestion pipelines from APIs, databases, files, data streams,

and external systems.

• Experience designing resilient pipelines with retry mechanisms, checkpointing, idempotent

processing, restartability, and failure recovery.

• Strong understanding of throughput optimization, parallel processing, concurrency, and

workload orchestration.

• Experience designing data movement patterns across Snowflake, Databricks, Azure services,

and downstream analytics platforms.

Databricks (Mandatory)

• Databricks Workflows.

• Delta Lake.

• Unity Catalog.

• PySpark.

• Spark SQL.

• Databricks Performance Tuning.

• Distributed Data Processing.

Snowflake (Mandatory)

• Snowpipe.

• Streams.

• Tasks.

• Dynamic Tables.

• Time Travel.

• Zero-Copy Cloning.

• RBAC.

• Query Optimization.

• Warehouse Optimization.

• Cost Management.

SQL & Data Modeling (Mandatory)

• Advanced SQL.

• CTEs.

• Window Functions.

• Stored Procedures.

• MERGE.

• Star Schema.

• Snowflake Schema.

• SCD Type 1 & Type 2.

• Dimensional Modeling.

Azure (Mandatory)

• Azure Data Factory.

• ADLS Gen2.

• Azure Synapse.

• Event Hubs.

• Azure Integration Services.

Strongly Preferred

• Apache Airflow.

• dbt.

• Kafka.

• Event Driven Architecture.

• Terraform.

• Infrastructure as Code.

• Azure OpenAI.

• AI/ML Data Pipelines.

• Feature Stores.

Git & DevOps (Mandatory)

• Git Branching Strategies.

• Pull Requests.

• Git Rebase.

• Cherry-picking.

• Merge Management.

• Repository Governance.

• CI/CD Pipelines.

• GitHub Actions / Azure DevOps.

• Secret Management.

• Git History Cleanup and Recovery.

Experience

• 5+ years of hands-on Data Engineering experience.

• Proven experience designing production-grade enterprise data platforms.

• Strong experience working directly with business stakeholders and translating requirements

into scalable technical solutions.

• Experience leading architecture discussions and solving complex technical problems

independently.