Data Engineer – Azure Databricks

Position: Data Engineer – Azure Databricks

Location: Nagpur, Pune

Type of Employment: Full-Time /Part-Time etc.

Purpose of the Position:

To design, build, and optimize scalable data pipelines and solutions using Azure Databricks and related technologies, to make faster, data-driven decisions as part of our clients data transformation journey. We are looking for data engineer with proficiency in data integration techniques, ETL processes and data pipeline architectures and well versed in Data Quality rules, principles and implementation.

Key Result Areas and Activities:

Data Pipeline Development: Design, develop, and maintain scalable and reliable ETL/ELT pipelines using Azure Data Factory and Azure Databricks to support analytical and AI-driven use cases

Data Processing & Transformation: Build, optimize, and manage large-scale data processing workflows using PySpark and Spark SQL, ensuring efficient ingestion, transformation, and validation of diverse data sources

Data Architecture & Storage Management: Implement Medallion Architecture and manage structured data layers within ADLS Gen2, applying hierarchical storage design and ACID-compliant Delta Lake tables

Performance Optimization & Reliability: Monitor, troubleshoot, and optimize pipeline performance for reliability, scalability, and cost efficiency across Spark jobs, Databricks clusters, and data workflows

Collaboration, Automation & Documentation: Collaborate with data architects, analysts, and business stakeholders to deliver high-quality data solutions, drive automation initiatives, support AI/ML use cases, and maintain SOPs and technical documentation

Governance & Security: Ensure adherence to data governance, security standards, and best practices, including cataloging, access control, and compliance requirements