ETL Databricks with AWS(PySpark/SQL)
Virtusa
Date: 19 hours ago
City: Chennai, Tamil Nadu
Contract type: Full time
Develop and maintain a metadata driven generic ETL framework for automating ETL code
Design, build, and optimize ETL/ELT pipelines using Databricks (PySpark/SQL) on AWS .
Ingest data from a variety of structured and unstructured sources (APIs, RDBMS, flat files, streaming).
Develop and maintain robust data pipelines for batch and streaming data using Delta Lake and Spark Structured Streaming.
Implement data quality checks, validations, and logging mechanisms.
Optimize pipeline performance, cost, and reliability.
Collaborate with data analysts, BI, and business teams to deliver fit for purpose datasets.
Support data modeling efforts (star, snowflake schemas) de norm tables approach and assist with data warehousing initiatives.
Work with orchestration tools Databricks Workflows to schedule and monitor pipelines.
Follow best practices for version control, CI/CD, and collaborative development
Skills
Hands-on experience in ETL/Data Engineering roles.
Strong expertise in Databricks (PySpark, SQL, Delta Lake), Databricks Data Engineer Certification preferred
Experience with Spark optimization, partitioning, caching, and handling large-scale datasets.
Proficiency in SQL and scripting in Python or Scala.
Solid understanding of data lakehouse/medallion architectures and modern data platforms.
Experience working with cloud storage systems like AWS S3
Familiarity with DevOps practices Git, CI/CD, Terraform, etc.
Strong debugging, troubleshooting, and performance-tuning skills.
Design, build, and optimize ETL/ELT pipelines using Databricks (PySpark/SQL) on AWS .
Ingest data from a variety of structured and unstructured sources (APIs, RDBMS, flat files, streaming).
Develop and maintain robust data pipelines for batch and streaming data using Delta Lake and Spark Structured Streaming.
Implement data quality checks, validations, and logging mechanisms.
Optimize pipeline performance, cost, and reliability.
Collaborate with data analysts, BI, and business teams to deliver fit for purpose datasets.
Support data modeling efforts (star, snowflake schemas) de norm tables approach and assist with data warehousing initiatives.
Work with orchestration tools Databricks Workflows to schedule and monitor pipelines.
Follow best practices for version control, CI/CD, and collaborative development
Skills
Hands-on experience in ETL/Data Engineering roles.
Strong expertise in Databricks (PySpark, SQL, Delta Lake), Databricks Data Engineer Certification preferred
Experience with Spark optimization, partitioning, caching, and handling large-scale datasets.
Proficiency in SQL and scripting in Python or Scala.
Solid understanding of data lakehouse/medallion architectures and modern data platforms.
Experience working with cloud storage systems like AWS S3
Familiarity with DevOps practices Git, CI/CD, Terraform, etc.
Strong debugging, troubleshooting, and performance-tuning skills.
How to apply
To apply for this job you need to authorize on our website. If you don't have an account yet, please register.
Post a resumeSimilar jobs
Senior Software Quality Engineer
UPS,
Chennai, Tamil Nadu
19 hours ago
Avant de postuler à un emploi, sélectionnez votre langue de préférence parmi les options disponibles en haut à droite de cette page.Découvrez votre prochaine opportunité au sein d'une organisation qui compte parmi les 500 plus importantes entreprises mondiales. Envisagez des opportunités innovantes, découvrez notre culture enrichissante et travaillez avec des équipes talentueuses qui vous poussent à vous développer chaque jour....
Brand & Marketing Designer (Part-time / Freelance / Full-time )
Tagalys,
Chennai, Tamil Nadu
20 hours ago
Job Title: Brand & Marketing DesignerLocation: Remote / Chennai Company: TagalysEngagement: Part-time / Freelance / Full-time Compensation: Based on engagement, (5-8.5 LPA)About TagalysTagalys is a visual merchandising platform built for fashion and lifestyle brands on Shopify. We work with brands like Fila, Crocs, Lovisa, Timex, Meshki, and many more across the UK, US, Australia, and beyond.We’re a bootstrapped, profitable startup helping...
Technical Writer
Zoho,
Chennai, Tamil Nadu
4 days ago
Zoho is one of the world's most prolific software companies. With 55+ applications in nearly every major business category, including sales, marketing, customer service, accounting and back office operations, and an array of productivity and collaboration tools built from the ground up, Zoho has the depth and breadth to solve even the most complex business challenges.With more than 130 million+...
