Senior Data Engineer

The Sr AWS/Azure/GCP Databricks Data Engineer at Koantek will use comprehensive modern data engineering techniques and methods with Advanced Analytics to support business decisions for our clients. Your goal is to support the use of data-driven insights to help our clients achieve business outcomes and objectives. You can collect, aggregate, and analyze structured/unstructured data from multiple internal and external sources and patterns, insights, and trends to decision-makers. You will help design and build data pipelines, data streams, reporting tools, information dashboards, data service APIs, data generators, and other end-user information portals and insight tools. You will be a critical part of the data supply chain, ensuring that stakeholders can access and manipulate data for routine and ad hoc analysis to drive business outcomes using Advanced Analytics. You are expected to function as a productive member of a team, working and communicating proactively with engineering peers, technical leads, project managers, product owners, and resource managers.

Requirements

• Strong experience as an AWS/Azure/GCP Data Engineer and must have AWS/Azure/GCP Databricks experience.
• Expert proficiency in Spark Scala, Python, and spark.
• Must have data migration experience from on-prem to cloud.
• Hands-on experience in Kinesis to process and analyze Stream Data, Event/IoT Hubs, and Cosmos.
• In-depth understanding of Azure/AWS/GCP cloud and Data lake and Analytics solutions on Azure.
• Expert level hands-on development.
• Design and develop applications on Databricks.
• Extensive hands-on experience implementing data migration and data processing using AWS/Azure/GCP services.
• In-depth understanding of Spark Architecture, including Spark Streaming, Spark Core, Spark SQL, Data Frames, RDD caching, and Spark MLib.
• Hands-on experience with the Technology stack available in the industry for data management, data ingestion, capture, processing, and curation: Kafka, StreamSets, Attunity, GoldenGate, MapReduce, Hadoop, Hive, Hbase, Cassandra, Spark, Flume, Hive, Impala, etc.
• Hands-on knowledge of data frameworks, data lakes, and open-source projects such as Apache Spark, MLflow, and Delta Lake.
• Good working knowledge of code versioning tools [such as Git, Bitbucket, or SVN].
• Hands-on experience in using Spark SQL with various data sources like JSON, Parquet, and Key Value Pair.
• Experience preparing data for Data Science and Machine Learning with exposure to model selection, model lifecycle, hyperparameter tuning, model serving, deep learning, etc.
• Demonstrated experience preparing data, automating, and building data pipelines for AI Use Cases (text, voice, image, IoT data, etc. ).
• Good to have programming language experience. NET or Spark/Scala.
• Experience in creating tables, partitioning, bucketing, loading, and aggregating data using Spark Scala, Spark SQL/PySpark
• Knowledge of AWS/Azure/GCP DevOps processes like CI/CD as well as Agile tools and processes, including Git, Jenkins, Jira, and Confluence.
• Working experience with Visual Studio, PowerShell Scripting, and ARM templates.
• Able to build ingestion to ADLS and enable BI layer for Analytics.
• Strong understanding of Data Modeling and defining conceptual logical and physical data models.
• Big Data/analytics/information analysis/database management in the cloud.
• IoT/event-driven/microservices in the cloud- Experience with private and public cloud architectures, pros/cons, and migration considerations.
• Ability to remain up to date with industry standards and technological advancements that will enhance data quality and reliability to advance strategic initiatives.
• Working knowledge of RESTful APIs, OAuth2 authorization framework, and security best practices for API Gateways.
• Guide customers in transforming big data projects, including the development and deployment of big data and AI applications.
• Guide customers on Data engineering best practices, provide proof of concept, architect solutions, and collaborate when needed.
• 2+ years of hands-on experience designing and implementing multi-tenant solutions using AWS/Azure/GCP Databricks for data governance, data pipelines for near real-time data warehouse, and machine learning solutions.
• Over all 5+ years' experience in a software development, data engineering, or data analytics field using Python, PySpark, Scala, Spark, Java, or equivalent technologies.
• Hands-on expertise in Apache SparkTM (Scala or Python).
• 3+ years of experience working in query tuning, performance tuning, troubleshooting, and debugging Spark and other big data solutions.
• Bachelor's or Master's degree in Big Data, Computer Science, Engineering, Mathematics, or similar area of study or equivalent work experience.
• Ability to manage competing priorities in a fast-paced environment.
• Ability to resolve issues.
• Basic experience with or knowledge of agile methodologies.
• AWS Certified: Solutions Architect Professional.
• Databricks Certified Associate Developer for Apache Spark.
• Microsoft Certified: Azure Data Engineer Associate.
• GCP Certified: Professional Google Cloud Certified.
  
  

This job was posted by Bhoomika Varshney from Koantek.