Big Data Operations Engineer

Job Description – Big Data Operations Engineer

Location : Onsite, Hyderabad Softility

Experience : 5+ years

Role Overview

We are looking for a skilled Big Data Operations Engineer with hands-on experience in supporting and maintaining large-scale data platforms. The role involves ensuring platform availability, managing incidents, monitoring performance through dashboards, and coordinating with development teams for enhancements. While the primary focus is on operations and support, the candidate should also have exposure to Big Data development concepts and tools.

Key Responsibilities

• Monitor and maintain Hadoop clusters, HDFS storage, and YARN resource management.
• Troubleshoot and resolve production issues, ensuring minimal downtime and quick recovery.
• Support Informatica workflows in production, including error handling and performance tuning.
• Manage Kafka topics, producers/consumers, and cluster health monitoring.
• Work with Anthos for hybrid cloud platform management and operational tasks.
• Support observability and log pipelines using Cribl.
• Build and maintain dashboards for monitoring and reporting using tools like New Relic.
• Perform incident management, root cause analysis, and preventive action planning.
• Collaborate with development teams to roll out new jobs, updates, and fixes.
• Maintain operational documentation, runbooks, and knowledge repositories.
  
  

Required Skills & Experience

• 5-6 years of experience in Big Data operations/support.
• Strong knowledge of the Hadoop ecosystem, including HDFS, YARN, Hive, and Oozie.
• Experience supporting ETL workflows (preferably Informatica).
• Familiarity with Kafka operations and troubleshooting.
• Exposure to Anthos and Cribl for platform and pipeline monitoring.
• Experience with dashboard/monitoring tools (Grafana, Kibana, or equivalent).
• Hands-on with Linux/Unix administration, shell scripting, and automation.
• Solid experience in incident, change, and problem management.
• Good communication and coordination skills for working with cross-functional teams.
  
  

Nice-to-Have

• Exposure to Big Data development (Hive, Spark, Python, or SQL).
• Knowledge of cloud platforms (GCP/AWS/Azure).
• Familiarity with Cloudera Manager or other cluster monitoring tools.