Data Engineer

SparkBrains


Date: 3 weeks ago
City: Panchkula, Haryana
Salary: ₹700,000 - ₹800,000 per year
Contract type: Full time
Skills:
Python, SQL, Apache Spark, Apache Kafka, AWS Redshift, ETL Tools, Data Warehousing, Data Lake,

  • Data Acquisition
  • Candidate should manage the existing Data pipelines built for data ingestion.
  • Create and manage new data pipelines following the best practices for the new ingestion of data.
  • Continuously monitor the data ingestion through Change Data Capture for the incremental load.
  • Any failed batch job schedule to be analyzed and fixed to capture the data.
  • Maintaining and continuously updating the technical documentation of the ingested data and maintaining the centralized data dictionary, with necessary data classifications.
  • Data Extraction and Cleaning
  • Extraction of data from the data sources to be cleaned and ingested into a big data platform.
  • Automation of data cleaning has to be defined before ingestions.
  • Data cleaning to handle the missing data and remove any outliers and resolve any inconsistencies.
  • Data quality check has to be performed in terms of accuracy, completeness, consistency, timeliness, believability, and interpretability.
  • Data Integration, Aggregation and Representation
  • Exposing Data views or Data models to Reporting and source systems using Hive or Impala, or similar tools.
  • Exposing cleansed data to the Artificial Intelligence team for building data science models.
  • Informatica Data Catalog
  • Implement and configure the Informatica Enterprise Data Catalog (EDC) solution to discover and catalog data assets across the organization.
  • Develop and maintain custom metadata scanners, resource configurations, and lineage extraction processes.
  • Integrate EDC with other Informatica tools, such as Data Quality (IDQ), Master Data Management (MDM), and Axon Data Governance.
  • Define and implement data classification, data profiling, and data quality rules to improve data visibility, accuracy, and trustworthiness.
  • Collaborate with data stewards, data owners, and data governance teams to identify, document, and maintain business glossaries, data dictionaries, and data lineage information.
  • Establish and maintain data governance policies, standards, and procedures within the EDC environment.
  • Monitor and troubleshoot EDC performance issues, ensuring optimal performance and data availability.
  • Train and support end-users in effectively utilizing the data catalog for data discovery and analysis.
  • Keep up to date with industry best practices and trends, continuously improving the organization's data catalog implementation.
  • Collaborate with cross-functional teams to drive data catalog adoption and ensure data governance compliance across the organization.

Skill Set

  • Certified Big Data Engineer from Cloudera/AWS/Azure
  • Expertise with Big data products Cloudera stack.
  • Expertise in Big Data querying tools, such as Hive, Hbase, and Impala.
  • Expertise in SQL, writing complex queries/views, partitions, and bucketing.
  • Strong Experience in Spark using Python/Scala.
  • Expertise in messaging systems, such as Kafka or RabbitMQ.
  • Hands-on experience in the Management of the Hadoop cluster with all included services.
  • Implementing ETL process using Sqoop/Spark.
  • Implementation including loading from disparate data sets, Pre-processing using Hive.
  • Ability to design solutions independently based on high-level architecture.
  • Collaborate with other development teams.
  • Expertise in building stream-processing systems, using solutions such as Spark-Streaming, Apache NIFI, and KAFKA.
  • Expertise with NoSQL databases such as HBase.
  • Experience with Informatica Enterprise Data Catalog (EDC) implementation and administration.
  • Strong knowledge of data management, data governance, and metadata management concepts.
  • Proficiency in SQL and experience with various databases (e.g., Oracle, SQL Server, PostgreSQL) and data formats (e.g., XML, JSON, CSV).
  • Experience with data integration, ETL/ELT processes, and Informatica Data Integration.

How to apply

To apply for this job you need to authorize on our website. If you don't have an account yet, please register.

Post a resume

Similar jobs

Sales Executive

NetAmbit, Panchkula, Haryana
2 weeks ago
Job DescriptionThis job is provided by apna.co JOB DESCRIPTIONProfile - BDE (Business Development Executive)Qualification:-12th passedBike Mandatory & Android phoneExperience- Minimum 6 Months Experience In Any Field SalesJob Description- Apko Field Par rhe ke Onboarding +Sound box sale krna hai(Per day 2-3 sound box sale + 20-30 shops visit)PHONEPE के क्या फायदे हैं?PhonePe ऐप 11 भारतीय भाषाओं में उपलब्ध है।PhonePe का...

QA/QC (medical devices)

Bhargava Consulting Group, Panchkula, Haryana
₹18,000 - ₹24,000 per month
2 weeks ago
Skills:iso13485, Medical Devices, SOPs, Quality Assurance, Validation, Quality Control,Purpose of the Role: To ensure quality management systems are implemented and maintained always Responsibilities:To handle all activities related to quality management system, ISO 9001 & 13485, 93/42/EEC Directive including-Technical files making & updating including risk management, clinical evaluation, post-market surveillance, quality plan, ESRs, etc.Fulfilment of regulatory requirements related to EC directive,...

Digital Marketing Executive

Astroport Global, Panchkula, Haryana
3 weeks ago
We are looking for a talented Digital Marketing Executive with experience to join our team. The ideal candidate will be responsible for planning and executing online marketing strategies, optimizing digital platforms for maximum visibility, and ensuring accurate targeting of the client base. ates should have a strong understanding of digital marketing tools and techniques, good communication skills, and basic content...