Azure PySpark Solution Architect

UST


Date: 1 day ago
City: Thiruvananthapuram, Kerala
Contract type: Full time

We are seeking a highly skilled Azure PySpark Solution Architect to design and implement scalable data solutions on Microsoft Azure. The ideal candidate will have strong expertise in Azure services, PySpark, and solution architecture, ensuring efficient data processing and analytics workflows for enterprise applications.


Key Responsibilities:

1. Design and implement end-to-end data solutions using Azure Data Services and PySpark.

2. Develop high-performance ETL pipelines leveraging Azure Databricks, Azure Data Factory, and Synapse Analytics.

3. Architect scalable, secure, and cost-efficient cloud solutions that align with business objectives. 4. Collaborate with data engineers, data scientists, and business stakeholders to define technical requirements and solution roadmaps.

5. Optimize big data processing and ensure data governance, security, and compliance standards are met.

6. Provide technical leadership and best practices for Azure and PySpark-based data solutions.

7. Conduct performance tuning and troubleshooting for PySpark workloads.

8. Ensure seamless integration with third-party tools, APIs, and enterprise systems.


Must-Have Skills & Qualifications:

• Expertise in Azure Cloud Services (Azure Databricks, Data Factory, Synapse, Azure Storage, ADLS). • Strong hands-on experience with PySpark for data processing and transformation.

• Deep understanding of solution architecture, including microservices, eventdriven architectures, and cloud-native patterns.

• Experience with SQL, NoSQL databases, and data modeling in Azure.

• Knowledge of CI/CD pipelines, DevOps, and Infrastructure-as-Code (Terraform, ARM templates, or Bicep).

• Strong problem-solving skills and ability to optimize large-scale data workloads.

• Excellent communication and stakeholder management skills. Good-to-Have Skills:

• Familiarity with streaming technologies such as Kafka, Event Hubs, and Spark Streaming.

• Knowledge of containerization technologies (Docker, Kubernetes).

• Experience with machine learning frameworks and integration with big data pipelines.

• Familiarity with Agile/Scrum methodologies and related tools (JIRA, Confluence).

• Understanding of data security frameworks and compliance regulations (GDPR, HIPAA, etc.).


Skills

Azure, Pyspark, Solution Architecture

How to apply

To apply for this job you need to authorize on our website. If you don't have an account yet, please register.

Post a resume

Similar jobs

Freelance Interviewer-Full Stack

futuremug, Thiruvananthapuram, Kerala
1 week ago
Senior Full Stack (Optimizely)-Freelance InterviewerAbout the companyWe are an HR Tech company based in Trivandrum, offering hiring support to MNCs across India through interview ,assessments and recruitment services. We have a network of 4000+ experienced professionals who take interviews in their available time slots.We’re looking for experienced professionals across various domains who can take up freelance interviews for our clients....

Senior Software Engineer – C++,QT/QML

Quest Global, Thiruvananthapuram, Kerala
3 weeks ago
Job RequirementsAs a member of the team, the primary job will be to work in customer project to perform requirement analysis, effort estimate, design, feasibility, development and maintenance of embedded UI applications in Agile work flow, work closely with customer development team(including daily interactions), Managing offshore team members in Task/Sprint planning and execution, Handling weekly calls and update status to...

Applications Developer 4

Oracle, Thiruvananthapuram, Kerala
3 weeks ago
Job DescriptionAnalyze, design develop, troubleshoot and debug software programs for commercial or end user applications. Writes code, completes programming and performs testing and debugging of applications.As a member of the software engineering division, you will analyze and integrate external customer specifications. Specify, design and implement modest changes to existing software architecture. Build new products and development tools. Build and execute...