Azure PySpark Solution Architect

We are seeking a highly skilled Azure PySpark Solution Architect to design and implement scalable data solutions on Microsoft Azure. The ideal candidate will have strong expertise in Azure services, PySpark, and solution architecture, ensuring efficient data processing and analytics workflows for enterprise applications.

Key Responsibilities:

1. Design and implement end-to-end data solutions using Azure Data Services and PySpark.

2. Develop high-performance ETL pipelines leveraging Azure Databricks, Azure Data Factory, and Synapse Analytics.

3. Architect scalable, secure, and cost-efficient cloud solutions that align with business objectives. 4. Collaborate with data engineers, data scientists, and business stakeholders to define technical requirements and solution roadmaps.

5. Optimize big data processing and ensure data governance, security, and compliance standards are met.

6. Provide technical leadership and best practices for Azure and PySpark-based data solutions.

7. Conduct performance tuning and troubleshooting for PySpark workloads.

8. Ensure seamless integration with third-party tools, APIs, and enterprise systems.

Must-Have Skills & Qualifications:

• Expertise in Azure Cloud Services (Azure Databricks, Data Factory, Synapse, Azure Storage, ADLS). • Strong hands-on experience with PySpark for data processing and transformation.

• Deep understanding of solution architecture, including microservices, eventdriven architectures, and cloud-native patterns.

• Experience with SQL, NoSQL databases, and data modeling in Azure.

• Knowledge of CI/CD pipelines, DevOps, and Infrastructure-as-Code (Terraform, ARM templates, or Bicep).

• Strong problem-solving skills and ability to optimize large-scale data workloads.

• Excellent communication and stakeholder management skills. Good-to-Have Skills:

• Familiarity with streaming technologies such as Kafka, Event Hubs, and Spark Streaming.

• Knowledge of containerization technologies (Docker, Kubernetes).

• Experience with machine learning frameworks and integration with big data pipelines.

• Familiarity with Agile/Scrum methodologies and related tools (JIRA, Confluence).

• Understanding of data security frameworks and compliance regulations (GDPR, HIPAA, etc.).

Skills

Azure, Pyspark, Solution Architecture