PySpark Developer
₹2,000,000 – ₹3,000,000 / yr base salary
Posted 13 Oct 2026
BranchFactor Summary
The original posting with the fluff stripped out
- Expected experience
- 8–13 yrs
- Management level
- People management
- Contract
- Full-time
| Base salary | ₹2,000,000 – ₹3,000,000 / yr |
|---|
This role is for one of Weekday’s clients
Salary range: Rs 2000000 - Rs 3000000 (ie INR 20 - 30 LPA)
Min Experience: 8+ years
Location: Bengaluru, Chennai, Tamil Nadu, India
JobType: full-time
We are looking for an experienced PySpark Developer with 8–13 years of experience in data engineering, Python programming, and large-scale data processing. The ideal candidate will have strong expertise in developing, optimizing, and maintaining scalable data pipelines using PySpark and Apache Spark. This role requires hands-on experience in processing structured and semi-structured data, implementing ETL/ELT workflows, and building reliable data solutions that support business intelligence, analytics, and data-driven decision-making.
The candidate will collaborate with data engineers, architects, analysts, and other technical stakeholders to design efficient data processing solutions, improve pipeline performance, and ensure data quality across the organization.
Requirements
Key Responsibilities
1. Data Pipeline Development
- Design, develop, and maintain scalable, high-performance data pipelines using PySpark and Python.
- Build robust ETL/ELT workflows to ingest, transform, validate, and load large volumes of data from multiple sources.
- Develop batch and, where required, streaming data processing solutions to support business and analytical requirements.
- Implement reusable data processing frameworks and ensure pipelines are reliable, maintainable, and scalable.
- Integrate data from databases, APIs, cloud storage, and enterprise data platforms.
2. PySpark Development and Optimization
- Develop complex data transformations using PySpark DataFrames, Spark SQL, and distributed processing techniques.
- Optimize Spark jobs by analyzing execution plans, partitioning strategies, caching, and resource utilization.
- Troubleshoot performance bottlenecks, resolve processing failures, and improve job execution efficiency.
- Work with large datasets while ensuring efficient memory management and distributed computing performance.
- Apply best practices for handling joins, aggregations, data skew, and incremental data processing.
3. Python Programming
- Write clean, modular, reusable, and well-documented Python code for data engineering applications.
- Develop utilities, validation scripts, and automation solutions to streamline data workflows.
- Implement exception handling, logging, testing, and debugging practices to improve code quality.