Job Title: ETL Developer (Python / Big Data Engineer)
Location: Hybrid (2-3 days in customer office)
Mode of Work: Hybrid work environment, with 2-3 days onsite at the customer office.
Type: Contract (Initial duration 6 months)
Job Description:
We are looking for an experienced ETL Developer with a strong background in Python development and Big Data technologies to join our team. As an ETL Developer, you will be responsible for the design, development, and implementation of data processing pipelines using Python, Spark, and other related technologies to handle large-scale data efficiently. You will also be involved in ensuring the integration of data into cloud environments such as Azure, alongside basic DevOps tasks and RDBMS fundamentals.
Responsibilities:
- Develop and maintain ETL pipelines using Python for data extraction, transformation, and loading.
- Utilize Apache Spark for big data processing to handle large datasets and optimize performance.
- Work with cloud technologies, particularly Azure, to deploy and integrate data solutions.
- Implement key Python concepts and leverage libraries/packages like Pandas, NumPy, and others for data manipulation.
- Perform data integration tasks involving various data sources and structures.
- Collaborate with cross-functional teams to design and implement robust, scalable data solutions.
- Apply basic DevOps practices to manage and automate workflows within the ETL process.
- Ensure best practices in database management and integration with RDBMS systems.
- Participate in troubleshooting, optimization, and performance tuning of data processing systems.
Required Skills and Experience:
- Proficient in Python with hands-on experience in key libraries (Pandas, NumPy, etc.) and a deep understanding of Python programming concepts.
- Solid experience in Big Data Processing using Apache Spark for large-scale data handling.
- Basic DevOps knowledge and familiarity with CI/CD pipelines for automating workflows.
- Understanding of Azure Fundamentals and cloud data solutions.
- Strong understanding of RDBMS database fundamentals (SQL, relational data modelling, etc.).
- Previous experience in ETL development and data integration.
- Senior/Lead level experience with hands-on development in relevant technologies.
- Excellent problem-solving skills and ability to optimize data workflows.
Additional Desirable Skills:
- Familiarity with cloud-based data storage and processing technologies in Azure.
- Experience working in Agile or other collaborative development environments.