Lead Data Engineer
At a glance
Senior Data Engineering role at Fa Ewjt Saasfaprod1. Pune · full-time.
Pay not stated Lead Data Engineer salary
dataroles summary, based on the employer's posting.
What you'll do
- Build data services that connect client products with external vendors
- Automate manual work to speed data delivery and reduce cloud costs
- Set up monitoring for data quality, observability, governance and lineage
- Support production operations, investigate incidents and document root causes
- Write unit and integration tests, document designs and contribute to the engineering wiki
What you bring
- At least eight years of experience, with a computer science degree or equivalent work background
- At least five years building production data pipelines in Python
- Lead-level hands-on ability with PySpark, Python, SQL, Databricks and AWS
- Working knowledge of distributed data systems, cloud platforms, and tools such as Spark, Hive and Airflow
Who this fits
This senior role suits someone who can lead technical discussions, guide teammates and work with internal teams and client stakeholders. It is a hybrid position in Pune.
From the employer
In-depth working experience of distributed systems Hadoop, Spark, Hive, DBT and Airflow/Dagster
At least 5 years of solid production quality coding experience in data pipeline implementation in Python
Experience working with public cloud platforms, preferably AWS
Experience working with Databricks and/or Snowflake
Experience in Git, JIRA, Jenkins, shell scripting
Familiarity with Agile methodology, test-driven development, source control management and test automation
Experience supporting and working with cross-functional teams in a dynamic environment
You have excellent collaborative problem solving and communication skills and are empathetic to others
You have a track record of building fast, reliable, and high-quality data pipelines
Strong hands-on experience with PySpark, Python, SQL, Databricks, and AWS Cloud is required at a Lead level. The candidate should possess excellent problem-understanding and problem-solving capabilities, be able to lead technical discussions, guide team members, and effectively communicate and collaborate with both internal teams and client stakeholders.
You will build services that will integrate directly with Client's products and with external vendors.
You will work with modern data technologies such as Hadoop, Spark, DBT, Dagster/Airflow, Atlan, Trino, etc., modern data platforms such as Databricks and Snowflake and cloud technologies across AWS stack
Identify, design, and implement automation of manual processes, optimizing data delivery, reducing Cloud cost, etc.
Implement processes and systems to monitor Data Quality, Observability, Governance and Lineage.
Support operations to manage the production environment and help in resolving production issues with RCA
Write unit/integration tests, adopt Test-driven development, contribute to engineering wiki, and document design/implementation etc.
Graduate in Computer Science, Data Science, Information Technology, or a related field.
8+ years of experience and a bachelor’s degree in computer science or a related field, or equivalent work experience
Data Engineering work from LinkedIn, X, Reddit and Upwork, in one list.
Each morning you get everything that fits, strongest first.
Data Engineering
Find me workBy the team behind dataroles.