Lead Software Engineer- Data Engineer
At a glance
Senior Data Engineering role at Cat (CaterpillarCareers). Chennai · full-time.
Pay not stated Data Engineering salary
dataroles summary, based on the employer's posting.
What you'll do
- Lead data engineers and platform developers, setting delivery practices and maintaining strong code and CI/CD standards
- Design secure AWS data platforms and ingestion approaches for batch and near-real-time processing
- Build scalable pipelines for structured and unstructured data, then tune Snowflake for speed and cost
- Use AWS services and CI/CD tools to deliver resilient pipelines, reusable frameworks, and operationally ready systems
- Turn stakeholder needs into data solutions and support analytics, streaming, and machine-learning workflows
What you bring
- At least ten years in data engineering or data platform work, including experience leading teams and delivering at scale
- Strong AWS data-platform skills and deep Snowflake knowledge across architecture, tuning, and data modeling
- Advanced Python and SQL skills, with substantial experience building ingestion pipelines and ETL or ELT frameworks
- Knowledge of streaming, distributed systems, software engineering practices, and Agile delivery
Who this fits
This senior lead role suits someone who can guide engineers while staying close to platform architecture and pipeline delivery. You’ll work onsite five days a week in Chennai; relocation is available.
From the employer
Career Area:
Technology, Digital and DataJob Description:
Your Work Shapes the World at Caterpillar Inc.
When you join Caterpillar, you're joining a global team who cares not just about the work we do – but also about each other. We are the makers, problem solvers, and future world builders who are creating stronger, more sustainable communities. We don't just talk about progress and innovation here – we make it happen, with our customers, where we work and live. Together, we are building a better world, so we can all enjoy living in it.
Software Engineering Lead – Data Platform
About the Role
We are seeking a highly skilled Software Engineering Lead – Data Platform to lead the development of Caterpillar’s next-generation Digital Manufacturing Data Platform. This platform enables large-scale data ingestion, transformation, and analytics across manufacturing, supply chain, and engineering ecosystems.
The ideal candidate will bring deep expertise in data engineering, large-scale data ingestion, AWS-based architectures, and Snowflake, along with strong leadership and software engineering discipline.
Key Responsibilities
- Leadership & Delivery
- Lead and mentor a team of data engineers and platform developers
- Drive Agile execution and ensure predictable, high-quality delivery
- Establish engineering best practices, code quality, and CI/CD standards
- Data Platform & Architecture
- Architect scalable and secure data platforms on AWS
- Design robust data ingestion frameworks for batch and near real-time pipelines
- Define best practices in data modeling, governance, and metadata management
- Data Engineering & Ingestion
- Lead design and development of scalable ingestion pipelines (structured and unstructured data)
- Build and optimize Snowflake-based data platforms for performance and cost
- Enable ingestion of diverse sources (databases, APIs, files, streaming data)
- Cloud & Platform Engineering
- Leverage AWS services (S3, Glue, Lambda, EMR, Redshift, etc.) for end-to-end pipelines
- Implement CI/CD pipelines using Azure DevOps / Jenkins
- Ensure system scalability, resiliency, and operational readiness
- Software Engineering Excellence
- Enforce software engineering principles (modular design, code quality, testing, version control)
- Drive automation and continuous improvement
- Promote reusable frameworks for ingestion and transformation
- Stakeholder Collaboration
- Partner with product managers, SMEs, and business stakeholders
- Translate business needs into scalable data solutions
Must-Have Skills
- 10+ years of experience in Data Engineering / Data Platform roles
- Strong experience in AWS data ecosystem (S3, Glue, Lambda, EMR, Redshift)
- Deep expertise in Snowflake (architecture, optimization, data modeling)
- Strong programming skills in Python and SQL
- Extensive experience with data ingestion pipelines and ETL/ELT frameworks
- Exposure to real-time streaming (Kafka, Spark Streaming)
- Experience with CI/CD tools (GitHub, Jenkins, AWS CloudFormation etc.)
- Solid understanding of distributed systems and scalable architectures
- Strong foundation in software engineering principles (Git, testing, design patterns)
- Experienced in working with Agile teams
- Collaborate with Data Science and AI teams to operationalize ML models and analytics workflows.
- Promote integration of AI capabilities into data engineering pipelines (e.g., GenAI, MCP, ATA).
- Support real-time analytics and edge AI use cases in manufacturing environments.
- Use AI extensively in building and testing Data Ingestion and Data pipeline
Nice-to-Have Skills
- Experience with Graph Databases (Neo4j, Neptune)
- Experience with Vector Databases (Milvus, OpenSearch)
- Knowledge of NVIDIA ecosystem and RAPIDS (cuDF, cuML, cuGraph)
- Experience integrating AI/ML pipelines or GenAI workflows
Qualifications
- Bachelor’s or Master’s degree in Computer Science / Engineering
- Proven track record in leading engineering teams and delivering at scale
- Strong leadership, communication, and stakeholder management skills
Relocation is available for this position.
Posting Dates:
November 20, 2026 - November 30, 2026Caterpillar is an Equal Opportunity Employer. Qualified applicants of any age are encouraged to apply
Not ready to apply? Join our Talent Community.