AWS Data Engineer
At a glance
Mid-level Data Engineering role at Fa Ewjt Saasfaprod1. Pune · full-time.
Pay not stated Data Engineering salary
dataroles summary, based on the employer's posting.
What you'll do
- Design AWS data architectures with attention to security, scale and cost, then build ETL flows using Lambda and Glue
- Configure Glue Crawlers and the Data Catalog, including how schemas change over time
- Build, test and tune PySpark applications, and improve Iceberg lake tables through compaction and performance work
- Coordinate event-driven workflows with Step Functions and EventBridge, and connect Starburst to Lambda and Glue jobs
- Set up automated test and deployment pipelines, using PyTest to check Python work
What you bring
- Hands-on work with AWS services such as Lambda, Glue, Athena and S3, plus other listed AWS tools
- Strong PySpark or Scala development skills, including unit testing and performance tuning
- Strong Python skills with libraries such as pandas, requests or another listed library
- Experience with Kafka, Iceberg data lakes, and formats such as Avro, Parquet or another listed format
Who this fits
This mid-level role suits an AWS data engineer who has built Spark-based data applications and worked with real-time or event-driven data. The position is hybrid in Pune, India.
From the employer
We are seeking a highly skilled AWS Data Engineer with deep expertise in AWS cloud architecture, big data processing, real-time streaming, and modern data lake technologies. The ideal candidate will have strong hands-on experience in Spark (PySpark), Iceberg, EMR, Starburst/Trino, and event-driven architectures, along with experience building real-time and API-driven data applications who can design and build generic solutions for one of our Fortune 500 Client programs in the realm of Financial Master & Reference Data Management. This is high visibility, fast-paced key initiative will integrate data across internal and external sources, provide analytical insights, and integrate with the customer’s critical systems.
Key Responsibilities
- Design and implement scalable, secure, and cost-optimized AWS data architectures.
- Develop and maintain ETL pipelines using AWS Lambda and AWS Glue ETL.
- Configure and manage AWS Glue Crawlers, Glue Data Catalog, and schema evolution.
- Build, optimize, and unit test applications on the Apache Spark framework using PySpark.
- Design and optimize data lakes using Apache Iceberg on AWS, including table compaction and Iceberg performance tuning.
- Work extensively with data formats such as Avro, Parquet, JSON, XML, and CSV.
- Orchestrate event-driven workflows using AWS Step Functions and Amazon EventBridge.
- Connect and integrate Starburst from Lambda and Glue ETL jobs for federated querying.
- Implement CI/CD pipelines for automated testing and deployment.
- Perform unit testing using PyTest, and performance tuning of Spark and Python applications
- Strong understanding of AWS architecture best practices, scalability, security, and cost optimization strategies.
- Strong hands-on experience with AWS services including Lambda, Glue ETL, Athena, S3, DynamoDB, Step Functions, EventBridge, SNS, and SQS.
- Deep experience in Apache Spark (PySpark/Scala) development, unit testing, and performance optimization.
- Strong Python programming skills using libraries such as pandas, requests, json, and awswrangler.
- Experience on Apache Kafka and Confluent Kafka.
- Experience designing and optimizing data lakes using Apache Iceberg, including compaction and Iceberg optimization techniques.
Data Engineering work from LinkedIn, X, Reddit and Upwork, in one list.
Each morning you get everything that fits, strongest first.
Data Engineering
Find me workBy the team behind dataroles.