Senior Databricks Engineer
At a glance
Senior Data Engineering role at Epic Software. Hyderabad · full-time.
Pay not stated Data Engineering salary
dataroles summary, based on the employer's posting.
What you'll do
- Build scalable ELT pipelines with Databricks, PySpark, SQL, dbt and Delta Lake
- Shape Bronze, Silver and Gold data layers around enterprise modeling standards and business rules
- Connect data from source systems and deliver pipeline outputs to applications, APIs and analytical layers
- Protect data reliability with validation, error handling, schema-change support and incremental refreshes
- Tune Spark and Delta workloads, create reusable pipeline frameworks, and document lineage and metadata
What you bring
- You have five to eight years building data ingestion and transformation pipelines
- You have hands-on dbt experience for building, testing and documenting modular transformations
- You bring strong SQL and PySpark skills, plus practical experience with Databricks, Spark jobs and Delta Lake
- You have worked with ELT and Medallion layers, and know an orchestration tool such as Airbyte or ADF
Who this fits
This senior role suits an engineer who can turn complex business and technical needs into reliable, performant data pipelines. You’ll work across data architecture, modeling, cloud and application teams; experience with operational data such as logistics, telematics or IoT is a plus.
From the employer
About Aubrant
Aubrant Digital is a leader in multi-shore custom application development. We are passionate about solving our clients’ business problems through consultative teamwork, innovative software, and proven processes. We’ve served more than 50 clients and delivered hundreds of high quality, custom enterprise applications. Our clients value us as integral team members who get the job done on time and on spec, and we are proud of our high client retention rate and under 2% staff turnover. With offices in New Jersey, Boston, Costa Rica, and Eastern Europe, we execute the full software lifecycle, from architecture and design through development, QA and application maintenance & support. Our company culture emphasizes client service, trust-based relationships, and innovation.
About the role
The Senior Databricks Engineer will design, build, and optimize scalable data ingestion and transformation pipelines to support enterprise‑grade data modernization programs. This role focuses on implementing ELT workflows within modern Data Lakehouse architectures, ensuring high‑quality, reliable, secure, and efficient movement of data from diverse source systems into curated analytical and application‑ready layers.
You will work closely with Data Architects, Data Modelers, Cloud Engineers, and Application teams to develop repeatable processing frameworks, enforce data quality rules, and operationalize Medallion‑layered datasets (Bronze/Silver/Gold or equivalents). This role is domain‑agnostic but requires the ability to translate complex business and technical requirements into performant data pipelines. Experience with high‑volume operational data domains (e.g., logistics, telematics, IoT, ERP, mobility platforms) is a plus.
Key Qualifications:
• Bachelor’s/Master’s in Computer Science, Information Systems, Engineering, or related field (or equivalent experience).
• 5–8 years of experience building data ingestion and transformation pipelines.
• Strong proficiency in SQL and PySpark for large‑scale data processing.
• Hands‑on experience with dbt (data build tool) for building, testing, and documenting modular data transformation pipelines — a core requirement for this role.
• Hands‑on experience with Databricks, Spark jobs, and Delta Lake‑based transformations.
• Experience with ELT patterns and Medallion Architectures (Bronze/Silver/Gold).
• Practical knowledge of Airbyte, ADF, or equivalent orchestration tools.
• Experience with data quality frameworks such as Great Expectations or Databricks expectations.
• Strong understanding of schema evolution, partitioning, optimization, and Delta Lake performance tuning.
• Familiarity with Azure components including ADLS Gen2, Event Hub, Synapse Serverless/Dedicated SQL.
• Experience integrating pipelines with applications and APIs (read/write patterns).
• Strong debugging skills for data integrity, ingestion failures, schema drift, and performance issues.
• Excellent communication, documentation, and cross‑team collaboration skills.
Responsibilities:
• Build scalable ELT pipelines using Databricks (PySpark/SQL), dbt, and Delta Lake.
• Develop Bronze, Silver, and Gold transformations aligned with enterprise modeling standards.
• Implement ingestion using tools like Airbyte, ADF, or custom connectors.
• Collaborate with Data Modelers to operationalize business rules, mappings, and canonical definitions.
• Integrate data pipelines with downstream applications (mobile apps, web apps, APIs).
• Implement data quality and validation rules, error handling, and automated remediation steps.
• Manage schema evolution, CDC ingestion, late‑arriving data, and incremental refresh patterns.
• Optimize Spark/Delta workloads through partitioning, indexing, caching, Z‑ordering, and cluster tuning.
• Develop reusable pipeline templates and frameworks for cross‑domain scalability.
• Collaborate with Cloud Engineers to ensure pipelines meet security, monitoring, and DevOps standards.
• Produce technical documentation, lineage, and metadata for governance teams.
• Support ML workloads by preparing feature datasets and inference‑ready tables.
• Participate in cross‑functional design sessions, reviews, and architectural decisions.
About Aubrant & Our Culture
Data Engineering work from LinkedIn, X, Reddit and Upwork, in one list.
Each morning you get everything that fits, strongest first.
Data Engineering
Find me workBy the team behind dataroles.