la26.042 - Data Engineer
At a glance
Mid-level Data Engineering role at Tiger Analytics Inc. Remote, Mexico.
Pay not stated Data Engineering salary
dataroles summary, based on the employer's posting.
What you'll do
- Build and maintain production data pipelines with Azure Databricks, PySpark and Azure Data Factory
- Develop Lakehouse pipelines across the Bronze, Silver and Gold layers
- Tune Delta Lake storage and query performance using Z-Ordering, liquid clustering, vacuuming and partitioning
- Deploy pipeline infrastructure through Terraform, Git and Azure DevOps, and manage access and lineage with Unity Catalog
- Investigate Spark cluster slowdowns by adjusting sizing, memory and execution plans
What you bring
- More than four years working with modern data stack technologies, especially Azure Databricks and Medallion Architecture
- Hands-on experience designing and improving scalable cloud data pipelines
- Written and spoken English proficiency
Who this fits
This role suits a mid-to-senior data engineer with experience across Azure Databricks, Lakehouse pipelines and distributed Spark performance. The working day runs from 8:00 AM to 5:00 PM PST, and the role is based in Mexico with flexible working arrangements.
From the employer
Tiger Analytics is a leading AI and advanced analytics consulting firm dedicated to delivering impactful data-driven solutions to Fortune 100 companies worldwide. We specialize in helping organizations solve complex business challenges through innovative Data Engineering, Machine Learning, and AI techniques. We are currently seeking a highly skilled Data Engineering Lead to manage and lead complex data engineering projects, driving excellence and innovation in our delivery.
Requirements
• We are seeking a highly experienced Data Engineer with over four years of expertise in modern data stack technologies, specifically focusing on Azure Databricks and the Medallion Architecture encompassing the Bronze, Silver, and Gold processing layers.
• Proven experience in designing, developing, and optimizing scalable, cloud-based data pipelines is essential.
• Pipeline Development & Orchestration: Responsible for creating, deploying, and maintaining production-grade data pipelines utilizing Azure Databricks, PySpark, and Azure Data Factory (ADF).
• Medallion Framework Management: Accountable for developing and managing the comprehensive Lakehouse pipeline architecture across the Bronze, Silver, and Gold layers.
• Delta Lake Optimization: Expertise in applying advanced Delta Lake features such as Z-Ordering, liquid clustering, vacuuming, and partitioning techniques to minimize query costs and improve read/write efficiency.
• Infrastructure as Code & CI/CD: Collaborate in managing and deploying cloud infrastructure using Terraform, Git, and Azure DevOps, automating the deployment of data pipelines.
• Data Governance & Security: Implement enterprise-level data governance frameworks, enforce access controls, and manage data lineage tracking utilizing Unity Catalog.
• Performance Tuning: Monitor, analyze, and address performance bottlenecks in distributed Spark clusters by optimizing cluster sizing, memory allocation, and execution plans.
• Proficiency in English, both spoken and written, is required for effective communication.
• Working hours are from 8:00 AM to 5:00 PM PST.
Benefits
- A collaborative and innovative work environment.
- Opportunities to work on impactful projects with global clients.
- Competitive salary and benefits tailored to mid to senior professionals.
- Continuous learning and career development opportunities.
- Flexible working arrangements based in Mexico.