Senior Data Engineer (Databricks and Pyspark)-Mandarin Speaker
Unison Group Open At a glance
Senior Data Engineering role at Unison Group. Kuala Lumpur · full-time.
Pay not stated Data Engineer salary
dataroles summary, based on the employer's posting.
What you'll do
- Build and tune data pipelines in Databricks, PySpark and Python, covering both ingestion and transformation
- Set up Delta Lake and Medallion patterns for scalable enterprise data platforms
- Connect data from APIs, SFTP, SharePoint and cloud sources, then automate processing with workflows and CI/CD
- Improve Spark job speed, reliability and cost, while adding data checks, monitoring, alerts and retries
- Design governed datasets and guide junior engineers as you deliver curated data for reporting and analytics
What you bring
- You have at least five years in data engineering and strong practical skills with Databricks, PySpark, Delta Lake, SQL and Python
- You have used Azure data services or comparable AWS tools, and integrated data from APIs, SFTP and cloud storage
- You bring hands-on Git, Azure DevOps CI/CD and YAML pipeline experience, plus Spark performance tuning skills
- You can work in Mandarin, which is required for the role
Who this fits
This suits a senior engineer who combines hands-on architecture and technical leadership with delivery work. The position is full-time and on-site in Kuala Lumpur; the listing also names Malaysia and Singapore as locations.
From the employer
Position : Senior Data Engineer.
Location : Malaysia and Singapore.
Language : Mandarin Language must.
Role : Full Time Perm role + one month Bonus.
Requirements
We are looking for a Senior Data Engineer with strong expertise in Databricks, PySpark, Delta Lake, and cloud-based data pipelines. implement Lakehouse/Medallion architectures, and integrate data from multiple internal and external systems. This role requires strong technical leadership and hands-on architecture experience.
Key Responsibilities
- Design, build, and optimize data ingestion and transformation pipelines using Databricks, PySpark, and Python.
- Implement Delta Lake and Medallion architecture for scalable enterprise data platforms.
- Develop ingestion frameworks for data from SFTP, REST APIs, SharePoint/Graph API, AWS, and Azure sources.
- Automate workflows using Databricks Workflows, ADF, Azure Functions, and CI/CD pipelines.
- Optimize Spark jobs for performance, reliability, and cost efficiency.
- Implement data validation, quality checks, and monitoring with automated alerts and retries.
- Design secure and governed datasets using Unity Catalog and cloud security best practices.
- Collaborate with analysts, business users, and cross-functional teams to deliver curated datasets for reporting and analytics.
- Provide technical leadership and guidance to junior team members.
Required Skills
- 5–8+ years of experience in Data Engineering.
- Strong hands-on experience with Databricks, PySpark, Delta Lake, SQL, Python.
- Experience with Azure Data Lake, ADF, Azure Functions, or AWS equivalents (S3, Lambda).
- Experience integrating data from APIs, SFTP servers, vendor data providers, and cloud storage.
- Knowledge of ETL/ELT concepts, Lakehouse/Meddalion architecture, and distributed processing.
- Strong experience with Git, Azure DevOps CI/CD, and YAML pipelines.
- Ability to optimize Spark workloads (partitioning, caching, Z-ordering, performance tuning).
Good to Have
- Exposure to Oil & Gas or trading analytics (SPARTA, KPLER, IIR, OPEC).
- Knowledge of Power BI or data visualization concepts.
- Familiarity with Terraform, Scala, or PostgreSQL.
- Experience with SharePoint development or .NET (optional)
Data Engineering work from LinkedIn, X, Reddit and Upwork, in one list.
Each morning you get everything that fits, strongest first.
Data Engineering
Find me workBy the team behind dataroles.