Data Engineer
At a glance
Mid-level Data Engineering role at Inetum. Lisbon · full-time.
Pay not stated Data Engineer salary
dataroles summary, based on the employer's posting.
What you'll do
- Build and maintain the infrastructure behind the existing data architecture
- Create Airflow pipelines to extract, process and load data, then monitor and maintain their stability
- Deploy Airflow workflows to production with DevOps tools, and improve the related infrastructure and processes
- Tune code and queries, review code, and make data useful and accessible to analysts and end-users
- Document analysis of regulatory topics and assess the impact and cost of changing business requirements
What you bring
- Advanced Python skills, including building APIs with Flask and documenting and testing Python code
- Strong SQL and relational database skills, including Oracle, plus Hadoop and Spark architecture experience
- Advanced PySpark skills and hands-on work with HDFS, Yarn, distributed Hadoop and Spark environments, and data lakes
- Experience developing Airflow pipelines and deploying Airflow in distributed setups, with Docker, Kubernetes, Kafka and Jenkins
- Fluent English and experience working with Scrum or another listed Agile methodology
Who this fits
This mid-level role suits a data engineer who can work across Python, SQL, data platforms and DevOps deployment. The position is based onsite in Lisbon.
From the employer
Inetum is a European leader in digital services. Inetum’s team of 28,000 consultants and specialists strive every day to make a digital impact for businesses, public sector entities and society. Inetum’s solutions aim at contributing to its clients’ performance and innovation as well as the common good. Present in 19 countries with a dense network of sites, Inetum partners with major software publishers to meet the challenges of digital transformation with proximity and flexibility. Driven by its ambition for growth and scale, Inetum generated sales of 2.5 billion euros in 2023.
We are looking for a Data Engineer to be responsible for building and maintaining the infrastructure that supports the organization’s data architecture.
The role involves creating and managing data pipelines using Airflow for data extraction, processing, and loading, ensuring their maintenance, monitoring, and stability.
The engineer will work closely with data analysts and end-users to provide accessible and reliable data
Main Tasks and Responsibilities:
- Responsible for maintaining the infrastructure that supports the current data architecture
- Responsible for creating data pipelines in Airflow for data extracting, processing, and loading
- Responsible for data pipelines maintenance, monitoring, and stability
- Responsible for providing data access to data analysts and end-users
- Responsible for DevOps infrastructure
- Responsible for deploying Airflow dags to production environment using DevOps tools
- Responsible for code and query optimization
- Responsible for code review
- Responsible for improving the current data architecture and DevOps processes
- Responsible for delivering data in useful and appealing ways to users
- Responsible for performing and documenting analysis, review, and study on specified regulatory topics
- Responsible for understanding business change and requirement needs, assess the impact and the cost
Technical skills:
- Advanced Python(Mandatory)
- Experience in creating APIs in Python - At least Flask (Mandatory)
- Experience in documenting and testing in Python (Mandatory)
- Advanced SQL skills and relational database management (Oracle is Mandatory, SQL server is desirable, PostgreSQL is desirable )
- Experience with Data Warehouses
- Hadoop ecosystem - HDFS + Yarn (Mandatory)
- Spark Environment Architecture (Mandatory)
- Advanced PySpark - (Mandatory)
- Experience in creating and maintaining distributed environments using Hadoop and Spark
- Data Lakes - Experience in organizing and maintaining data lakes (Mandatory) - S3 is preferred
- Experience with Parquet file format is mandatory, Avro is a plus
- Apache Airflow - Experience in both pipeline development and deploying Airflow in distributed environment (Mandatory)
- Containerization - Docker is Mandatory
- Kubernetes (Mandatory)
- Apache Kafka (Mandatory)
- Experience in automating applications deployment using DevOps tools - Jenkins is Mandatory, Ansible is a plus
- Agile methodologies (At least SCRUM)
Language Skills:
- Fluent in English (Mandatory)
Data Engineering work from LinkedIn, X, Reddit and Upwork, in one list.
Each morning you get everything that fits, strongest first.
Data Engineering
Find me workBy the team behind dataroles.