Data Engineer III

Position Title:

Data Engineer III

Pay Rate:

$83,000 - $121,000

Multiple factors, including Individual experience, skills and abilities will determine where an employee is ultimately placed in the pay range.

Category/Shift:

Salaried Full-Time

Physical Location:

75 5th Street NW
Atlanta, GA 30308
United States

The Job You Will Perform:

The Data Engineer is an important member of the Cost Innovation Team, which is a strategic, innovative, and cross-functional team that is focused on creating value for the company through the advanced analytics of manufacturing data. This position offers a flexible working arrangement provided that the home office is within a typical commuting distance to the A2C office.

Core Responsibilities

Analytics & Data Science Enablement

• Partner with analysts, data scientists, engineers, and business stakeholders to translate functional requirements into scalable technical data solutions.

• Deliver self-service, analytics-ready datasets that empower downstream teams to consume data without deep knowledge of underlying source systems.

• Enable industrial analytics use cases by providing trusted, contextualized time-series, asset, event, and operational data for analytics platforms and applications.

• Eliminate redundant data-preparation and contextualization efforts across disparate analytics and application teams.

• Help establish standard data definitions, reusable data products, and consistent business context across enterprise and industrial analytics environments.

• Support downstream analytics workflows, including advanced analytics, data science, visualization, and industrial process analytics.

Data Ingestion & Integration

• Design, build, and maintain robust pipelines to ingest data from relational databases, operational systems, industrial historians, APIs, cloud platforms, and flat files.

• Establish reliable and secure connections between source systems, cloud data platforms, and industrial data platforms.

• Write optimized SQL queries and extraction logic for high-performance data retrieval.

• Implement flexible ingestion patterns including batch, incremental, streaming, and replication-based approaches based on business latency requirements.

• Integrate high-volume time-series data and industrial metadata while maintaining relationships between assets, measurements, events, and other operational entities.

• Support ingestion and integration patterns used by platforms such as Cognite Data Fusion (CDF), Seeq, and Microsoft Fabric.

Data Storage & Lakehouse Engineering

• Design and manage scalable storage structures across raw (bronze), intermediate (silver), and curated (gold) layers.

• Leverage Spark-native formats and engines such as Delta Lake to support scalable analytics workloads.

• Design curated data structures that support both enterprise analytics and industrial data consumption.

• Enforce data retention, archiving, and purging policies to manage data lifecycles.

• Monitor and optimize storage, compute costs, and cloud resource utilization.

Data Transformation

• Develop Python and SQL-based transformations to clean, standardize, enrich, and unify data.

• Develop, tune, and proactively optimize scalable PySpark applications to efficiently process high-volume datasets.

• Build modular, reusable transformation components rather than siloed, application-specific logic.

• Resolve complex data-quality anomalies including schema drift, missing values, duplicates, timestamp alignment, and inconsistent identifiers.

• Engineer efficient incremental processing patterns to handle high-volume, frequently changing datasets.

• Transform raw operational and historian data into analytics-ready time series, calculations, conditions, events, and contextualized datasets for downstream consumption.

• Enable reusable data transformations that can support platforms such as Seeq, Cognite, Fabric, and custom analytics applications.

Semantic Modeling & Industrial Data Contextualization

• Translate complex source schemas into intuitive, business-aligned semantic entities, asset hierarchies, and industrial data models.

• Build and maintain relationships between assets, equipment, time series, events, files, and other operational data.

• Standardize naming conventions, identifiers, metadata, and relationship mappings across source systems.

• Perform data contextualization to connect operational data with relevant equipment, process, and business context.

• Support asset-centric and knowledge-graph-style data models commonly used in industrial data platforms such as Cognite.

Pipeline Orchestration & Operations

• Architect end-to-end workflows that seamlessly coordinate ingestion, transformation, contextualization, validation, and publication.

• Orchestrate data movement between source systems, lakehouse environments, industrial data platforms, and analytics applications.

• Implement resilient error handling including retries, automated recovery, checkpointing, and alerting.

• Monitor pipeline health, SLA compliance, data freshness, and data drift using centralized observability tools.

• Design pipelines capable of supporting both traditional enterprise data and high-frequency industrial time-series workloads.

Data Governance, Security & DevOps

• Enforce data security and privacy standards, including row- and column-level security, RBAC (Role-Based Access Control), and masking of sensitive data.

• Apply consistent access-control and governance patterns across enterprise data platforms and industrial analytics environments.

• Embed automated validation checks as needed to enforce data quality.

• Maintain Git-based development workflows using branching, code reviews, and pull requests.

• Implement automated CI/CD pipelines for code testing, data validation, configuration management, and repeatable environment deployments.

• Support deployment and lifecycle management of data pipelines, transformation logic, semantic models, and analytics configurations across development, test, and production environments.

• Maintain comprehensive documentation of data architectures, pipeline schemas, source-to-target mappings, contextual relationships, and lineage maps.

Preferred Experience

• Bachelor’s Degree in Computer Science, Software Engineering, Data Science, Engineering, or related field + 2 years of relevant experience; equivalent Master’s degree accepted; non-relevant bachelor’s degree requires 5+ years of equivalent hands-on experience.

• Experience working with:

  • SQL and Python for data engineering and transformation.

  • PySpark and Microsoft Fabric for lakehouse transformations.