Senior Data Engineer
At a glance
Senior Data Engineering role at Hdjm. Madison · full-time · no salary stated · we estimate $142,325–$192,038 across the United States.
Hdjm has not published a salary for this role. The figure below is our estimate, not theirs.
dataroles summary, based on the employer's posting.
What you'll do
- Build and support Databricks ETL pipelines, addressing failures and tuning performance across Databricks and SSIS
- Create Python, Spark and SQL datasets for analytics and AI/ML, including batch and streaming data lake designs
- Move older SSIS packages and SQL Server procedures into Databricks pipelines, and connect them to other systems
- Maintain SSRS reports and dashboards, resolving slowdowns and mismatches in their data
- Work with business users on reporting needs, add data checks and cleansing, and document development work
What you bring
- A bachelor's degree in computer science, information systems or a related discipline
- At least seven years working in data integration and reporting
- Extensive Databricks work with Python, Spark and Delta Lake, plus a solid grasp of warehouse design
- Strong SQL Server skills, including T-SQL and stored procedures, alongside SSIS and SSRS development experience
- Strong ability with AI-assisted development tools
Who this fits
This senior role suits someone who can work independently while collaborating with business users, analysts and IT teams. You should be comfortable explaining technical work clearly and documenting your processes.
From the employer
This role is responsible for the design, development, and maintenance of data integration and reporting solutions. Working primarily in Databricks, you will build and maintain the pipelines and datasets that feed our reporting, analytics, and AI/ML workloads. The ideal candidate will possess expertise in Databricks and strong skills in SQL Server, SSIS and SSRS, and experience with other modern data engineering tools such as Azure Data Factory. This position requires a proactive and results-oriented individual with a passion for data and a strong understanding of data warehousing principles. A strong grasp of AI-assisted development tools as a regular part of the engineering workflow is expected.
Responsibilities
Data Integration
Design, develop, and maintain robust and efficient ETL pipelines and processes on Databricks.
Troubleshoot and resolve Databricks pipeline errors and performance issues.
Troubleshoot and resolve SSIS package errors and performance issues.
Optimize data flow performance and minimize data latency.
Implement data quality checks and validations within ETL processes.
Maintain legacy SSIS packages for ETL processes.
Databricks Development
Develop and maintain Databricks pipelines and datasets using Python, Spark and SQL.
Drive the design and implementation of batch and streaming data lake architectures supporting both analytics and AI/ML workloads.
Optimize Databricks jobs for performance and cost-effectiveness.
Integrate Databricks with other data sources and systems.
Migrate legacy systems (e.g. SSIS packages, SQL Server procs) to Databricks pipelines.
Data Warehousing
Participate in the design and implementation of data warehousing.
Support data quality initiatives and implement data cleansing procedures.
Reporting and Analytics
Collaborate with business users to understand data requirements for department driven reporting needs.
Maintain existing library of complex SSRS reports, dashboards, and visualizations.
Troubleshoot and resolve SSRS report issues, including performance bottlenecks and data inconsistencies.
Collaboration and Communication
Comfortable in entrepreneurial, self-starting, and fast-paced environment, working both independently and with our highly skilled teams.
Collaborate effectively with business users, data analysts, and other IT teams.
Communicate technical information clearly and concisely, both verbally and in writing.
Document all development work and procedures thoroughly.
Continuous Growth
Keep abreast of the latest advancements in data integration, reporting, data engineering.
Actively contribute to company’s AI-assisted tooling, strategies, and best practices.
Continuously improve skills and knowledge through training and self-learning.
Requirements
Bachelor's Degree in Computer Science, Information Systems, or a related field.
7+ years of experience in data integration and reporting.
Extensive experience with Databricks, including Python, Spark, and Delta Lake
Experience with data warehousing concepts and best practices.
Strong proficiency with AI-assisted development tools.
Strong proficiency in SQL Server, including T-SQL, stored procedures, and functions.
Experience with SSIS (SQL Server Integration Services) development and maintenance.
Experience with SSRS (SQL Server Reporting Services) report design and development.
Experience with Microsoft Azure cloud platform and Microsoft Fabric desirable.
Strong analytical and problem-solving skills.
Excellent communication and interpersonal skills.
Ability to work independently and as part of a team.
Experience with Agile methodologies.
Data Engineering work from LinkedIn, X, Reddit and Upwork, in one list.
Each morning you get everything that fits, strongest first.
Data Engineering
Find me workBy the team behind dataroles.