Senior Data Engineer

The Senior Data Engineer is responsible for designing, building, and maintaining robust data pipelines and integration solutions across Tradin Organic application landscape. Working closely with IT Business Analysts, process owners, and third-party vendors, this role ensures that data flows reliably and securely between SAP S/4HANA, BTP Integration Suite, SaaS platforms (HubSpot, Novulo, TraceGains, Osapiens), and internal systems. The Senior Data Engineer champions API-first, zero-trust, and process-first integration principles and plays a key role in automating ETL processes — increasingly leveraging AI-assisted tooling. This role is the primary technical owner of integration pipelines and acts as a trusted partner to business stakeholders, translating business process requirements into scalable, maintainable data solutions.

Accountability

Description of key activities and responsibilities

Pipeline Design & Development

Design, develop, and maintain data pipelines using Azure DevOps CI/CD practices and orchestration tools such as Kestra; build and maintain SAP BTP Integration Suite iFlows for inbound and outbound integrations.

Develop ETL/ELT processes in Python (primary language) or equivalent; apply software engineering best practices (version control, code review, unit testing, documentation).

Integrate AI-assisted capabilities into ETL workflows — including LLM-based data transformation, anomaly detection, intelligent field mapping, and process automation.

Ensure pipelines are idempotent, observable, and recoverable; implement logging, alerting, and monitoring for pipeline health.

API & System Integration

Design and implement API integrations (REST, SOAP, event-based) between internal and external systems following the company’s API-first architecture principles.

Manage end-to-end integration with third-party SaaS systems including HubSpot, Novulo (PIM), TraceGains, Osapiens, Taulia/Ariba, and SAP Analytics Cloud.

Build and maintain business process integrations such as customer/vendor master synchronisation, invoice interfacing, product catalogue sync, and ERP data flows.

Work with SAP BTP Integration Suite (iFlows, Event Mesh, API Management) and SAP Cloud Connector for secure on-premise connectivity.

Willing to learn and work with SAP Business Technology Platform (BTP) as the central integration middleware — including iFlow development and BTP Credential Store.

Security & Secret Management

Own and operate centralised secret management using HashiCorp Vault — including KV secrets, PKI certificates, dynamic credentials, and Vault Agent sidecar integration.

Manage public/private key infrastructure, TLS certificate lifecycle, and token-based authentication for service accounts.

Ensure all credentials, API keys, and certificates are stored in Vault and never hard-coded; enforce secrets hygiene across all integration projects.

Adhere to and enforce Zero Trust principles: short-lived tokens, mTLS where supported, no implicit network trust.

3rd Party & Vendor Management

Act as the primary technical contact for third-party integration providers; coordinate onboarding, testing, go-live, and ongoing support of external integrations.

Follow up on incidents, change requests, and SLA compliance with external partners; escalate where necessary.

Maintain integration documentation and runbooks for all external connections.

Data Quality & Reporting

Ensure data quality, consistency, and integrity across integrated systems; implement validation and reconciliation checks in pipelines.

Support reporting and analytics by delivering clean, structured data to BI tools (QlikSense, SAP Analytics Cloud, Hyperion).

Develop and maintain SQL queries, stored procedures, and data models that support operational and financial reporting.

Upload Templates & Data Loading

Design and build structured upload templates (Excel, CSV, or web-based) that allow business users to load master data and transactional data into ERP and other systems in a controlled, validated manner.

Develop the underlying import logic and validation rules that process uploaded data — including error handling, duplicate detection, field mapping, and transformation prior to loading.

Maintain and version-control upload templates in line with system changes; communicate template updates to end users and provide guidance where needed.

Work with business stakeholders to understand data load requirements for migrations, bulk updates, and periodic data refreshes (e.g. product master updates, vendor lists, price lists).

BI Data Warehouse

Design and develop data pipelines that extract, transform, and load (ETL/ELT) data from source systems (SAP S/4HANA, HubSpot, Novulo, and others) into the BI data warehouse, supporting tools such as QlikSense, SAP Analytics Cloud, and Hyperion.

Model and maintain data warehouse structures (staging, integration, and presentation layers); ensure data is clean, consistent, and fit for reporting and analytics consumption.