Context

As part of the DEP (Data Exchange Platform) program, we are building an on-premise data platform for the Belgian Ministry of Defence. The platform must meet strict requirements in terms of security, governance, and data sovereignty.

You will join the Sierra Team and work alongside a Data Architect, a Solution Architect, and a BI developer.

Your main missions

Ingestion & Data Pipelines

• Design and deploy ingestion pipelines from heterogeneous sources: files, APIs, relational databases.
• Implement pipelines for incremental and streaming ingestion.
• Ensure the reliability, idempotence, and traceability of each pipeline.

Transformation & Semantic Layer

• Develop transformation models on the Bronze, Silver, and Gold layers (Medallion Architecture).
• Implement best practices for data historization (e.g.: SCD2, upsert, append only, …).
• Develop exports to Delta Lake.

ACID Transactions & Time Travel

• Manage and maintain Delta Lake tables.
• Ensure transactional consistency of data in a multi-domain environment.

Orchestration

• Implement and maintain assets to be orchestrated.
• Monitor runs, manage errors, and automate alerts.

Governance & Lineage

• Push technical and business metadata to the data platform.
• Maintain end-to-end lineage.
• Contribute to the drafting of DUAs, SLAs, and associated governance documents.

Data Exposure

• Develop endpoints to expose the Gold layer to BI Dashboards.
• Collaborate with reporting teams for feeding BI dashboards.

Profile sought

Required Technical Skills

• Proficient in Python
• Proficient in SQL
• Proven experience with dbt-core (models, snapshots, macros, tests, profiles).
• Good knowledge of DuckDB as an embedded analytical query engine.
• Experience with Delta Lake (ACID transactions, time travel, OPTIMIZE/VACUUM, …).
• Knowledge of relational databases: MSSQL, PostgreSQL (advanced SQL, CDC).
• Experience with a data orchestrator: Dagster, Airflow or equivalent.
• Comfortable with Linux/OpenShift and PowerShell (Windows dev environment).
• Familiarity with governance tools: DataHub, OpenMetadata or equivalent.
• Experience in BI development.

Appreciated Skills

• Experience with Kafka/Debezium for real-time data capture.
• Knowledge of Spark (Spark SQL, Thrift Server, Beeline).
• Experience with Power BI Report Server (PBIRS) or Apache Superset.
• Sensitivity to data security in restricted-access environments.
• Knowledge of Lakehouse concepts (Medallion Architecture), DWH and Datalake.

Soft Skills

• Rigor and autonomy in managing complex multi-domain pipelines.
• Ability to document technical decisions and explain them to non-technical stakeholders.
• Team spirit.
• Curiosity and a solution-oriented mindset.

What we offer

• A stable, long-term mission.
• A high-impact project in a secure and technically challenging environment.
• A modern and open-source stack: DLT, dbt, DuckDB, Delta Lake, Dagster, …
• An experienced and supportive team, focused on quality and best practices.
• Opportunities to develop skills in governance, orchestration, and lakehouse.

Given the nature of the activities, the candidate must have Belgian nationality.