The RVA has recently set up a Modern Data Platform (MDP) as a center of excellence for multiple data teams. The platform consists of:

• Data ingestion: Airbyte
• Orchestration: Dagster (or similar orchestrator)
• Transformations: dbt
• Lakehouse: Databricks
• Visualization: Power BI
• Metadata: OpenMetadata

The management team requests a central monitoring solution to track the usage and performance of the MDP (uptime, errors, usage, costs, etc.). This assignment is divided into several phases.

Objective for the Data ingestion & integration phase

Technically expose all necessary monitoring data and make it centrally available in the Databricks lakehouse, based on the analysis from phase 1.

Tasks

• Implement data ingestion flows for all selected sources:
• Set up and configure Airbyte connectors (standard and/or custom)

• Set up API calls to the various tools (Airbyte, Dagster, dbt, Databricks, OpenMetadata, etc.)

• Implement authentication and connections:

• API keys, service principals, VPN/network configuration if needed

• Load raw monitoring data into Databricks (source layers / raw zones)

• Model and transform data:

• Unnesting / exploding JSON structures
• Build normalized and aggregated tables for monitoring

• Optionally use dbt for transformations

• Ensure repeatable, robust pipelines (logging, error handling, restartability)

• Handover & documentation to the dashboarding profile:

• Clear description of tables, columns, refresh logic, and dependencies

Profile

Experience as a Data Engineer with:

• Data ingestion via Airbyte (ideally also custom connectors)
• Working with Databricks (Delta Lake, notebooks, SQL/PySpark)
• API integrations (REST, JSON) and basic network concepts
• Knowledge of dbt is a strong asset
• Experience in setting up monitoring/logging pipelines is a plus, as is certification DataCamp – Data Analyst Associate and dbt Fundamentals

Together with your CV, we ask you to send your answer to the question below. Failure to submit an answer or if the answers are insufficient will result in the candidate not being considered:

How have you worked with Airbyte and Databricks in the past? Provide a concrete example.