RVA has recently set up a Modern Data Platform (MDP) as a center of excellence for multiple data teams. The platform consists of:

• Data ingestion: Airbyte
• Orchestration: Dagster (or similar orchestrator)
• Transformations: dbt
• Lakehouse: Databricks
• Visualization: Power BI
• Metadata: OpenMetadata

The executive team requests a central monitoring solution to track the usage and performance of the MDP (uptime, errors, usage, costs, ...). This project is divided into three consecutive phases. For this phase, we are looking for a profile with the following objective:

Technical Analysis & Scoping

Mapping out which monitoring metrics are available and needed, how they can be technically accessed, and what the conceptual end-reporting should look like.

Tasks:

• Analysis of all relevant tools within the MDP:
• Airbyte, Dagster, Databricks, dbt, Power BI, OpenMetadata, …
• Identification of desired monitoring metrics per tool such as:
• Uptime / availability
• Number of runs / pipelines
• Errors / failed jobs
• Usage of reports / datasets
• Cost-related indicators
• Determining how these metrics can be technically accessed:
• Available API endpoints
• Authentication mechanisms (API key, service principal, …)
• Possible export options (CSV/Excel/…)
• Documentation of:
• Concrete API endpoints per tool
• Required parameters & authentication
• Expected data structure (JSON, CSV, tables, …)
• Developing proposals for:
• Logical data models in the lakehouse for monitoring
• High-level design of the monitoring dashboards (mock-ups / wireframes)
• List of metric and KPI definitions serving as a basis for SLAs

Dashboarding & Reporting

Objective:

Developing an executive-level monitoring dashboard in Power BI based on the modeled monitoring data in Databricks.

Tasks:

• Setting up the Power BI – Databricks integration
• Designing and building one or more dashboards focused on:
• Platform usage (number of pipelines, reports, users, etc.)
• Reliability & quality (uptime, failed jobs, error trends)
• Performance / throughput times
• Cost indicators where possible
• Implementing:
• Clear KPI visualizations and drill-downs
• Filters, slicers, and navigation structure for executive/management
• Basic security/access model (roles, permissions) in consultation with RVA
• Optimizing performance (modeling in Power BI, aggregations, etc.)

Delivery & Handover

• User manual / short training for end users
• Documentation of the data model and reporting structure

Profile

• Strong experience as data/BI consultant
• Experience reading API documentation and performing functional/technical analyses
• Good communication with both technical teams and business stakeholders
• Experienced Power BI consultant
• Strong in data modeling and DAX
• Experience with dashboards for management/executive reporting
• Experience with Databricks sources is a plus
• Experience with monitoring or governance reports is a plus, as is a certification in DP-600 Implementing Analytics Solutions Using Microsoft Fabric

Together with your CV, we ask you to answer the question below. Failure to submit an answer or if the answers are insufficient, will result in the candidate not being considered.

Tell us more about how you have previously researched which KPIs are important for reporting and how you then implemented them.