Dashboard

Pipeline Framework

Admin UI

Databases

Authentication

https://github.com/moosi312-master-thesis/architecture/blob/main/architecture.drawio

https://viewer.diagrams.net/?tags=%7B%7D&lightbox=1&highlight=0000ff&layers=1&nav=1&title=architecture.drawio&dark=auto#Uhttps%3A%2F%2Fraw.githubusercontent.com%2Fmoosi312-master-thesis%2Farchitecture%2Fmain%2Farchitecture.drawio

Description

Dashboard

We want to visualise the data generated by the Data

Backend to handle queries made by the UI

UI to display data in map/details view

Data Scraper

Gather information from different sources and save them in an intermediary database

Data Pipeline

Take scraped data and normalize it for the dashboard (combine duplicates, bring to standardized form, gather/flag missing/additional data)

Admin UI

Used to start runs of Scraper and Pipeline with different configurations

Databases

We want data to be versioned for each scraper/pipeline run, so we can easily compare the results from different configurations.

Data pipeline should be able to choose from different Raw-Data versions Dashboard should be able to choose from different Pipeline runs