Dashboard

Pipeline Framework

Data Scraper

Data Pipeline

Admin UI

Databases

Authentication

https://github.com/moosi312-master-thesis/architecture/blob/main/architecture.drawio

https://viewer.diagrams.net/?tags={}&lightbox=1&highlight=0000ff&layers=1&nav=1&title=architecture.drawio&dark=auto#Uhttps%3A%2F%2Fraw.githubusercontent.com%2Fmoosi312-master-thesis%2Farchitecture%2Fmain%2Farchitecture.drawio

Description

Dashboard

We want to visualise the data generated by the Data

Backend to handle queries made by the UI

UI to display data in map/details view

Data Pipeline

Take scraped data and normalize it for the dashboard (combine duplicates, bring to standardized form, gather/flag missing/additional data)

Data Scraper

Gather information from different sources and save them in an intermediary database

Admin UI

Used to start runs of Scraper and Pipeline with different configurations