Should use the Pipeline framework as its base.

https://viewer.diagrams.net/?border=0&tags={}&lightbox=1&highlight=0000ff&edit=_blank&layers=1&nav=1&title=transformer-pipeline.drawio&dark=auto#Uhttps%3A%2F%2Fraw.githubusercontent.com%2Ftech-atlas-master-thesis%2Farchitecture%2Fmain%2Ftransformer-pipeline.drawio

architecture/transformer-pipeline.drawio at main · tech-atlas-master-thesis/architecture

Steps

• Create Data Set
  • Creates a Data Set ID for the rest of the steps to use
• Get Technology Configuration / Get Results from Scraper Pipeline
  • Gather data generated in previous Scraper Pipeline
  • Configuration
    • Pipeline Name + Step
• Extract Data
  • Extracts relevant Data from raw scraper data
    • f.e. all unique organizations
• Normalize Data
  • De-duplicate data, fix errors with data
• Enrich Data
  • Enrich with external data
    • Organizations
      • Firmenbuch-Nummer (https://www.northdata.com/, https://justizonline.gv.at/jop/web/firmenbuchabfrage/suchergebnis)
      • Geo-Location Data (https://maps.bev.gv.at/, https://nominatim.org/)
    • Programmes/Grants
      • Transparenzportal-Daten
• Save to Database
• Calculate Preprocessed Queries
  • Because every generated DataSet is immutable, we can precalculate some expensive queries, like number of projects per Technology/Grant
• Publish Data Set
  • Make Data Set available for viewing in the dashboard