Uses the Pipeline framework as its base.

https://viewer.diagrams.net/?border=0&tags={}&lightbox=1&highlight=0000ff&layers=1&nav=1&title=scraper-pipeline.drawio&dark=auto#Uhttps%3A%2F%2Fraw.githubusercontent.com%2Ftech-atlas-master-thesis%2Farchitecture%2Fmain%2Fscraper-pipeline.drawio

architecture/scraper-pipeline.drawio at main · tech-atlas-master-thesis/architecture

Steps

• Get Technology Configuration
  • Get user configured technology-field and sub-category set
  • Configuration
    • Configuration + Version
• Gather FFG data
  • Uses search endpoint to gather relevant project IDs for keywords: https://projekte.ffg.at/projekt?advanced_search=1
    • HTML parsing for project IDs is still necessary
  • Uses the excel export endpoint: https://projekte.ffg.at/projekt/excel
    • Endpoint provides a fixed amount of fields
    • List needs to be de-duplicated
      • as the project is repeated for every organization associated with the project
      • abstract is only listed on the first entry.
  • Configuration
    • URLs for search and export
    • additional search parameters like start/end date
    • Configuration about HTML parsing and ID extraction
    • Configuration for how fields are mapped to new names and de-duplicated
• Gather FWF data
  • WIP
  • Uses export-csv endpoint for search and export https://fr.fwf.ac.at/iFinder/export-csv
    • Returns csv file
    • Allows user to specify fields and their labels (almost direct DB access?)
  • Configuration
    • URL for search and export
    • Search parameters (needs deeper research, if start/end date is possible)