This topic is part of the FactCheck research project.
(Data) Orchestration involves the creation, coordination, and (often scheduled) execution of tasks - sometimes across systems, applications, or services. As the FactCheck framework grows and becomes more distributed, the importance of scheduling our information extraction, comparison, and analysis jobs grows. Your task is to research, implement, and compare orchestration tools for the FactCheck framework, and reflect on their core differences, strengths, and drawbacks in written form.
As part of your project, you will…
- learn about the orchestration framework landscape, and select one [P1] or more data orchestration platform(s) - e.g., Apache Airflow, Dagster, or Prefect
- gain an overview of the FactCheck prototype, and break its monolithic components into light-weight, atomic services and tasks
- design a modular and scalable FactCheck pipeline able to extract, compare, and analyze information from across the Web
- implement your pipeline design using your orchestration framework(s) of choice
- reflect on your design and the chosen orchestrator(s) in the form of a written thesis [SPBA] or report
Depending on your strengths and interests, you may place a special focus on select aspects of pipeline design or data governance such as, for example, …
- data lineage and lineage tracking using, e.g., OpenLineage
- integrating marimo notebooks to provide a visual workbench for pipeline execution and monitoring
- select aspects of observability to improve monitoring
- creating a command-line interface (CLI) that allows the administration of pipelines via a terminal
Technologies: Python; Docker; Apache Airflow; Dagster; Prefect
Tags: Aichinger; FactCheck; Data Science; Framework
