Problem Statement: The Web contains vast amounts of conflicting information about entities (e.g., Persons, Movies, Events). A core challenge in Web Information Integration is that different sources use different identifiers (e.g., "J. Doe" vs. "John Doe"). Before any "Fact Checking" can occur, the system must first solve the Identity Problem: determining which heterogeneous data points refer to the same real-world object without having a shared Primary Key.
Project Goal: The goal is to design and implement an Entity Resolution Pipeline for the "FactCheck" framework. The student will build a system that ingests semi-structured data (JSON/HTML from different web sources), normalizes it, and applies similarity algorithms to link disparate records to a single "Golden Entity."
Core Tasks:
- Data Normalization: Develop a preprocessing module to clean and standardize noisy web data (e.g., handling date formats, removing special characters from names).
- Blocking & Indexing: Implement a "Blocking Strategy" to efficiently find candidate matches. (Comparing every record to every other record is too slow; the system must intelligently narrow down the search space).
- Similarity Metrics: Implement and compare different matching algorithms.
- Resolution Logic: Design the decision logic (e.g., weighted thresholding) that decides if two records are a "Match," "Potential Match," or “Non-Match.”
Research Focus:
The research focus lies on a Quantitative Performance Evaluation. The student must evaluate their pipeline against a labeled "Ground Truth" dataset (e.g., standard benchmarks like Cora/Cora-Ref or a custom scraped dataset). The analysis must report Precision, Recall, and F1-Score to scientifically justify the chosen resolution strategy.
Technologies: Python (Pandas, RecordLinkage, Dedupe); String Metrics (FuzzyWuzzy); Vector Embeddings (Optional); JSON/Semi-structured Data
Tags: Web Information Integration; Semantic Matching; Multimedia Metadata; Data Quality; Algorithm Engineering; Kalchgruber
