Knowledge graphs generation from cultural heritage texts: combining LLMs and ontological engineering for scholarly debates Open Access

Knowledge graphs (KGs) have become the standard approach for representing and sharing cultural heritage (CH) information in the Linked Open Data (LOD) ecosystem, enabling interoperability between Libraries, Archives and Museums institutions (Barabucci et al., 2021). This effort has been concentrated for the most part on creating KGs of metadata, with diversified workflows dedicated to converting semi-structured or already structured sources (catalogues, inventories) into LOD (Bernasconi and Ferilli, 2024). However, the knowledge contained in unstructured texts (descriptive content, contextual information and analytical discourse) remains difficult to extract and structure systematically into queryable formats, and even when integrated into KGs, it is usually kept in long and description string fields (Barabucci et al., 2021; Giagnolini et al., 2025). Scholarly authenticity assessment debates exemplify this challenge, where complex interpretative knowledge is embedded in natural language discourse but practically absent from structured representations. Additional challenges stem from the inherently interpretative nature of humanities scholarship, which aligns with a constructivist epistemology viewing knowledge as situated, provisional and shaped by the observer's perspective. Checkland and Holwell distinguish between data – passively recorded facts – and capta – knowledge actively constructed by the observer. This distinction challenges the realist assumption that often underpins data practices, in which data are treated as an objective and context-independent representation of reality (Peter and Holwell, 2006). These epistemological tensions manifest across various forms of CH scholarship, from attribution studies and provenance research to interpretative analysis and critical evaluation. In authenticity assessment, scholars from different humanities disciplines (e.g., Diplomatics, Paleography, Philology, History) and scientific fields (e.g., Forensics, Materials science, Chemical analysis) frequently arrive at divergent conclusions based on different evidential priorities (Barone, 1912). Inherent factors contributing to this diversity include historical uncertainty, gaps in documentary transmission and subjectivity (Blau, 2011; Gadamer, 2013). Recent theoretical advancements acknowledge the subjectivity and uncertainty inherent in interpreting CH data, recognizing these as essential epistemic characteristics that must be preserved in digital representations (Pasqual, 2025; Piotrowski and Neuwirth, 2020; Piotrowski, 2023). However, current KG implementations represent only simplified versions of scholarly discourse. Whether dealing with artistic attribution, provenance disputes, historical interpretation or authenticity assessment, complex scholarly reasoning gets reduced to simple categorical assertions. Major knowledge bases like Wikidata ^[1] and DBpedia ^[2] exemplify this limitation. While Wikipedia articles contain rich discussions with detailed scholarly arguments, evidence analysis and alternative hypotheses, their structured counterparts reduce this complexity to sparse, categorical statements that fail to capture the evidential reasoning, methodological disagreements and evolving consensus that characterize authentic scholarly discourse. Consider the famous Donation of Constantine, a supposed 4th-century decree by Emperor Constantine transferring authority over Rome and the western Roman Empire to the Pope. In the 15th century, Lorenzo Valla exposed the document as a forgery through philological analysis (Valla, 2023), demonstrating that its Latin contained anachronisms from the 8th rather than the 4th century. Despite Valla's compelling evidence, acceptance of this finding evolved gradually over centuries.

As shown in Figure 1, Wikidata categorizes the Donation as a “historical forgery” ^[3] with no representation of the scholarly debate, while DBpedia ^[4] similarly lacks structured representation of the authenticity discourse. In contrast, the corresponding Wikipedia page contains extensive discussions of Valla's philological arguments, the specific linguistic evidence, the Church's resistance and subsequent scholarly confirmation. Three interconnected challenges exist. The first is syntactic: representing competing scholarly opinions within formal knowledge representation (KR) systems requires sophisticated mechanisms that traditional implementations struggle to handle effectively (Pasqual, 2025). While theoretical frameworks such as RDF-star, Named Graphs and reification methods provide the necessary expressive power, their practical application demands complex modeling decisions about contradictory evidence, evolving consensus and methodological disagreements, often resulting in oversimplified categorical assertions or unmanageably complex representations. The second challenge is practical: extracting complex scholarly information from textual sources requires enormous manual labor, creating insurmountable scalability barriers. This process requires expert annotators to identify scholarly agents, extract evidential reasoning and capture alternative hypotheses while maintaining consistency across large document collections. CH institutions possess vast textual resources containing sophisticated scholarly analyses, but lack practical means to transform this knowledge into queryable, machine-readable formats. Large Language Models (LLMs) present a promising solution due to their ability to process complex academic discourse, identify implicit relationships and handle domain-specific vocabularies (Khorashadizadeh et al., 2024) without additional training. LLMs can bootstrap knowledge extraction (KE) pipelines by eliminating the need for specific annotated training corpora and exploiting transfer learning across domains beyond those on which they were explicitly trained (Brown et al., 2020).

The third challenge is methodological: despite advances in both ontology development and extraction techniques, the literature lacks systematic methodologies that integrate ontology-driven KG generation specifically for CH contexts. While general-purpose text-to-KG frameworks exist (Maynard et al., 2017; Hotho et al., 2020), they typically assume the availability of large annotated training corpora and domain-agnostic entity types. What remains absent is a comprehensive methodological framework that guides practitioners through the complete process: from analyzing source materials and identifying extractable patterns, through developing appropriate annotation schemas and extraction pipelines, to evaluation. Such a methodology must accommodate the resource constraints typical of CH institutions while maintaining the representational fidelity required for humanities scholarship.

To systematically address this primary question, we investigate the following sub-questions, which we answer using our authenticity assessment case study:

1. Methodological Framework: What methodological approach can effectively integrate LLM-based KE with existing ontological frameworks to capture complex scholarly interpretations in CH texts?

2. Extraction Performance: How accurately can systematic LLM-based pipelines extract different components of scholarly discourse, including metadata, agents, evidential reasoning and interpretative hypotheses?

3. Representation Fidelity: Do automatically generated KGs adequately represent the complexity and nuance of scholarly interpretations when following structured methodological approaches?

4. Model Comparison: How do different LLMs perform within structured extraction pipelines for CH texts, and what are the implications for cost-effective deployment?

5. Methodology Validation: What insights does authenticity assessment validation provide about the methodology's broader applicability to other forms of CH interpretative scholarship?

Our contributions are threefold. First, we present the ATR4CH (Adaptive Text-to-RDF for Cultural Heritage) methodology, which combines annotation development, ontological alignment and pipeline-based extraction. The methodology is a replicable framework that can be adapted across CH domains and institutional resources. Second, we demonstrate the practical implementation by connecting ATR4CH to the Scholarly Evidence Based Interpretation (SEBI) ontology ^[5] (Pasqual, 2025) to develop an annotation model and an extraction pipeline for complex scholarly discourse. Third, we provide a comprehensive evaluation on a manually annotated sample of Wikipedia articles, establishing performance benchmarks across multiple extraction tasks and model architectures.

Our evaluation demonstrates the methodology's effectiveness, achieving F₁-scores of 0.96–0.99 for metadata extraction, 0.7–0.8 for scholarly entity recognition, 0.65–0.75 for hypothesis extraction and 0.95–0.97 for evidence extraction, with 0.62 G-EVAL overall discourse representativeness.

The remainder of the paper is organized as follows: Section 2 reviews related works in KR, extraction methods for the Semantic Web, opinion mining and LLM-based KE approaches. Section 3 presents the ATR4CH methodology, detailing the five-task iterative approach for integrating LLM-based extraction with ontological frameworks in the CH domain. Section 4 describes ATR4CH implementation for the authenticity assessment use case, following the five-task structure: Section 4.1 presents foundational analysis and design, including corpus collection, the SEBI ontology and preliminary analyses; Section 4.2 describes iterative annotation schema development and ground truth (GT) preparation; Section 4.3 details pipeline architecture development; Section 4.4 and Section 4.5 describe pipeline refinement, KG generation and comprehensive evaluation. Section 5 provides experimental results across five evaluation questions (EQs), comparing Claude Sonnet 3.7, Llama 3.3 70B and GPT-4o-mini on metadata extraction, entity recognition, evidence mining, hypothesis extraction and overall KE fidelity. Section 6 discusses findings in relation to research questions, analyzes performance trade-offs, addresses deployment implications and outlines contributions, limitations and future directions. All code is available at https://github.com/aschimmenti/SEBI-Knowledge-Extraction.

The challenges of representing and extracting interpretative knowledge in the CH domain have received increasing attention in recent research. We focus first on conceptual and ontological models developed for multi-perspective KR, and then turn to methods for extracting such interpretations from unstructured texts, including recent advances in LLMs.

Recent theoretical advancements acknowledged the subjectivity and uncertainty inherent in interpreting CH data, recognizing these aspects as essential epistemic characteristics in analyzing and representing such data (Piotrowski and Neuwirth, 2020). Uncertainty in the CH domain arises not only from the data itself (for example, data extracted from the digitization of a birth certificate) but also from the interpretative connections made by scholars regarding such data, such as identifying a name on a birth certificate with a specific historical figure (Piotrowski, 2023). However, these advancements have not translated into widely adopted practical tools and standards in KGs. LOD is the standard for encoding and publishing CH data on the Web, promoting interoperability and data exchange between institutions. Standard online catalogues (e.g., Europeana) ^[6] typically provide single-perspective flat metadata, relegating discussions, debates and uncertain facts to free text descriptions (Barabucci et al., 2021).

To the best of our knowledge, Wikidata is the only large-scale data catalogue that employs a custom reification method to integrate claims with varying degrees of truthfulness, i.e., its ranking mechanism. Other knowledge bases like YAGO4 ^[7] integrate RDF-Star to model provenance (e.g., for temporal information) but lack statements on the certainty of the given triple (Govindapillai et al., 2021). Despite the adequate expressive power made available by the Wikidata model, annotators in the CH domain underutilize this feature. Additionally, claims related to CH data often make use of numerous qualifiers to encode contextual metadata, likely due to the increased effort required for this type of annotation (Di Pasquale et al., 2024).

Some ontologies have been designed to structure multi-perspective representations in CH data. ICON (Sartini et al., 2023; Baroncini et al., 2023) encodes visual recognitions in art history using n-ary relations to encode contextual metadata. Digital Hermeneutics (Daquino et al., 2020) employs a layered approach using Named Graphs (Carroll et al., 2005) to represent scholarly interpretations in archival and literary sources. HiCo (Daquino and Tomasi, 2015) and the STAR model (Andrews, 2023) have been designed to represent historical interpretations and arguments. Previous work has introduced the SEBI ontology ^[8] Pasqual (2025), aimed at representing scholarly claims on a hand-curated catalogue of forged manuscripts, which resulted in the BROAST application ^[9].

KE for the Semantic Web transforms unstructured textual content into machine-processable representations conforming to established ontological models. The text-to-KG task refers to systems that process natural language text to generate RDF KGs, where extraction is guided by predefined ontologies and outputs conform to ontological constraints (Maynard et al., 2017). This approach, termed Closed KE or Ontology-Based Information Extraction (Wimalasuriya and Dou, 2010), operates within predefined ontological frameworks specifying permitted entity types, relations and semantic constraints.

The Closed KE paradigm aligns with the Semantic Web vision, as extraction targets conform to ontological schemas designed for machine reasoning and cross-system integration (Berners-Lee et al., 2001). The requirements for ontology-oriented extraction systems manifest across three critical dimensions (Hotho et al., 2020): entity mentions must be mapped to URIs serving as globally unique identifiers within the LOD cloud, elevating Entity Linking from optional refinement to fundamental requirement; data alignment challenges emerge across T-Box alignment (schema extension for novel entity types), A-Box alignment (deduplication and consistency checking) and URI alignment (entity resolution against existing knowledge base entries); provenance tracking and validation mechanisms must ensure generated RDF conforms to both syntactic requirements and semantic constraints defined in the ontology. Text-to-KG systems following the Closed KE paradigm typically decompose extraction into specialized subtasks executed sequentially: preprocessing and sentence segmentation, Named Entity Recognition (NER) to identify and classify entity mentions, Entity Linking to map mentions to canonical identifiers, Relation Extraction to identify semantic relationships, Event Detection when using event-centric ontologies and graph assembly with validation (Maynard et al., 2017). Recent CH projects demonstrate diverse pipeline implementation approaches. The Musical Meetups Knowledge Graph (Alba Morales et al., 2023) combines DBpedia Spotlight entity recognition with LLMs (GPT-3.5-Turbo) for temporal normalization and purpose classification. MusicBO (Gangemi et al., 2024) employs Abstract Meaning Representation as an intermediate layer (Meloni et al., 2017) between linguistic structure and ontological requirements. The Odeuropa project (Lisena et al., 2022) achieves tight annotation–ontology integration through frame-based schemes mapping directly to CIDOC CRM extensions using multilingual BERT models.

LLMs have introduced new text-to-KG paradigms (Khorashadizadeh et al., 2024; Mihindukulasooriya et al., 2023; Meyer and Stadler, 2024). Allen et al. (2023) identify hybrid neuro-symbolic systems and natural language interfaces as primary architectural directions, the latter enabling domain experts to guide extraction without technical expertise in KR. Lairgi et al.’s iText2KG (Lairgi et al., 2024) employs a zero-shot, incremental approach enabling knowledge base expansion without annotated training data. Ringwald (2024) explores pattern-based methods for learning from Wikipedia-DBpedia/Wikidata pairs. For CH-specific applications, Santini (2024) shows LLM-based relation extraction outperforming specialized models like mREBEL on 19th-century Italian texts through broader linguistic knowledge. Schimmenti et al. (2024) demonstrate zero-shot tools like GLiNER (Zaratiana et al., 2024), enabling on-the-fly entity type specification through natural language descriptions. Giagnolini et al. (2025) employ Llama 3.3 70B for text-to-KG extraction from archival metadata, classifying paragraphs by event types before applying event-specific extraction schemas mapped to RiC-O ^[10].

The integration of LLMs alters resource requirements and performance characteristics. LLMs leverage In-Context Learning (ICL) (Brown et al., 2020), Few-Shot learning strategies (Petroni et al., 2019) and Chain-of-Thought (CoT) prompting to perform extraction with minimal task-specific examples. This capability proves valuable for domains where creating large annotated corpora is often infeasible. On the other hand, LLM-based approaches introduce new challenges: ontological compliance requires sophisticated prompt engineering or post-processing, hallucination risks necessitate careful verification mechanisms and the implicit knowledge encoded in model parameters may not align with domain-specific conceptual frameworks embodied in CH ontologies.

Evaluation methodologies for generated KGs remain heterogeneous and often inadequate for capturing semantic fidelity beyond surface-level metrics. Traditional precision–recall calculations on entity mentions and relations fail to assess whether extracted knowledge structures adequately represent the complexity and nuance of scholarly discourse. Back-translation approaches (Gangemi et al., 2024) and LLM-based evaluation frameworks like G-EVAL (Liu et al., 2023) offer promising directions, but their applicability across different types of CH interpretative content requires further validation.

This section introduces the ATR4CH methodology, an iterative approach for extracting KGs from CH documents using LLMs. ATR4CH recognizes the fundamental interdependence of annotation, KE and ontology alignment, approaching them conjunctively.

The ATR4CH methodology transforms three foundational inputs – unstructured document corpus, target ontology and Competency Questions (CQs) – into a validated KE system through five interconnected tasks, producing a working extraction pipeline, refined annotation model and comprehensive evaluation framework.

A flowchart is shown in Figure 2. The methodology draws on eXtreme Design (Presutti et al., 2009) for iterative ontology engineering centered on CQs, selection methodologies (Tomasi, 2020) and Odeuropa's integrated annotation–ontology approach (Lisena et al., 2022).

ATR4CH focuses on integrating annotation and KE pipeline development with ontologies in the CH domain, including CIDOC CRM (Doerr, 2003), Dublin Core ^[11], FRBR/FRBRoo (IFLA Working Group on FRBR/CRM Dialogue, 2017), HiCO (Daquino and Tomasi, 2015), SKOS ^[12] and PROV-O (Lebo et al., 2013). The methodology presupposes dual alignment: the ontology must represent relevant domain knowledge, and this knowledge must be present (explicitly or inferably) within source documents. It suits unstructured texts (informative, narrative, scholarly sources) rather than semi-structured documents like catalogues. The methodology leverages LLM capabilities in ICL (Brown et al., 2020), Few-Shot and CoT strategies (Petroni et al., 2019; Lairgi et al., 2024), based on established KE practices (Tamasauskaitė and Groth, 2022).

ATR4CH adopts an incremental, pattern-by-pattern development strategy, iteratively focusing on one Core Ontological Pattern (COP) at a time, identified through target ontology and CQ analysis. Each COP progresses through the complete development cycle – annotation schema design, RDF mapping validation and automated extraction module implementation – before advancing to the next pattern. After several pattern-specific iterations, the methodology transitions from the Pilot Corpus phase to full-scale corpus processing. The annotation model is consolidated into a production-ready version to work as the GT for pipeline evaluation.

Task I establishes foundational understanding by analyzing the corpus and ontology to identify COPs, addressing data sparseness common in Information Extraction from unstructured texts.

1. Corpus Analysis: This ontology-dependent activity examines knowledge manifestation in textual discourse, including linguistic patterns, discourse structures and representational strategies. Challenges include implicit mentions requiring contextual inference, long-distance dependencies where KG components are separated by substantial text spans, nested entities in relational structures and ambiguous references. For Wikipedia articles about forged CH items, structural analysis identifies which sections contain scholarly opinions versus tangential debates, enabling focused extraction from high-density sections like “Scholarly analysis.” Content analysis determines whether articles present complete scholarly reasoning or merely final judgments, guiding the methodology toward sources with sufficient depth.

2. Ontology Analysis: This parallel activity assesses which ontology parts can be populated from source documents, examining alignment between the ontology's conceptual framework and available textual information. It identifies which classes and properties have sufficient textual evidence, which relationships can be inferred from corpus patterns and which elements need omission. The aim is to determine what data are present rather than immediately addressing how to extract. CQs guide prioritization of ontological coverage based on research requirements.

3. COPs Identification: Based on analyses, this process identifies essential KG patterns required to answer CQs. COPs represent central ontological nodes and relationships that are both present as extractable information and necessary for addressing research questions. Patterns emerge through systematic intersection analysis of CQs, ontological capabilities and textual evidence. Each candidate pattern is evaluated on necessity (required for answering CQs?) and feasibility (sufficient textual evidence for reliable extraction?). Patterns scoring high on both dimensions form the initial set, refined by considering dependencies and complexity. Simpler patterns are prioritized for early iterations to establish baseline functionality. The final selection represents a manageable subset forming the semantic backbone for KE, with patterns ordered by their structural role – foundational metadata patterns preceding interpretative reasoning patterns. These COPs will be processed incrementally through subsequent tasks.

4. Pilot Corpus Selection: The Pilot Corpus is a representative document set serving as a development sandbox. It is not a quantitatively representative sample but a qualitative one that must be linguistically, structurally and epistemically representative while remaining manageable for intensive manual work. Selection ensures coverage of linguistic patterns, discourse structures and diverse COP manifestations. Size can be three to five documents, depending on length and information complexity. This set will be used iteratively for developing and validating extraction pipelines for each COP.

Task II develops annotation schemas incrementally, processing one COP at a time. For each COP identified in Task 3.2, an annotation schema is developed, applied to the Pilot Corpus and validated through RDF mapping before proceeding to the next pattern, producing an annotation model serving as the target schema for automated extraction.

Pattern-by-Pattern Annotation Schema Development: Development proceeds iteratively through identified COPs. For each pattern, an annotation schema captures essential knowledge structures while remaining practical for manual annotation and automated extraction. Schema design accounts for diverse knowledge manifestations in the corpus, including explicit textual mentions and information requiring inference or contextualization.

The annotation schema should prioritize simplicity and feasibility while ensuring adequate coverage. “Minimal” refers to including only necessary annotation elements for extracting identified ontological patterns in the first iterations, avoiding over-annotation that complicates extraction without contributing to answering CQs. If COPs require complex semantic structures beyond simple triple patterns, the annotation schema should include appropriate mechanisms for representing these relationships mappable to RDF (e.g., Named Graphs, reification).

Knowledge Base Integration Strategy: Knowledge base integration enables consistent entity identification and vocabulary alignment between textual mentions and the target ontology. Since COPs typically involve ontological individuals, entities, controlled vocabularies or standardized terminologies, annotators need access to these resources to ensure textual references link to correct ontological entities, preventing inconsistent annotation that would hamper aggregation and reasoning in the final KG.

Integration – whether through local vocabularies or external resources like Wikidata or DBpedia – must be designed early to establish clear protocols for entity linking and vocabulary alignment, guiding both manual annotation and automated extraction in Task 3.4. Choice between local and external knowledge bases depends on domain coverage, data quality requirements and specific entity types required by COPs.

Annotation Paradigm: Annotation should follow established corpus linguistics and NLP practices. When resources permit, multiple annotators should annotate the same documents to enable inter-annotator agreement measurement using Cohen's kappa (Carletta, 1996; Cohen, 1960) or Krippendorff's alpha (Krippendorff, 2019), identifying ambiguous categories and revealing where guidelines require clarification. In resource-constrained settings, a single experienced annotator may suffice, but annotation guidelines must be thoroughly documented for reproducibility. The process should be iterative: initial guidelines are refined based on encountered edge cases.

Iterative Development Process for Each COP: Development follows a systematic cycle, ensuring the annotation schema produces RDF structures satisfying COPs:

1. Schema Design: Develop initial annotation layers based on the current COP, incorporating knowledge base integration protocols through tagsets, controlled vocabularies and standardized terminologies aligning with the target ontology.

2. Pilot Corpus Annotation: Annotate the entire pilot corpus using the current schema iteration to identify gaps, inconsistencies or bottlenecks.

3. Mapping Validation: Conduct preliminary mapping exercises from annotated data to RDF format, testing whether resulting KGs satisfy the COP and adequately represent source document semantic content. This mapping serves as a unit test, validating that annotation patterns correctly transform to valid RDF.

4. Schema Refinement: Refine the annotation model based on issues identified during mapping validation, returning to previous activities as necessary.

5. Pipeline Development for Current COP: Once the annotation schema has been validated through successful RDF mapping, proceed to Task 3.4 to develop automated extraction for this pattern using the same Pilot Corpus documents. Only after completing pipeline development and validation for the current COP should development proceed to the next pattern.

These preliminary mapping exercises validate that the annotation schema produces target knowledge structures, serving as an early validation mechanism before proceeding to automated extraction development. The Minimal Working Annotation emerges as the aggregation of annotation schemas developed for individual COPs.

Task III designs and implements computational tools to automatically extract COPs from text using the annotation model as the target schema, addressing the CH corpora's domain-specific characteristics and limited annotated training data. This task develops and validates extraction capabilities incrementally for each COP using the same Pilot Corpus documents annotated in the previous task.

Task Decomposition and Architecture Design: KE is designed around annotation model elements, prioritizing based on COP semantic importance and accounting for information manifestation patterns from corpus analysis. This modular approach enables incremental extraction where KG components are progressively identified through sequential processing, facilitating debugging and targeted optimization while minimizing error propagation. For the current COP, the extraction pipeline targets the relative annotation schema developed in Task 3.3.

Tool Selection Strategy: Tool choice aligns with available resources and data characteristics:

1. Low data, low resources: API-based LLMs with few-shot prompting and rule-based entity linking

2. Moderate data, moderate resources: Hybrid approaches combining pre-trained models with domain-specific fine-tuning

3. Large data, extensive resources: Custom model training and ensemble methods

4. Large data, low resources: Structured pipeline approaches leveraging smaller models with knowledge distillation.

LLM-based approaches use structured output generation through JSON schemas (Schick et al., 2023; Qin et al., 2024) and ICL strategies (Brown et al., 2020; Min et al., 2022), combined with specialized NER tools (Devlin et al., 2019) for precise span identification when character-level accuracy is critical.

Pipeline Implementation: Development targets the annotation schema for the current COP, integrating knowledge base resources and vocabulary standardization protocols through prompt integration or RAG (Lewis et al., 2020). Initial implementation focuses on basic functionality before optimization.

Immediate Validation and Benchmarking: Once extraction for the current COP has been implemented, the pipeline is tested on the Pilot Corpus and results are compared against manual annotations from Task 3.3. Evaluation strategies range from basic (standard metrics on a pilot corpus) to comprehensive (ablation studies and hybrid approach exploration) based on project constraints. This immediate validation enables rapid identification of extraction bottlenecks or misalignments before proceeding to the next COP.

Output: An extraction pipeline capable of processing raw text and generating structured outputs following the annotation schema for the specific pattern being developed. Only after successfully validating extraction for the current COP should development proceed to the next pattern, returning to Task 3.3 to develop its annotation schema. This cycle continues until extraction pipelines have been developed and validated for all identified COPs using the Pilot Corpus.

Task IV harmonizes the COPs (Task 3.2), annotation schemas (Task 3.3) and pipeline components (Task 3.4) into a coherent end-to-end KE system, transitioning the experimental pipeline to production-ready status. The annotation model emerging from processing individual COPs is consolidated into a production version suitable for full corpus processing.

Pipeline Integration: Modular extraction components developed for individual COPs are integrated into a unified text-to-KG pipeline. Integration addresses dependencies between patterns, ensures consistent entity resolution across components and optimizes overall processing architecture. The integrated pipeline processes documents from raw text to complete KGs, instantiating all identified COPs.

End-to-End Pipeline Testing: Comprehensive testing over the pilot corpus processes documents from raw text to final KGs, revealing systematic issues including data sparseness patterns, interaction effects between COP extractors, inconsistent tool coverage across discourse types and representation generation errors. Testing systematically evaluates performance across document types and semantic phenomena, with particular attention to error propagation through pipeline stages.

Annotation Model Refinement to Production Version: Based on testing results, the annotation model evolves into a production-ready version suitable for both manual annotation and automated extraction. This may involve adding elements crucial for automated extraction – coreference chains spanning multiple COPs, disambiguation tags and confidence indicators – while maintaining backward compatibility with COPs. The production annotation model serves as the schema for GT creation in Task 3.6.

Mapping Algorithm Enhancement: Preliminary mapping algorithms from Task 3.3 are consolidated and enhanced to handle the complete KG structure, improving handling of complex semantic structures emerging from COP interactions and adding validation using tools such as SHACL, OWL reasoners, SPARQLAnything (Asprino et al., 2023) and RML (Dimou et al., 2014). Error handling mechanisms manage extraction failures and partial results.

The final validation phase employs technical validation and domain-expert evaluation to ensure knowledge structures accurately represent domain-specific discourse complexity, applying the refined system from Task 3.5 to test data separate from the Pilot Corpus.

GT Preparation: Comprehensive GT creation using the production annotation model involves annotating a test dataset separate from the pilot corpus, covering all COPs from Task 3.2. Annotation follows the same paradigms established in Task 3.3, including multiple annotators and inter-annotator agreement measurement when resources permit. The GT serves as the gold standard for systematic evaluation, with mapping algorithms from Task 3.5 applied to generate reference RDF. Test dataset size should balance evaluation rigor with annotation resource constraints, typically ranging from 10 to 50 documents, depending on length and complexity.

KE: Test datasets are processed through the complete pipeline under realistic deployment conditions, with systematic documentation of performance and failure modes. This represents the first application of the extraction pipeline beyond the Pilot Corpus used for development.

Multi-Level Evaluation: Multiple complementary approaches address KG evaluation challenges:

1. Technical Evaluation: Component-level assessment using precision, recall and F₁-score evaluates individual extraction tasks independently for each COP. Coverage analysis for CQs examines whether the KG contains sufficient information to answer the original CQs.

2. Semantic Evaluation: KG “rehydration” (Gardent et al., 2017; Gangemi et al., 2024) enables comparison when structural alignment is impossible. This approach reconstructs natural language text from KG information, using metrics like BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), BARTScore (Yuan et al., 2021), CHRF+++ (Popović, 2015) and G-EVAL (Liu et al., 2023) as proposed by He et al. (2025).

3. Competency-Based Evaluation: SPARQL query suites derived from original CQs verify that the KG satisfies the functional requirements that motivated its construction, aligned with evaluation practices from eXtreme Design (Presutti et al., 2009) using tools like TestaLOD (Carriero et al., 2019).

Domain Expert Validation: A comprehensive review by domain specialists evaluates extraction quality and coherence, with rehydration enabling evaluation by experts without RDF expertise by presenting KG content as natural language.

Iteration Strategy: Evaluation results may trigger returns to earlier tasks: coverage issues to Task 3.3 or Task 3.5, extraction bottlenecks to Task 3.4, systematic errors requiring architectural restructuring in Task 3.5 or fundamental ontological misalignments necessitating a return to Task 3.2 for COP reassessment.

This section describes the implementation of the ATR4CH methodology for authenticity assessment debates. The methodology operates on the intersection between ontological representation and textual evidence: the entities, relations and concepts defined by the SEBI ontology must correspond to information that can be identified and extracted from the documents. Following the five-task structure of ATR4CH (Section 3), we present: foundational analysis and design (Section 4.1), iterative annotation schema development (Section 4.2), pipeline architecture development (Section 4.3), system integration and refinement (Section 4.4) and comprehensive evaluation (Section 4.5).

This subsection implements Task I of the ATR4CH methodology (Section 3.2), establishing a foundational understanding of source materials by analyzing both corpus and ontology to identify COPs for KE. As specified in the methodology, Task I requires three foundational inputs: a corpus of unstructured documents, a target ontology defining KR and CQs specifying information requirements. This subsection presents these inputs and the analytical activities that identify extractable patterns.

Following Task I guidelines (Section 3.2), we collected Wikipedia articles on historical forgeries, hoaxes and authenticity controversies through web scraping from Wikipedia's categorical organization system. Initial selection covered 31 categories, including Document and Literary Forgeries, Historical Myths, Conspiracy Theories, Pseudepigraphy and Political forgery. From 1,301 retrieved documents ^[13], 16 categories and 717 articles were excluded because they presented no scholarly debate or were not about CH items. The final dataset encompasses 581 articles across 15 categories (Table 1).

Articles average 8,150 characters and 1,249 tokens, with unique vocabulary averaging 464 tokens per article, indicating substantial lexical diversity. Figure 3 shows a right-skewed distribution, with most articles ranging from 2k to 15k characters and outliers extending beyond 40k characters.

Distribution across categories (Figure 4) reflects the natural prevalence of different forgery types, with literary forgeries representing the largest category (138 articles), followed by pseudepigraphy forms (136 articles across subcategories), and archaeological and artistic forgeries (132 articles combined).

Token count distribution by category (Figure 5) reveals substantial variability. The Shakespeare authorship question demonstrates the highest token density (nearly 10K tokens), while musical hoaxes and modern pseudepigraphy exhibit more consistent, moderate-length articles.

Notable examples include the Demodocus ^[14], a fabricated Platonic dialogue whose Wikidata entry ^[15] employs a deprecated rank for authorship attribution to Plato (Figure 6), demonstrating how existing knowledge bases represent disputed attributions.

Implementing the ontology analysis component of Task I (Section 3.2), we examined the SEBI ontology to assess which ontological elements can be populated from the source documents. The SEBI ontology ^[16] (Pasqual, 2025) was developed based on scholarly articles (e.g., Härtel, 2017), a catalogue describing 153 known forgeries from Styria (Haider, 2022), and discussions with an expert diplomatist. The data model represents authenticity assessment claims using RDF-star (Hartig, 2017) as a reification method to represent (possibly concurrent) claim contents and contextual information (Daquino et al., 2020).

Each claim provides information about the document: authenticity classification, date and place of creation, author, and intention behind creation. Contextual information includes evidence collected by scholars to reach conclusions using evidence-based evaluations, the author of the claim and relevant bibliographic entries (using HiCo ^[17] and PROV-O ^[18]). RDF-star (Hartig, 2017) was chosen as the reification method to express both claim content and context, allowing representation of the complete evaluation process.

As shown in Figure 7, each claim contains an authenticity classification. Items are instances of sebi:Forgery, sebi:Authentic, sebi:FormalForgery or sebi:ContentForgery, all subclasses of sebi:Document.

Additionally, each RDF-star quoted triple includes details such as:

1. the believed creator: sebi:Document → dct:creator → dct:Agent

2. date of creation: sebi:Document → dct:date → time:Interval

3. location of creation: sebi:Document → dct:coverage → dct:Location

4. intention behind creation: sebi:Document → sebi:intended → sebi:Intention.

The dct:date property connects to time:Interval, which includes time:hasBeginning and time:hasEnd properties to specify creation periods and handle fuzzy time-spans. Concerning contextual information (Figure 8), each interpretation (set of claims represented as quoted triples) is categorized as hico:InterpretationAct connected to prov:Agent for authoriality and linked to supporting evidence (sebi:support sebi:Evidence).

Document features and their evaluation are components of the ontology. Document features (sebi:Feature) are extrinsic features (sebi:ExtrinsicFeature), intrinsic ones (sebi:IntrinsicFeature) or provenance information (sebi:Provenance), capturing aspects such as ink, support, handwriting and orthography. Each feature is evaluated on established criteria (sebi:Evidence) such as consistency, presence, completeness, veridicality and reliability. A score is associated with each evidence as xsd:Literal using forgont:hasEvaluationScore. The evaluation score allows integration of negatives (e.g., absence of signature is represented as evidence based on the feature “authentication marks” with evaluation “presence” and score false or 0).

We identified COPs following the COP identification process specified in Task I (Section 3.2). The identification process involved: (1) assessing alignment between CQ, ontological structures and available textual content, (2) identifying patterns with sufficient textual evidence for reliable extraction, (3) prioritizing based on extractability feasibility and CQ relevance and (4) selecting a manageable subset forming the semantic backbone for KE.

Four COPs were identified for extraction, presented in hierarchical priority order:

1. CH Item Metadata – Alleged information that the object claims about itself (creator, date, location) before scholarly critical analysis. For instance, in the case of the Donation of Constantine, a relevant CQ would be “What does the Donation claim about its author, date and place of creation?”

2. Scholarly Opinions – Authenticity assessments expressed by scholarly agents, classified as Authentic, Forgery, FormalForgery or ContentForgery. A typical pattern would be “Scholar X evaluates Document Z and concludes Authenticity Status S.” For instance, “Which scholars identify the Donation of Constantine as authentic?”

3. Evidential Features – Characteristics examined by scholars to support assessments, organized by type with evaluations. Example CQ: “What evidence does Lorenzo Valla cite to support his assessment of the Donation?”

4. Alternative Hypotheses – Competing scholarly claims about the actual creator, date, location or intended purpose. Example CQ: “What are the competing scholarly hypotheses about the actual date of the Donation's creation?”

These COPs emerged from the intersection of CQs, ontological structures and extractable content patterns identified in corpus analysis, following the systematic approach described in the methodology.

Following the Task I guidelines for pilot corpus selection (Section 3.2), we defined a qualitative sample of the corpus. Seven articles were chosen (Donation of Constantine, Eremin Letter, Getty Kouros, Historia Augusta, Life of Homer, Marriage Charter of Empress Theophanu and Protocols of the Elders of Sion), each belonging to a different category. Selection criteria included: (1) presence of multiple scholarly perspectives, (2) clear attribution of claims, (3) discussion of evidence-based reasoning and (4) representation of different temporal periods and document types.

This subsection documents the implementation of Task II (Section 3.3), developing annotation schemas incrementally by processing one pattern at a time. Each pattern identified in Task I received iterative schema development, pilot corpus application and RDF mapping validation before proceeding to the next pattern.

Annotation Schema Design. Following Task II guidelines, the annotation model was developed through INCEpTION (Klie et al., 2018), implementing three core patterns from SEBI corresponding to the first two COPs: CH item metadata, scholarly agents and authenticity opinions. CH item metadata establishes entity linking using INCEpTION's Wikidata integration, reconciling item types to DCMI Type Vocabulary classes (dcmitype:Text, dcmitype:PhysicalObject and dcmitype:Collection). Scholarly agents (Cognizers) correspond to dct:Agent, linked to Wikidata when possible, with fallback strategies for entities without entries. Authenticity claims were modeled through directed relations between Cognizer and item spans, labeled according to SEBI's authenticity categories (Authentic, FormalForgery, ContentForgery, Forgery and Neutral) as shown in Figure 9.

Knowledge Base Integration. Integration with Wikidata through INCEpTION's Knowledge Base module ensures consistent entity identification, preventing inconsistent annotation that would hamper aggregation and reasoning in the final KG. Wikidata was selected for its domain coverage, data quality and compatibility with the entity types required by the COPs.

Annotation Process. Following established corpus linguistics and NLP practices, a single experienced annotator performed the annotation work iteratively, with guidelines refined based on encountered edge cases and thoroughly documented to ensure reproducibility.

RDF Mapping Validation. Preliminary mapping exercises validated that the annotation schema could produce target knowledge structures. Using the Pilot Corpus, annotation-to-RDF mapping was validated through the algorithm in Listing 4.2, serving as a unit test confirming that annotation patterns correctly transformed to valid RDF instantiating SEBI classes and properties.

Successful RDF generation from annotations validated the initial schema design, confirming that annotation patterns adequately captured the semantic content required to instantiate SEBI's authenticity claim structure. Following successful validation, development proceeded to the next pattern following the iterative cycle specified in Task II.

This subsection implements Task III of the ATR4CH methodology (Section 3.4), designing and implementing computational tools to automatically extract COPs from text using the annotation model developed in Task II as the target schema. Following the methodology's emphasis on incremental development, extraction capabilities were developed and validated for each COP using the same Pilot Corpus documents that were annotated in Task II, rather than building the complete extraction pipeline before testing.

Following the task decomposition and tool selection guidelines in Task III (Section 3.4), the pipeline was designed around the annotation model elements developed in Task II, prioritizing based on COP semantic importance and accounting for information manifestation patterns identified during corpus analysis in Task I. The KE task addresses CH corpora's domain-specific characteristics and limited annotated training data through a modular architecture enabling incremental extraction.

The pipeline integrates three complementary technologies, each addressing specific extraction requirements identified through the methodology:

GliNER for NER: Provides lightweight, generalist NER using custom entity types, enabling precise character-level span identification for entity extraction without requiring task-specific training data. This addresses the limited annotated training data characteristic of CH domains, as identified in the methodology's tool selection strategy.

LLMs for Structured Extraction: Handle complex information extraction through JSON schema-based responses (Schick et al., 2023; Qin et al., 2024) using ICL strategies (Brown et al., 2020; Min et al., 2022). We evaluated three models at varying parameter scales to understand performance trade-offs and cost-effectiveness, following the methodology's resource-adaptive approach:

1. Claude Sonnet 3.7 ^[19] as the largest model

2. Llama 3.3 70B (Dubey et al., 2024) as a medium-sized model

3. GPT-4o-mini ^[20] as the smallest model [estimated 8–14 billion active parameters (Ben Abacha et al., 2025)].

Rule-Based Entity Linking: Employs the Wikibase API ^[21] with domain-specific heuristics, integrating the knowledge base resources identified during annotation model development (Task II, Section 4.2). After evaluating various state-of-the-art solutions, this approach proved most effective for historical entities and CH concepts, providing reliable external knowledge base integration while handling the specialized vocabulary of authenticity assessment debates.

Paragraph-Level Processing Strategy: While the selected LLMs can process complete documents, the system automatically selects only relevant paragraphs whenever possible. This design serves three purposes: (1) reducing content volume per processing step to minimize potential opinion overlap between entities, which we hypothesize improves precision; (2) demonstrating scalability to documents of arbitrary length; and (3) maintaining computational efficiency and cost-effectiveness by minimizing token consumption per API call.

Following the modular approach specified in Task III (Section 3.4), the KE pipeline consists of six sequential components, each enriching the output before passing it to the next. Each component targets specific annotation schema elements from Task II and produces a JSON output following a predefined schema designed for conversion to RDF. Development proceeded through preliminary implementations and sequential testing over the COPs identified in Task I until the complete KG could be extracted.

Figure 10 presents the pipeline architecture, showing the correspondence between pipeline components and the COPs identified in Task I:

1. Metadata Extraction: Targets COP 1 (CH Item Metadata) – Raw text documents → alleged and settled item metadata

2. Opinion Holder Identification: Targets COP 2 (Scholarly Opinions) – Item metadata + text → entity mentions with opinion classifications

3. Entity Resolution: Integrates knowledge bases identified in Task II – Entity mentions → Wikidata-linked entity clusters

4. Opinion Extraction: Completes COP 2 extraction – Linked entities + paragraphs → structured authenticity opinions

5. Evidence Mining: Targets COP 3 (Evidential Features) – Opinions + contexts → feature evaluations with polarity

6. Hypothesis Extraction: Targets COP 4 (Alternative Hypotheses) – Evidence + full context → conflicting statements and alternative theories.

The following subsections detail each component's implementation, demonstrating how the annotation schemas developed in Task II guided extraction design.

COP Addressed: COP 1 (CH Item Metadata, Section 4.1.3)

Annotation Schema: CH Item Metadata Layer (Section 4.2)

Input: Raw Wikipedia articles in markup (.txt files)

Output: Cleaned articles; JSON with alleged item metadata.

This component implements extraction for the first COP, identifying and extracting metadata about CH items discussed in each article. The LLM is instructed to extract a JSON schema describing all items under discussion, specifically targeting the alleged metadata elements defined in the annotation model – what items claim to be – including purported authors, creation dates, locations, item types and subject matter. The task relies on ICL in a Few-Shot setting (with three examples), using CoT reasoning. Figure 11 shows an example input and JSON output.

COP Addressed: COP 2 (Scholarly Opinions, Section 4.1.3)

Annotation Schema: Entity and Opinion layers (Section 4.2)

Input: Cleaned article + item metadata (output of Component 1)

Output: JSON with is_cognizer classification, coreferences.

This component begins extraction of the second COP by identifying scholarly agents. Following the entity identification approach developed during annotation, it employs GliNER for NER, targeting people, organizations, groups and locations as specified in the annotation schema. GliNER identifies precise character-level spans, enabling exact identification and grouping of paragraphs in which each entity appears. The LLM then performs binary classification (is_expressing_opinion: True/False) to identify which entities function as Cognizers, alongside additional textual mentions and co-references. The task relies on ICL in a Few-Shot setting (with three examples), using CoT reasoning.

COP Addressed: Supporting infrastructure for all COPs

Annotation Schema: Knowledge Base integration (Section 4.2)

Input: Cognizers and coreferences (output of Component 2)

Output: JSON with relevant paragraphs grouped by Cognizer and biographical information.

This component implements the knowledge base integration strategy established during annotation model development (Task II, Section 4.2). It performs coreference resolution and Entity Linking through a three-stage pipeline. First, it collects all entity mentions across paragraphs and groups them by exact string match. Second, it applies coreference resolution by calculating mention similarity using a Jaccard coefficient over word sets (with common stop words removed), clustering mentions with similarity scores exceeding 0.7 when entity types are compatible. The system selects the longest string as the representative mention for each cluster. Third, it performs Entity Linking by querying the Wikidata Search API (wbsearchentities) with each mention variant, retrieving up to five candidates per query and deduplicating results by Wikidata identifier.

Each candidate undergoes scoring through the Wikidata API (wbgetentities) to retrieve claims and labels. The scoring function combines three weighted components: (1) name similarity between cluster mentions and candidate labels/aliases, calculated using Jaccard similarity over word sets with special handling for first name/initial matching (weight: 0.6 for labels, 0.3 for aliases); (2) entity type compatibility verified through Wikidata property P31 (instance of), with type mappings defined for persons (Q5), organizations (Q43229, Q7278), locations (Q2221906) and groups (Q16334295); and (3) for person entities, occupation relevance assessed through Wikidata property P106 (occupation), comparing retrieved occupation identifiers against a curated vocabulary of scholarly occupations (weight: 0.1). The system applies a minimum threshold of 0.4 for candidate acceptance and selects the highest-scoring candidate per cluster. When multiple candidates exceed the threshold, the system retrieves detailed data in batch requests and calculates an overlap score measuring the proportion of cluster mentions matching each candidate's labels and aliases (using both exact matches and fuzzy matching with a similarity threshold of 0.7), combining this with the initial score through weighted average (0.6 for overlap, 0.4 for initial score).

COP Addressed: COP 2 (Scholarly Opinions, Section 4.1.3)

Annotation Schema: Opinion classification layer (Section 4.2)

Input: Entity + Wikidata Information (if linked) + paragraphs where entity is mentioned

Output: JSON describing (1) the Cognizer's opinion, (2) their opinion type and (3) the metadata of the opinion.

This component completes extraction of the second COP by extracting and classifying authenticity opinions. If the entity has been successfully linked to Wikidata in Component 3, this information is provided to the model. The extraction process captures the main elements of the annotation schema: opinion targets (which documents or artifacts), opinion types following SEBI classifications (Authentic, Forgery, Formal forgery, Content forgery, Neutral), confidence levels expressed by the Cognizer, temporal contexts (when opinions were expressed) and geographic contexts where relevant.

COP Addressed: COP 3 (Evidential Features, Section 4.1.3)

Annotation Schema: Evidence and Features Layer (Section 4.4.3)

Input: Structured opinions (output of Component 4) + contextual paragraphs

Output: JSON with supporting evidences and evaluations for each opinion.

This component implements extraction for the third COP, enriching opinions with evidences and features being evaluated. Features are organized into three categories following the SEBI ontology and the annotation model developed in Task II: intrinsic features (content, language, style, orthography), extrinsic features (handwriting, ink, material support, physical characteristics) and provenance information (historical context, witness accounts, transmission history). For each feature, the system determines evaluation criteria, including consistency, presence, completeness, reliability and veridicality, as specified in the annotation schema. Each evaluation receives polarity assignment (positive, negative, neutral evidence) and links to supporting scholarly opinions.

COP Addressed: COP 4 (Alternative Hypotheses, Section 4.1.3)

Annotation Schema: Scholarly Hypotheses Layer (Section 4.4.3)

Input: Opinions + evidence evaluations + full document context

Output: JSON with hypotheses about document origins, intent, etc.

This final component implements extraction for the fourth COP, enriching the output with scholars' hypotheses. The hypotheses types correspond directly to the relation types defined in the annotation model: authorship hypotheses (who actually created items if not alleged authors?), dating hypotheses (when were items actually created if not alleged dates?), location hypotheses (where were items actually created if not alleged locations?) and motivation hypotheses (why were items created or forged?). The system handles cases where Cognizers accept alleged metadata as authentic. For consistency and to avoid negated categories, polarity (positive/negative) is included as a field.

Following the immediate validation strategy specified in Task III (Section 3.4), once extraction for each COP was implemented, the pipeline was tested on the Pilot Corpus and results were compared against manual annotations created in Task II. This immediate validation cycle enabled rapid identification of extraction bottlenecks or misalignments before proceeding to the next COP. This cycle continued until extraction pipelines had been developed and validated for all identified COPs using the Pilot Corpus.

This subsection implements Task IV of the ATR4CH methodology (Section 3.5), harmonizing the COPs (Task I), annotation schemas (Task II) and pipeline components (Task III) into a coherent, end-to-end KE system. Task IV represents a critical transition point: the modular components developed and validated on the Pilot Corpus are now integrated into a production-ready system, and the annotation model that emerged from processing individual COPs is consolidated into a production version suitable for full corpus processing and comprehensive GT creation.

Following the integration approach specified in Task IV (Section 3.5), the modular extraction components developed for individual COPs were integrated into a unified text-to-knowledge-graph pipeline. Integration addressed dependencies between patterns, ensured consistent entity resolution across components and optimized the overall processing architecture. The integrated pipeline processes documents from raw text to complete KGs that instantiate all identified COPs.

Following the comprehensive testing approach in Task IV (Section 3.5), testing over the pilot corpus processed documents from raw text to final KGs, revealing systematic issues including data sparseness patterns, interaction effects between COP extractors, inconsistent tool coverage across discourse types and representation generation errors. Testing systematically evaluated performance across document types and semantic phenomena, with particular attention to error propagation through pipeline stages.

Following the annotation model refinement approach in Task IV (Section 3.5), based on testing results, the annotation model evolved into a production-ready version suitable for both manual annotation and automated extraction. Following successful development and validation of extraction pipelines for individual COPs (Tasks II–III), the annotation schemas were consolidated into a production model suitable for comprehensive GT creation. This refined model captures CH item metadata, evidence and features, and scholarly hypotheses through additional layers developed iteratively during pipeline testing on the Pilot Corpus, maintaining backward compatibility with COPs while adding elements crucial for automated extraction.

CH Item Metadata Layer. This layer captures alleged metadata – descriptive information (creator, date, location) that the document or artifact purports about itself, before scholarly critical analysis. This includes face-value claims presented within the item or by whoever claimed to find the item regarding authorship, creation date, geographic origin and other identifying characteristics. Annotations include AllegedCreator, AllegedDate, AllegedLocation, ItemSubject and ItemType, plus properties for formal forgeries (ItemCreator, ItemDate, ItemLocation) (see Figure 12).

Evidence and Features Layer. This layer generates Evidence nodes connected to InterpretationAct Named Graphs. It employs four tagsets: Feature (SEBI vocabulary terms for intrinsic/extrinsic features and provenance), FeatureAssessment (evaluation perspectives: consistency, presence, completeness, reliability, veridicality), FeatureAssessmentPolarity (negative, neutral, positive) and FeatureAssessmentConfidence.

Consider Lorenzo Valla's assessment of the Donation's language features (Figure 15), which converts to three evidence structures linking textual features to evaluation criteria and polarities.

Listing 4.4.3 shows the evidence mapping algorithm.

Scholarly Hypotheses Layer. This layer captures alternative hypotheses through four relation types linking Cognizers to Wikidata entities: stm:CreatorHypothesis, stm:DatingHypothesis, stm:LocationHypothesis and stm:ReasonHypothesis (see Figures 13–15).

Listing 4.4.3 details the hypotheses mapping algorithm.

Following the mapping algorithm enhancement approach in Task IV (Section 3.5), preliminary mapping algorithms from Task II were consolidated and enhanced to handle the complete KG structure. As the output JSON model closely resembles the GT structure, the mapping uses similar logic, differing only in that it processes from JSON rather than the JSON UIMA CAS (Content Analysis System) format used by INCEpTION. Error handling mechanisms manage extraction failures and partial results using validation tools such as SHACL and OWL reasoners as specified in the methodology.

Each annotation layer maps to RDF following SEBI ontology principles, with Wikidata integration providing entity resolution. The INCEpTION project is available on GitHub alongside mapping scripts ^[22]. Statistics for annotation results are shown in Table 2.

This subsection implements Task V of the ATR4CH methodology (Section 3.6), employing technical validation and domain-expert evaluation to ensure knowledge structures accurately represent domain-specific discourse complexity. The refined system from Task IV is applied to test data separate from the Pilot Corpus used for development, representing the first application of the complete pipeline to unseen documents. The final iterations of the prompts are available in  Appendix.

Following the KE approach in Task V (Section 3.6), test datasets were processed through the complete pipeline under realistic deployment conditions, with systematic documentation of performance and failure modes. This represented the first application of the extraction pipeline beyond the Pilot Corpus used for development. The final output is mapped to RDF using the algorithms explained in previous sections. This subsection showcases produced KGs in RDF-star format (specifically, this example was generated by the pipeline using Llama 3.3 70B).

Figure 16 shows the general structure of a generated KG from the GraphDB interface (Ontotext, 2024). Each CH item is represented with both alleged metadata (what the item claims to be) and scholarly assessments, as shown in Listing 4.5.1. The Donation of Constantine exemplifies this pattern:

Listing 4.5.1 shows Lorenzo Valla's interpretation of the Donation.

The supporting evidence for Valla's conclusions is captured through the Evidence graph, shown in Listing 4.5.1.

Following the multi-level evaluation approach specified in Task V (Section 3.6), the evaluation framework provides a multi-dimensional assessment of the KG generation pipeline. Multiple complementary approaches address KG evaluation challenges, integrating human assessment throughout the evaluation pipeline and using F₁ score and G-EVAL metrics. The framework systematically addresses five EQs, based on the CQs that defined the original SEBI ontology (Pasqual, 2025).

• EQ1: CH Item Metadata Extraction Precision. How accurately does the pipeline extract alleged item metadata compared to expert annotations?

Methodology: This is formulated as a multiclass classification task, evaluating the metadata extraction component against the GT. The classification scheme follows standard evaluation practices: True Positive (TP) for exact matches between model output and GT, False Positive (FP) for incorrect model predictions, True Negative (TN) for correctly identified absence of metadata when GT is also empty and False Negative (FN) for missing outputs when GT contains valid metadata. To accommodate acceptable semantic variations (e.g., alternative titles, location aliases), all FP cases are manually reviewed to identify outputs that are semantically equivalent to the GT and should be reclassified as TP.

Metrics: Micro-averaged results for individual metadata categories (Title, Creator, Date, Location) and macro-averaged overall performance using standard precision, recall and F₁-score calculations.

• EQ2: Scholarly Entity Recognition Coverage. How effectively does the entity recognition and opinion frame module identify scholarly agents (Cognizers) present in the source documents?

Methodology: The entity extraction component is evaluated by conducting frequency-based analysis comparing GT entities with model-identified entities.

Metrics: Entity-level recall (proportion of GT entities correctly identified) and the total number of entities detected by the model to assess both coverage and potential over-generation.

• EQ3: Evidential Reasoning Extraction Quality. How accurately does the model capture the multi-dimensional evidential reasoning employed by scholars in their interpretations?

Methodology: Given the complex structure of scholarly evidence identified in the ontological framework, where each piece of evidence comprises multiple semantic dimensions (evaluated feature, evaluation perspective, broader feature class, polarity), a custom scoring metric operating on a 4-point scale is implemented. For each evidence prediction, points are assigned based on accuracy across these four dimensions, subtracting one point for each incorrectly identified component. This approach accommodates cases where model outputs are semantically similar but not lexically identical to GT annotations.

Score Interpretation: 0 points indicates complete extraction failure (equivalent to FN or total FP); 1–2 points indicates weak but partially acceptable outputs; and 3–4 points indicates acceptable to strong outputs meeting semantic requirements.

Scope: Evidence evaluation is restricted to entities successfully matched between model output and GT from EQ2.

• EQ4: Hypothesis and Judgment Identification. How accurately does the model extract scholars' interpretative hypotheses and overall authenticity judgments?

Methodology: The same precision, recall and F₁-score evaluation framework established for EQ1 is applied to assess the hypothesis extraction component. Model outputs are compared against expert-annotated GT for both specific scholarly hypotheses and overall authenticity determinations.

Scope: Evaluation is limited to the subset of successfully matched entities identified in EQ2 to ensure fair comparison.

• EQ5: Overall Discourse Representation Fidelity. Does the complete generated KG provide an adequate representation of the scholarly debate surrounding the CH items' authenticity?

Methodology: To evaluate representation fidelity, G-EVAL (Liu et al., 2023) is employed. Since the KGs only represent opinions inside the text, comparing the source document with a rehydrated version of the KG would heavily bias the evaluation metric. This led to avoiding similarity-based metrics like BLEU, ROUGE and COMET with the source corpus as used in Gangemi et al. (2024).

G-EVAL evaluates two metrics: debate correctness and debate representativeness. The first evaluates how well individual scholarly entities and their arguments are represented compared to the GT, penalizing omission of specific entities while rewarding accurate representation of facts, claims and evidence with proper domain-specific terminology. The second assesses how comprehensively the overall structure and flow of the authenticity debate is captured, including the breadth of scholarly perspectives and their relationships within the discourse narrative.

Previous evaluation metrics mostly covered matchable entries between GT and output, whereas G-EVAL evaluates the complete output.

Scope: G-EVAL over rehydrated KGs covers the complete pipeline output against the rehydrated GT.

This section presents a comprehensive evaluation and preliminary discussion of findings across the five EQs outlined in Section 4.5. We evaluate Claude Sonnet 3.7, Llama 3.3 70B and GPT-4o-mini across multiple dimensions of the authenticity debate extraction task (the tables will show only Claude, GPT and Llama for brevity). We begin with simple exploratory SPARQL queries across the 3 KGs and compare the results with the GT, as shown in Figure 17.

Table 3 and Figure 18 provide an overview of the KGs generated by each model compared to the GT. The models produce more triples than the GT (10,000–12,000 vs 4,026), as the GT relies heavily on Wikidata entity linking, while the models extract and create explicit triples for information found directly in the text (such as dates, locations and descriptive metadata). Despite this, the models generate comparable numbers of Interpretation Acts and Cognizers to the GT, suggesting at this stage a similar density of extracted information.

How accurately does the pipeline extract alleged CH item metadata compared to expert annotations?

As shown in Table 4, the performance is high across all models, with F₁-scores ranging from 0.97 to 0.99.

Claude Sonnet 3.7 achieves the highest overall performance with an F₁-score of 0.987. All models show nearly perfect recall, indicating successful extraction of all relevant metadata elements, with precision differences primarily reflecting varying FP rates. Date extraction shows more variability, with Llama 3.3 achieving the lowest precision (0.867) due to higher FP rates, as it misclassified the forging date with the alleged dating. For this particular task, the challenge was to distinguish between alleged metadata and settled metadata. All models successfully understood the task, showing only small precision drops at varying parameter sizes.

How effectively does the entity recognition and opinion frame module identify scholarly agents (Cognizers) present in the source documents? As shown in Table 3, the number of Cognizers is relatively similar across models – Table 5 shows the number of overlapping entities between the model's KG and the GT.

GPT-4o-mini demonstrates superior entity recognition coverage, identifying 77.3% of scholarly agents present in the GT, significantly outperforming Claude (49.5%) and Llama 3.3 (58.8%). It identified the most entities that were expressing opinions. The perfect match rates indicate the proportion of identified entities that exactly match GT annotations. GPT-4o-mini maintains the highest accuracy at 66.0%.

How accurately does the model capture the multi-dimensional evidential reasoning employed by scholars in their interpretations?

Table 6 presents evidence extraction performance using our custom 4-point scoring system that evaluates the accuracy of feature identification, evaluation perspective, feature classification and polarity assessment.

All models demonstrate strong evidence extraction capabilities, with mean accuracies above 0.95%. While GPT-4o-mini achieves the highest precision and recall for entities, as shown in 5, Claude shows the highest evidence coverage (0.968) in Table 6. This pattern highlights that the lower recall in identifying Cognizers by Claude returns in higher precision in downstream tasks.

How accurately does the model extract scholars' interpretative hypotheses and overall authenticity judgments?

Table 7 presents performance on extracting scholarly hypotheses about items’ origins and authenticity judgments.

GPT-4o-mini achieves the highest overall F₁-score (0.749) for hypothesis extraction, with particularly strong performance in authenticity type classification (0.845). However, the model shows weaker performance in creator hypothesis identification (0.484), suggesting challenges in extracting attribution hypotheses. Claude demonstrates exceptional performance in geographic hypotheses (0.923 F₁) and temporal hypotheses (0.791 F₁), indicating strength in extracting location and dating alternative theories. Llama 3.3 shows the most balanced performance across hypothesis types, with particularly strong creator hypothesis extraction (0.712 F₁). The variation across hypothesis types reflects the inherent complexity of scholarly reasoning, with location and date hypotheses generally more explicitly stated than creator attributions or underlying motivations.

Does the complete generated KG provide an adequate representation of the scholarly debate surrounding CH Item authenticity?

The empirical threshold, using the scores produced by G-EVAL on three well-represented articles revised manually (Posthumous Diary, Centiloquium and Acámbaro figures), is set at 0.6–0.7. This result is consistent with other evaluation findings: while the other two models demonstrate higher debate coverage overall, they are penalized for generating more FPs, resulting in lower scores. Table 8 shows the per-statement G-EVAL scores for the three models, while Table 9 for the whole KG produced by a single Wikipedia article. This evaluation confirms a key pattern in our pipeline – when an entity is correctly identified as a Cognizer, its associated arguments are accurately represented. However, incorrect entity identification leads to error propagation throughout the pipeline, causing the generation of FPs in downstream components. Future iterations of the pipeline should incorporate self-consistency checks at the entity identification stage to reduce error accumulation and improve overall accuracy.

In this section, we discuss the overall performance patterns, identified bottlenecks and potential steps to enhance the KE while answering our RQs (Section 1), followed by our contributions, limitations and future steps.

To answer RQ1, our five-step ATR4CH methodology proves effective in developing the pipeline within the boundaries of an ontology. The granular evaluation demonstrates that our divide-and-conquer methodology enables systematic refinement of individual components while maintaining system coherence. The result is that different models excel at different subtasks, suggesting potential for hybrid approaches that leverage each model's strengths.

The alignment between G-EVAL and other evaluations suggests that self-consistency checks throughout the pipeline (such as prompting models to evaluate their own extraction results) could reduce FPs and FNs without reducing the necessity of external validation.

To answer RQ2, our evaluation reveals component-specific performance patterns across all tested models. Performance varies across extraction tasks, with all models achieving high scores on metadata extraction (F₁-scores of 0.965–0.991), moderate performance for entity recognition (F₁: 0.709–0.803), strong evidence extraction capabilities (95.3–96.8% accuracy) and more challenging hypothesis extraction (F₁-scores of 0.655–0.749). Extracting alleged metadata proves straightforward across models, while capturing nuanced scholarly hypotheses requires more sophisticated interpretation regardless of architecture. The evidence extraction results demonstrate that contemporary LLMs can effectively capture multi-dimensional evidential reasoning, but they can do so only when they can identify the Cognizer – this represents an error propagation problem we identified in the pipeline, as the out-of-GT outputs for evidence extraction are mostly empty or incorrect.

To answer RQ3, the generated KGs demonstrate adequate representation of scholarly debate complexity and nuance. While the representation model proves more than adequate, as already demonstrated in the BROAST catalogue (Pasqual, 2025), the quality of the automatically generated KGs can still be improved.

G-EVAL scores around 0.6 indicate acceptable discourse representation quality with room for improvement. The successful capture of multi-dimensional evidential reasoning (95.3–96.8% accuracy) shows that LLMs can handle complex semantic relationships, suggesting broader applicability to other humanities domains characterized by multi-perspectival interpretation and evidence-based reasoning. However, the model perspective on specific domain terminology and approaches requires improvement, as the G-EVAL evaluation demonstrates.

To answer RQ4, our findings challenge the conventional assumption that larger models always perform better for complex domain tasks. Claude 3.7 Sonnet demonstrates lower recall but higher precision, being more conservative in entity classification but achieving greater accuracy in subsequent extraction steps. GPT-4o-mini shows the opposite pattern with higher recall and competitive precision, while Llama 3.3 70B falls between these approaches. Notably, as seen in Table 7, GPT-4o-mini performs better since it managed to correctly identify more Cognizers covered in the GT than other models, while having the least parameters of the lot.

The precision–recall trade-off has significant implications for deployment strategies. In production environments where KGs undergo human review and correction, higher recall models may be preferable since updating or deleting erroneous triples is more efficient than creating new KGs from scratch. Conversely, in real-time applications such as RAG systems, where extraction occurs without human supervision, higher precision becomes critical to avoid propagating false information.

To answer RQ5, the performance differences between models are relatively modest, while model sizes and costs differ substantially ^[23]. This suggests that the step-by-step pipeline architecture effectively leverages the capabilities of smaller models, making deployment feasible and more cost-effective for CH institutions with varying computational budgets.

The competitive performance of different model sizes within sequential pipelines opens two promising research directions. First, fine-tuning approaches could specifically target bottlenecks like Cognizer classification of recognized entities. Second, enhanced pre-processing using specialized tools could filter irrelevant entities before they enter the extraction pipeline.

The methodology's adaptability accommodates diverse institutional landscapes: smaller projects can benefit from intensive human-in-the-loop approaches with API-based models, while larger projects can leverage automated scaling through extensive annotation datasets and local deployment.

In this work, we demonstrated the practical application of the SEBI ontology using RDF-star to represent multi-perspective authenticity claims, enabling structured representation of evidence-based scholarly interpretation while preserving provenance and alternative hypotheses. Second, we introduced a comprehensive five-step methodology for building LLM-centric KE pipelines that addresses the unique challenges of humanities texts through systematic integration of annotation models, ontological frameworks and computational tools. The methodology's technology-agnostic design provides a replicable blueprint adaptable to varying project scales and resource constraints. Third, our technical implementation achieved practical feasibility through a sequential LLM pipeline that successfully captures scholarly reasoning, including evidential features, evaluation polarities and alternative hypotheses.

Our approach faces some limitations that can be addressed in future work. The current focus on English Wikipedia sources limits multilingual applicability, particularly important given the glocal nature of CH scholarship. Performance on primary scholarly literature remains untested, and two key bottlenecks emerged: Cognizer classification difficulty and dependency on Wikidata linking for optimal performance.

Future work will prioritize developing multilingual extraction capabilities, implementing targeted improvements for Cognizer identification through fine-tuning or hybrid approaches and creating user-friendly tools that enable CH practitioners to customize the extraction process with appropriate human-in-the-loop interfaces. Additionally, working not only with secondary literature but also with primary works from scholars would be a relevant possible contribution. While works that try to summarize While LLMs show promise for structuring complex scholarly debates, complete automation remains premature, suggesting that balanced human-machine collaboration represents the most viable path forward.

This research employed LLMs (Claude Sonnet 3.7, Llama 3.3 70B and GPT-4o-mini) as research subjects for KE experiments, as detailed in the methodological sections of this article. Additionally, G-EVAL, an LLM-based evaluation framework, was used for assessing discourse representation quality. Claude Sonnet 4.5 was used as a writing aid for summarization, translation and code generation when necessary. No AI tools were used in the ideation, analysis or interpretation of this manuscript beyond the specified cases above. The authors maintain full responsibility for the research design, methodology, data interpretation, figure and listing creation, and all conclusions presented.

This research was partially funded by the European Union – Next Generation EU, investment I.4.1 PNRR Patrimonio Culturale, Decreto Ministeriale n. 351, 9 April 2022. We gratefully acknowledge Dr. Cristina Solidoro (Department of Classical Philology and Italian Studies, University of Bologna, Bologna, Italy) for her bibliographic suggestions and additional domain expertise in support of the case study.