Flex-Kartierung — Features¶

Flex-Kartierung covers the full path from a URL to a published profile (Steckbrief): configuration of the capture, crawling, multi-stage extraction, manual post-processing, entity unification, translation, and generation of a static website. The following sections describe the application from the user's perspective.

Use cases¶

Capture research projects of a university. A list of project web pages is registered in the application; the system extracts project name, institution, funding information, duration, team, keywords, and descriptions and provides the results as uniform profiles.
Inventory the AI services of an institution. From the web pages of an institution's AI offerings, the system extracts service name, target audience, access requirements, availability, license, and contact information and consolidates them into a searchable overview.
Collect guidelines and handouts. Existing guidelines on AI use in studies and teaching are captured via their web pages; from title, target audience, and contents, profiles are produced that are comparable without switching to the original page.
Map cooperative services across institutions. Cross-site offerings are captured with participating institutions, governance, and access paths; through entity normalization, universities appear with a unified spelling.
Improve data quality iteratively. Low-confidence results are reworked in a review queue; prompts can be adjusted and individual fields or entire sources can be re-extracted without changing other results.
Publish a public bilingual overview. From the reviewed profiles, a static website in German and English is produced with category pages, detail pages, full-text search, and a JSON API for external consumers.

At a glance¶

Configuration of new content types via YAML files (category, field groups, prompts, profile template).
Web crawler with multi-page mode (up to five linked subpages), robots.txt awareness, and domain-specific rate limit.
Two-stage extraction (extract + validate) with a confidence score, quality class, and justification per field.
Review queue for low-rated results, an inline editor, and targeted re-extraction of individual sources or fields.
LLM-supported entity management for universities and locations with variants, auto-linking, and manual confirmation.
Field-wise translation of validated results from German to English with rules to protect proper names, URLs, and technical abbreviations.
Export as Markdown profiles and as a static website (HTML + search + sitemap + API), separated by language.

Configuration and profile definition¶

The capture is described entirely through YAML category files. A category specifies a content type and consists of an internal name, a display name, a list of field groups, a selection of which groups appear in the profile, and a Markdown template with placeholders. Each field has a prompt with extraction instructions, format and fallback specifications, optionally a validate instruction, and a flag indicating whether the field is translatable. Pre-configured are the categories projekt (research projects), ki_service (AI services), handreichung (guidelines), and kooperativer_dienst (cooperative services). Custom categories are added as additional YAML files and synchronized on reload.

Connectors¶

Flex-Kartierung integrates four source and backend systems. External connectors are described in detail; internal connectors are only outlined, as they represent standard infrastructure.

Web crawler (external, HTTP/HTTPS). Reads public web pages and converts the HTML to Markdown. The crawler supports a multi-page mode in which a configurable number of linked subpages of the same domain is processed alongside a main page, as well as a list of manually added supplementary URLs. A per-domain rate limit applies; robots.txt rules are evaluated and can be disabled if necessary. SSL verification is optional, allowing capture of institutions with self-signed certificates.
LLM backend (external from the application's perspective, internal to the operating environment). The system addresses an OpenAI-compatible chat-completions interface and is designed to operate against a locally hosted model. Address, model name, temperature, token limit, requests per minute, and minimum interval between calls are configurable.
PostgreSQL (internal). Persistent storage of all master data, sources, Markdown content, extractions, profiles, and translations.
Redis (internal). Job queue for the pipeline steps and cache for internal intermediate state.

Import and export formats¶

Import: URLs (individually or with a list of additional subpages) via the operating interface and via the admin API; YAML files for categories and prompts; scripts to pre-populate university master data.
Profile export: Markdown profiles per source, optionally as HTML; generated against a category-specific Markdown template.
Website export: fully static HTML site with category pages, detail pages, search page, imprint, sitemap, and schema.org markup; one URL structure per language (German / English).
Machine interfaces: public API with category, profile, and statistics endpoints, as well as a JSON search index for client-side full-text search; OpenAPI documentation of the admin and public APIs.
Configuration: export of all category definitions as a ZIP for versioning outside the application.
Operational metrics: Prometheus metrics at the /metrics endpoint and structured JSON logs.

Quality assurance¶

The quality-assurance layer is modeled as a dedicated pipeline stage rather than as a check at the end. Every statement about a field carries an evaluation, an origin annotation, and possibly a manual intervention.

Validate phase. For each prompt, after the extract phase the system performs a second LLM request that evaluates the raw result and converts it into a quality class, a score, and a justification. The validated result may differ from the raw extraction.
Required confidence per prompt. Each field defines its own minimum confidence; values below are flagged as needing review.
Review queue with inline editor. Low-rated fields are consolidated in a dedicated view, with the source Markdown as context and the option to correct the value directly. Manual corrections are flagged separately.
Entity normalization queue. Extractions with entity references are matched against the existing entity inventory including variants. Above an auto-link threshold the link is created automatically, in a middle range a review task is generated, and below it the link is discarded.
Prompt dependencies. Fields with dependencies are only executed once their source fields are present; the system resolves the order automatically in waves and detects cyclical dependencies.
Re-extract / re-validate / re-generate. Individual sources, fields, or profiles can be re-processed without affecting other data; manual corrections are preserved or selectively overwritten.
Full separation of processing stages. Per extraction, the raw result, validated result, manual correction, and translation are available as separate fields; the crawled Markdown is retained for traceability.
Robustness against backend failures. A circuit breaker protects against repeated LLM errors; transient failures are absorbed by a retry manager with exponential backoff and jitter.

Multilingualism¶

The translation of German profiles into English is performed field by field as a separate pipeline step. Per field, the category definition determines whether it is to be translated; proper names, URLs, and technical abbreviations are left untouched according to an explicit rule. English display names for fields and categories as well as a bilingual UI vocabulary are maintained via i18n files; the static website is generated per language and connected via a language switch.

Operating interface¶

The admin interface is server-side rendered and uses htmx and Alpine.js for interactive components, without a separate frontend build step. It includes a dashboard with status and quality metrics, management views for categories, prompts, sources, extractions, entities, and profiles, as well as triggers for site generation, translation, and maintenance tasks. Low-rated extractions are editable directly in the list; the status of running jobs is updated via polling.