Skip to content

Sequence of tool development and LLM techniques learned or used

1. Doc (First experiment)

Basic prompt engineering techniques: - Capitalization in prompts to reinforce critical instructions - Two-step chunking strategy (chunk → meta summary) - Structural element extraction with tags (headings, tables, captions) - pymupdf4llm for LLM-friendly Markdown

2. Code Analyzer

Specialization and hierarchical analysis: - Multi-perspective analysis: 4 separate specialized LLM analyses per file (business logic, technical aspects, interfaces, issue detection) - Hierarchical analysis approach across multiple abstraction levels (file→package→module→system) - Dialog-based requirement clarification: LLM actively asks questions - Asynchronous processing with semaphore-based rate limiting for many LLM calls

3. AI survey

Initial agent workflows and structuring: - Single agent with workflow orchestration: Evaluation → Inquiry → Structuring - Structured JSON outputs as a basis for communication - Clarity score (0-1) for automatic evaluation - Pydantic for data validation in LLM outputs - Safeguards against infinite loops in JSON-driven systems

4. stt-helper

Cascading workflows: - Three-stage processing cascade: Cleaning → Revision → Formatting - Focused single-purpose prompts (one task per stage) - Development interface for prompt optimization:

  • Interactive prompt adjustment during processing
  • Insight into intermediate results of each stage
  • Character-based chunking for longer texts

5. ppt-helper

Multi-agent architectures: - Two-agent architecture with clear roles (chat agent + artifact agent) - Structured JSON communication protocol between agents - Bidirectional “return channel”: Artifact agent can ask questions to chat agent (not just unidirectional) - Model selection based on prompt-following capabilities rather than just benchmarks - <1000 lines per file for LLM maintainability

6. Translate

Scaling and context management: - Multi-stage process: Markdown conversion → chunking → parallel translation → composition - Context management system: Glossaries, translated key terms, chunk summaries are passed on to LLM - Asynchronous parallel API calls for performance - Uniform Markdown pipeline instead of format-specific processing - Consistency across chunk boundaries through context transfer

7. TalkToDocuments

Extensive context utilization: - Utilization of large contexts without traditional chunking or vector databases - Tiktoken integration for token counting and visualization - Simple reference system [P1], [P2] instead of complex metadata - Intelligent content cleaning and deduplication before LLM processing - Direct retention of multiple documents in context (up to 20 documents)

8. TextTool

State management and systematic documentation: - Implementation_Status document: LLM maintains implementation status itself after each step - State-less development: Each new interaction possible with complete context - Artifact-centered approach (input/output areas instead of dialog prompts) - Curated tool library with 12 optimized prompts without meta comments - Combination of prompt engineering + heuristic filtering: Prompt following reinforced by downstream filtering - Differentiated prompt engineering strategy: Structured aspects for good prompt following, weak continuous text instructions - Automatic title generation through separate LLM calls for history function

9. Web Helper

Rapid prototyping and two-stage analysis: - Two-stage LLM processing: Content structure analysis → Suggestions for improvement → Application to content - JSON handling of large structured data (12 MB) with 256K token context - In-memory session processing (no persistent state)

10. Chart Tool

Complex multi-agent architecture with pattern libraries: - Intent→Plan→Execute workflow (three-stage evolution) - IntentService for intent classification (modification, single chart, multiple charts, analysis)

  • PlanService translates intents into detailed execution plans
  • ExecutionService coordinates with retry logic

  • Pattern libraries as a hybrid approach: 44 code patterns (23 implementation, 7 anti-patterns, 9 modification, 5 semantic)

  • LLM selects and adapts patterns based on data types

  • Hybrid: LLM intelligence + templates
  • fix_code() method for LLM-based self-correction with error feedback
  • Retry logic: In case of errors, error message to LLM for code correction
  • Controlled code execution with restricted built-ins, predefined safe globals
  • Semantic analysis: SemanticColorHelper for natural language color specifications
  • Multi-service orchestration: Multiple specialized services work together

11. Personnel cost calculator

Integrated hybrid architecture with state-based dialog guidance: - Strict hybrid architecture: LLM exclusively for parameter extraction, deterministic calculations completely LLM-free (Python Decimal for exact arithmetic) - LLM as an intelligent interface: Bridge between natural language and structured processing - State machine for dialog guidance: 7 defined states (INITIAL→PARSING→CLARIFYING→CALCULATING→COMPLETE, plus FALLBACK/ERROR) - Three-stage fallback mechanism: After 3 parse errors, automatic manual form with pre-filled values - JSON interface with Pydantic: Clear contract definition between LLM and backend - Structured requirements gathering: Special prompt guides department through systematic questions before development - Specification-first approach: 50 min. specification enabled 40 min. implementation (5700 lines in one round)

Text Style Editor

Control-based text transformation with a two-step process: - Two-step process: Neutralisation (10 dimensions) → Stylisation (34 controls in 7 categories) - Three slider types: Polar sliders (-10 to +10), intensity sliders (0-10) and step sliders (discrete options) - Intensity levels: Precise control from ‘light’ (1-2) to ‘extreme’ (9-10) - 23 presets: Predefined slider combinations for typical use cases - Hash code export/import: Persistence of settings across sessions - Flat architecture: 6 Python modules (app.py, config.py, llm_client.py, models.py, prompt_builder.py, token_counter.py) - Prompt builder: Context-specific LLM prompt generation based on controller settings - Technical stack: Python, Gradio 6, OpenAI-compatible API (vLLM), tiktoken - Methodological approach: Detailed specification (1,400 lines) prior to implementation to reduce iteration loops

Development line of LLM techniques:

Phase 1 - Basics (Doc, Code Analyzer): - Prompt engineering basics (capitalization, double reinforcement) - Chunking strategies - Multi-perspective analysis - Dialogue-based interaction

Phase 2 - Workflows (AI survey, stt-helper): - Structured JSON outputs with Pydantic - Cascading focused prompts - Workflow orchestration by LLM - Development interfaces for prompt iteration

Phase 3 - Multi-agents (ppt-helper): - Two-agent architectures - Bidirectional structured communication - Prompt following as a selection criterion

Phase 4 - Scaling (Translate, TalkToDocuments): - Context management systems - Large context usage (256K tokens) - Parallel API calls - Token tracking

Phase 5 - Systematization (TextTool, Web-Helper): - LLM maintains its own metadata (Implementation_Status) - Artifact-centered approaches - Two-stage analysis workflows

Phase 6 - Complex systems (Chart-Tool): - Intent→Plan→Execute Pattern - Pattern libraries as a hybrid approach - Self-correction mechanisms - Multi-service orchestration - Semantic analysis components

Phase 7 - Integrated architectures (personnel cost calculator): - Strict separation of LLM/deterministic logic as an architectural principle - State machines for robust dialog control - Fallback mechanisms for LLM uncertainty - Specification-first as a development methodology - Synthesis of earlier techniques (JSON/Pydantic, workflows, dialog control) into a robust overall system