Sequence of tool development and LLM techniques learned or used#

1. Doc (First experiment)#

Basic prompt engineering techniques:

  • Capitalization in prompts to reinforce critical instructions
  • Two-step chunking strategy (chunk → meta summary)
  • Structural element extraction with tags (headings, tables, captions)
  • pymupdf4llm for LLM-friendly Markdown

2. Code Analyzer#

Specialization and hierarchical analysis:

  • Multi-perspective analysis: 4 separate specialized LLM analyses per file (business logic, technical aspects, interfaces, issue detection)
  • Hierarchical analysis approach across multiple abstraction levels (file→package→module→system)
  • Dialog-based requirement clarification: LLM actively asks questions
  • Asynchronous processing with semaphore-based rate limiting for many LLM calls

3. AI survey#

Initial agent workflows and structuring:

  • Single agent with workflow orchestration: Evaluation → Inquiry → Structuring
  • Structured JSON outputs as a basis for communication
  • Clarity score (0-1) for automatic evaluation
  • Pydantic for data validation in LLM outputs
  • Safeguards against infinite loops in JSON-driven systems

4. stt-helper#

Cascading workflows:

  • Three-stage processing cascade: Cleaning → Revision → Formatting

  • Focused single-purpose prompts (one task per stage)

  • Development interface for prompt optimization:

    • Interactive prompt adjustment during processing

  • Insight into intermediate results of each stage

  • Character-based chunking for longer texts

5. ppt-helper#

Multi-agent architectures:

  • Two-agent architecture with clear roles (chat agent + artifact agent)
  • Structured JSON communication protocol between agents
  • Bidirectional “return channel”: Artifact agent can ask questions to chat agent (not just unidirectional)
  • Model selection based on prompt-following capabilities rather than just benchmarks
  • <1000 lines per file for LLM maintainability

6. Translate#

Scaling and context management:

  • Multi-stage process: Markdown conversion → chunking → parallel translation → composition
  • Context management system: Glossaries, translated key terms, chunk summaries are passed on to LLM
  • Asynchronous parallel API calls for performance
  • Uniform Markdown pipeline instead of format-specific processing
  • Consistency across chunk boundaries through context transfer

7. TalkToDocuments#

Extensive context utilization:

  • Utilization of large contexts without traditional chunking or vector databases
  • Tiktoken integration for token counting and visualization
  • Simple reference system [P1], [P2] instead of complex metadata
  • Intelligent content cleaning and deduplication before LLM processing
  • Direct retention of multiple documents in context (up to 20 documents)

8. TextTool#

State management and systematic documentation:

  • Implementation_Status document: LLM maintains implementation status itself after each step
  • State-less development: Each new interaction possible with complete context
  • Artifact-centered approach (input/output areas instead of dialog prompts)
  • Curated tool library with 12 optimized prompts without meta comments
  • Combination of prompt engineering + heuristic filtering: Prompt following reinforced by downstream filtering
  • Differentiated prompt engineering strategy: Structured aspects for good prompt following, weak continuous text instructions
  • Automatic title generation through separate LLM calls for history function

9. Web Helper#

Rapid prototyping and two-stage analysis:

  • Two-stage LLM processing: Content structure analysis → Suggestions for improvement → Application to content
  • JSON handling of large structured data (12 MB) with 256K token context
  • In-memory session processing (no persistent state)

10. Chart Tool#

Complex multi-agent architecture with pattern libraries:

  • Intent→Plan→Execute workflow (three-stage evolution)

  • IntentService for intent classification (modification, single chart, multiple charts, analysis)

  • PlanService translates intents into detailed execution plans

    • ExecutionService coordinates with retry logic

  • Pattern libraries as a hybrid approach: 44 code patterns (23 implementation, 7 anti-patterns, 9 modification, 5 semantic)

  • LLM selects and adapts patterns based on data types

    • Hybrid: LLM intelligence + templates

  • fix_code() method for LLM-based self-correction with error feedback

  • Retry logic: In case of errors, error message to LLM for code correction

  • Controlled code execution with restricted built-ins, predefined safe globals

  • Semantic analysis: SemanticColorHelper for natural language color specifications

  • Multi-service orchestration: Multiple specialized services work together

11. Personnel cost calculator#

Integrated hybrid architecture with state-based dialog guidance:

  • Strict hybrid architecture: LLM exclusively for parameter extraction, deterministic calculations completely LLM-free (Python Decimal for exact arithmetic)
  • LLM as an intelligent interface: Bridge between natural language and structured processing
  • State machine for dialog guidance: 7 defined states (INITIAL→PARSING→CLARIFYING→CALCULATING→COMPLETE, plus FALLBACK/ERROR)
  • Three-stage fallback mechanism: After 3 parse errors, automatic manual form with pre-filled values
  • JSON interface with Pydantic: Clear contract definition between LLM and backend
  • Structured requirements gathering: Special prompt guides department through systematic questions before development
  • Specification-first approach: 50 min. specification enabled 40 min. implementation (5700 lines in one round)

Text Style Editor#

Control-based text transformation with a two-step process:

  • Two-step process: Neutralisation (10 dimensions) → Stylisation (34 controls in 7 categories)
  • Three slider types: Polar sliders (-10 to +10), intensity sliders (0-10) and step sliders (discrete options)
  • Intensity levels: Precise control from ‘light’ (1-2) to ‘extreme’ (9-10)
  • 23 presets: Predefined slider combinations for typical use cases
  • Hash code export/import: Persistence of settings across sessions
  • Flat architecture: 6 Python modules (app.py, config.py, llm_client.py, models.py, prompt_builder.py, token_counter.py)
  • Prompt builder: Context-specific LLM prompt generation based on controller settings
  • Technical stack: Python, Gradio 6, OpenAI-compatible API (vLLM), tiktoken
  • Methodological approach: Detailed specification (1,400 lines) prior to implementation to reduce iteration loops

Development line of LLM techniques:#

Phase 1 - Basics (Doc, Code Analyzer):

  • Prompt engineering basics (capitalization, double reinforcement)
  • Chunking strategies
  • Multi-perspective analysis
  • Dialogue-based interaction

Phase 2 - Workflows (AI survey, stt-helper):

  • Structured JSON outputs with Pydantic
  • Cascading focused prompts
  • Workflow orchestration by LLM
  • Development interfaces for prompt iteration

Phase 3 - Multi-agents (ppt-helper):

  • Two-agent architectures
  • Bidirectional structured communication
  • Prompt following as a selection criterion

Phase 4 - Scaling (Translate, TalkToDocuments):

  • Context management systems
  • Large context usage (256K tokens)
  • Parallel API calls
  • Token tracking

Phase 5 - Systematization (TextTool, Web-Helper):

  • LLM maintains its own metadata (Implementation_Status)
  • Artifact-centered approaches
  • Two-stage analysis workflows

Phase 6 - Complex systems (Chart-Tool):

  • Intent→Plan→Execute Pattern
  • Pattern libraries as a hybrid approach
  • Self-correction mechanisms
  • Multi-service orchestration
  • Semantic analysis components

Phase 7 - Integrated architectures (personnel cost calculator):

  • Strict separation of LLM/deterministic logic as an architectural principle
  • State machines for robust dialog control
  • Fallback mechanisms for LLM uncertainty
  • Specification-first as a development methodology
  • Synthesis of earlier techniques (JSON/Pydantic, workflows, dialog control) into a robust overall system
Backward Results Introduction Forward