pocketflow/docs/guide.md

layout	title	parent	nav_order
default	Design Guidance	Apps	1

LLM System Design Guidance

Example LLM Project File Structure

my_project/
├── main.py
├── flow.py
├── utils/
│   ├── __init__.py
│   ├── call_llm.py
│   └── search_web.py
├── tests/
│   ├── __init__.py
│   ├── test_flow.py
│   └── test_nodes.py
├── requirements.txt
└── docs/
    └── design.md

docs/

Store the documentation of the project.

It should include a design.md file, which describes

• Project requirements
• Required utility functions
• High-level flow with a mermaid diagram
• Shared memory data structure
• For each node, discuss
  • Node purpose and design (e.g., should it be a batch or async node?)
  • How the data shall be read (for prep) and written (for post)
  • How the data shall be processed (for exec)

utils/

Houses functions for external API calls (e.g., LLMs, web searches, etc.).

It’s recommended to dedicate one Python file per API call, with names like call_llm.py or search_web.py. Each file should include:

• The function to call the API
• A main function to run that API call

For instance, here’s a simplified call_llm.py example:

from openai import OpenAI

def call_llm(prompt):
    client = OpenAI(api_key="YOUR_API_KEY_HERE")
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content

def main():
    prompt = "Hello, how are you?"
    print(call_llm(prompt))

if __name__ == "__main__":
    main()

main.py

Serves as the project’s entry point.

flow.py

Implements the application’s flow, starting with node followed by the flow structure.

tests/

Optionally contains all tests. Use pytest for testing flows, nodes, and utility functions. For example, test_call_llm.py might look like:

from utils.call_llm import call_llm

def test_call_llm():
    prompt = "Hello, how are you?"
    assert call_llm(prompt) is not None

System Design Steps

1. Project Requirements

   • Identify the project's core entities.
   • Define each functional requirement and map out how these entities interact step by step.

2. Utility Functions

   • Determine the low-level utility functions you’ll need (e.g., for LLM calls, web searches, file handling).
   • Implement these functions and write basic tests to confirm they work correctly.

3. Flow Design

   • Develop a high-level process flow that meets the project’s requirements.
   • Specify which utility functions are used at each step.
   • Identify possible decision points for Node Actions and data-intensive operations for Batch tasks.
   • Illustrate the flow with a Mermaid diagram.

4. Shared Memory Structure

   • Decide how to store and update state, whether in memory (for smaller applications) or a database (for larger or persistent needs).
   • Define data schemas or models that detail how information is stored, accessed, and updated.

5. Implementation

   • Start coding with a simple, direct approach (avoid over-engineering at first).
   • For each node in your flow:
     • prep: Determine how data is accessed or retrieved.
     • exec: Outline the actual processing or logic needed.
     • post: Handle any final updates or data persistence tasks.

6. Optimization

   • Prompt Engineering: Use clear and specific instructions with illustrative examples to reduce ambiguity.
   • Task Decomposition: Break large, complex tasks into manageable, logical steps.

7. Reliability

   • Structured Output: Verify outputs conform to the required format. Consider increasing max_retries if needed.
   • Test Cases: Develop clear, reproducible tests for each part of the flow.
   • Self-Evaluation: Introduce an additional Node (powered by LLMs) to review outputs when the results are uncertain.