228 lines
11 KiB
Plaintext
228 lines
11 KiB
Plaintext
---
|
||
description: Guidelines for using PocketFlow, Agentic Coding
|
||
globs: **/*.py
|
||
alwaysApply: true
|
||
---
|
||
# Agentic Coding: Humans Design, Agents code!
|
||
|
||
> If you are an AI agents involved in building LLM Systems, read this guide **VERY, VERY** carefully! This is the most important chapter in the entire document. Throughout development, you should always (1) start with a small and simple solution, (2) design at a high level (`docs/design.md`) before implementation, and (3) frequently ask humans for feedback and clarification.
|
||
{: .warning }
|
||
|
||
## Agentic Coding Steps
|
||
|
||
Agentic Coding should be a collaboration between Human System Design and Agent Implementation:
|
||
|
||
| Steps | Human | AI | Comment |
|
||
|:-----------------------|:----------:|:---------:|:------------------------------------------------------------------------|
|
||
| 1. Requirements | ★★★ High | ★☆☆ Low | Humans understand the requirements and context. |
|
||
| 2. Flow | ★★☆ Medium | ★★☆ Medium | Humans specify the high-level design, and the AI fills in the details. |
|
||
| 3. Utilities | ★★☆ Medium | ★★☆ Medium | Humans provide available external APIs and integrations, and the AI helps with implementation. |
|
||
| 4. Node | ★☆☆ Low | ★★★ High | The AI helps design the node types and data handling based on the flow. |
|
||
| 5. Implementation | ★☆☆ Low | ★★★ High | The AI implements the flow based on the design. |
|
||
| 6. Optimization | ★★☆ Medium | ★★☆ Medium | Humans evaluate the results, and the AI helps optimize. |
|
||
| 7. Reliability | ★☆☆ Low | ★★★ High | The AI writes test cases and addresses corner cases. |
|
||
|
||
1. **Requirements**: Clarify the requirements for your project, and evaluate whether an AI system is a good fit.
|
||
- Understand AI systems' strengths and limitations:
|
||
- **Good for**: Routine tasks requiring common sense (filling forms, replying to emails)
|
||
- **Good for**: Creative tasks with well-defined inputs (building slides, writing SQL)
|
||
- **Not good for**: Ambiguous problems requiring complex decision-making (business strategy, startup planning)
|
||
- **Keep It User-Centric:** Explain the "problem" from the user's perspective rather than just listing features.
|
||
- **Balance complexity vs. impact**: Aim to deliver the highest value features with minimal complexity early.
|
||
|
||
2. **Flow Design**: Outline at a high level, describe how your AI system orchestrates nodes.
|
||
- Identify applicable design patterns (e.g., [Map Reduce], [Agent], [RAG]).
|
||
- For each node in the flow, start with a high-level one-line description of what it does.
|
||
- If using **Map Reduce**, specify how to map (what to split) and how to reduce (how to combine).
|
||
- If using **Agent**, specify what are the inputs (context) and what are the possible actions.
|
||
- If using **RAG**, specify what to embed, noting that there's usually both offline (indexing) and online (retrieval) workflows.
|
||
- Outline the flow and draw it in a mermaid diagram. For example:
|
||
```mermaid
|
||
flowchart LR
|
||
start[Start] --> batch[Batch]
|
||
batch --> check[Check]
|
||
check -->|OK| process
|
||
check -->|Error| fix[Fix]
|
||
fix --> check
|
||
|
||
subgraph process[Process]
|
||
step1[Step 1] --> step2[Step 2]
|
||
end
|
||
|
||
process --> endNode[End]
|
||
```
|
||
- > **If Humans can't specify the flow, AI Agents can't automate it!** Before building an LLM system, thoroughly understand the problem and potential solution by manually solving example inputs to develop intuition.
|
||
{: .best-practice }
|
||
|
||
3. **Utilities**: Based on the Flow Design, identify and implement necessary utility functions.
|
||
- Think of your AI system as the brain. It needs a body—these *external utility functions*—to interact with the real world:
|
||
|
||
|
||
- Reading inputs (e.g., retrieving Slack messages, reading emails)
|
||
- Writing outputs (e.g., generating reports, sending emails)
|
||
- Using external tools (e.g., calling LLMs, searching the web)
|
||
- **NOTE**: *LLM-based tasks* (e.g., summarizing text, analyzing sentiment) are **NOT** utility functions; rather, they are *core functions* internal in the AI system.
|
||
- For each utility function, implement it and write a simple test.
|
||
- Document their input/output, as well as why they are necessary. For example:
|
||
- `name`: `get_embedding` (`utils/get_embedding.py`)
|
||
- `input`: `str`
|
||
- `output`: a vector of 3072 floats
|
||
- `necessity`: Used by the second node to embed text
|
||
- Example utility implementation:
|
||
```python
|
||
# utils/call_llm.py
|
||
from openai import OpenAI
|
||
|
||
def call_llm(prompt):
|
||
client = OpenAI(api_key="YOUR_API_KEY_HERE")
|
||
r = client.chat.completions.create(
|
||
model="gpt-4o",
|
||
messages=[{"role": "user", "content": prompt}]
|
||
)
|
||
return r.choices[0].message.content
|
||
|
||
if __name__ == "__main__":
|
||
prompt = "What is the meaning of life?"
|
||
print(call_llm(prompt))
|
||
```
|
||
- > **Sometimes, design Utilities before Flow:** For example, for an LLM project to automate a legacy system, the bottleneck will likely be the available interface to that system. Start by designing the hardest utilities for interfacing, and then build the flow around them.
|
||
{: .best-practice }
|
||
|
||
4. **Node Design**: Plan how each node will read and write data, and use utility functions.
|
||
- One core design principle for PocketFlow is to use a [shared store], so start with a shared store design:
|
||
- For simple systems, use an in-memory dictionary.
|
||
- For more complex systems or when persistence is required, use a database.
|
||
- **Don't Repeat Yourself**: Use in-memory references or foreign keys.
|
||
- Example shared store design:
|
||
```python
|
||
shared = {
|
||
"user": {
|
||
"id": "user123",
|
||
"context": { # Another nested dict
|
||
"weather": {"temp": 72, "condition": "sunny"},
|
||
"location": "San Francisco"
|
||
}
|
||
},
|
||
"results": {} # Empty dict to store outputs
|
||
}
|
||
```
|
||
- For each [Node], describe its type, how it reads and writes data, and which utility function it uses. Keep it specific but high-level without codes. For example:
|
||
- `type`: Regular (or Batch, or Async)
|
||
- `prep`: Read "text" from the shared store
|
||
- `exec`: Call the embedding utility function
|
||
- `post`: Write "embedding" to the shared store
|
||
|
||
5. **Implementation**: Implement the initial nodes and flows based on the design.
|
||
- 🎉 If you've reached this step, humans have finished the design. Now *Agentic Coding* begins!
|
||
- **"Keep it simple, stupid!"** Avoid complex features and full-scale type checking.
|
||
- **FAIL FAST**! Avoid `try` logic so you can quickly identify any weak points in the system.
|
||
- Add logging throughout the code to facilitate debugging.
|
||
|
||
7. **Optimization**:
|
||
- **Use Intuition**: For a quick initial evaluation, human intuition is often a good start.
|
||
- **Redesign Flow (Back to Step 3)**: Consider breaking down tasks further, introducing agentic decisions, or better managing input contexts.
|
||
- If your flow design is already solid, move on to micro-optimizations:
|
||
- **Prompt Engineering**: Use clear, specific instructions with examples to reduce ambiguity.
|
||
- **In-Context Learning**: Provide robust examples for tasks that are difficult to specify with instructions alone.
|
||
|
||
- > **You'll likely iterate a lot!** Expect to repeat Steps 3–6 hundreds of times.
|
||
>
|
||
>
|
||
{: .best-practice }
|
||
|
||
8. **Reliability**
|
||
- **Node Retries**: Add checks in the node `exec` to ensure outputs meet requirements, and consider increasing `max_retries` and `wait` times.
|
||
- **Logging and Visualization**: Maintain logs of all attempts and visualize node results for easier debugging.
|
||
- **Self-Evaluation**: Add a separate node (powered by an LLM) to review outputs when results are uncertain.
|
||
|
||
## Example LLM Project File Structure
|
||
|
||
```
|
||
my_project/
|
||
├── main.py
|
||
├── nodes.py
|
||
├── flow.py
|
||
├── utils/
|
||
│ ├── __init__.py
|
||
│ ├── call_llm.py
|
||
│ └── search_web.py
|
||
├── requirements.txt
|
||
└── docs/
|
||
└── design.md
|
||
```
|
||
|
||
- **`docs/design.md`**: Contains project documentation for each step above. This should be *high-level* and *no-code*.
|
||
- **`utils/`**: Contains all utility functions.
|
||
- It's recommended to dedicate one Python file to each API call, for example `call_llm.py` or `search_web.py`.
|
||
- Each file should also include a `main()` function to try that API call
|
||
- **`nodes.py`**: Contains all the node definitions.
|
||
```python
|
||
# nodes.py
|
||
from pocketflow import Node
|
||
from utils.call_llm import call_llm
|
||
|
||
class GetQuestionNode(Node):
|
||
def exec(self, _):
|
||
# Get question directly from user input
|
||
user_question = input("Enter your question: ")
|
||
return user_question
|
||
|
||
def post(self, shared, prep_res, exec_res):
|
||
# Store the user's question
|
||
shared["question"] = exec_res
|
||
return "default" # Go to the next node
|
||
|
||
class AnswerNode(Node):
|
||
def prep(self, shared):
|
||
# Read question from shared
|
||
return shared["question"]
|
||
|
||
def exec(self, question):
|
||
# Call LLM to get the answer
|
||
return call_llm(question)
|
||
|
||
def post(self, shared, prep_res, exec_res):
|
||
# Store the answer in shared
|
||
shared["answer"] = exec_res
|
||
```
|
||
- **`flow.py`**: Implements functions that create flows by importing node definitions and connecting them.
|
||
```python
|
||
# flow.py
|
||
from pocketflow import Flow
|
||
from nodes import GetQuestionNode, AnswerNode
|
||
|
||
def create_qa_flow():
|
||
"""Create and return a question-answering flow."""
|
||
# Create nodes
|
||
get_question_node = GetQuestionNode()
|
||
answer_node = AnswerNode()
|
||
|
||
# Connect nodes in sequence
|
||
get_question_node >> answer_node
|
||
|
||
# Create flow starting with input node
|
||
return Flow(start=get_question_node)
|
||
```
|
||
- **`main.py`**: Serves as the project's entry point.
|
||
```python
|
||
# main.py
|
||
from flow import create_qa_flow
|
||
|
||
# Example main function
|
||
# Please replace this with your own main function
|
||
def main():
|
||
shared = {
|
||
"question": None, # Will be populated by GetQuestionNode from user input
|
||
"answer": None # Will be populated by AnswerNode
|
||
}
|
||
|
||
# Create the flow and run it
|
||
qa_flow = create_qa_flow()
|
||
qa_flow.run(shared)
|
||
print(f"Question: {shared['question']}")
|
||
print(f"Answer: {shared['answer']}")
|
||
|
||
if __name__ == "__main__":
|
||
main()
|
||
```
|