pocketflow/docs/llm.md

layout	title	nav_order
default	LLM Integration	3

Call LLM

For your LLM application, implement a function to call LLMs yourself. You can ask an assistant like ChatGPT or Claude to generate an example. For instance, asking ChatGPT to "implement a call_llm function that takes a prompt and returns the LLM response" gives:

def call_llm(prompt):
    from openai import OpenAI
    # Set the OpenAI API key (use environment variables, etc.)
    client = OpenAI(api_key="YOUR_API_KEY_HERE")
    r = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}]
    )
    return r.choices[0].message.content

# Example usage
call_llm("How are you?")

Improvements

You can enhance the function as needed. Examples:

1. Handle chat history:

def call_llm(messages):
    from openai import OpenAI
    client = OpenAI(api_key="YOUR_API_KEY_HERE")
    r = client.chat.completions.create(
        model="gpt-4",
        messages=messages
    )
    return r.choices[0].message.content

2. Add in-memory caching:

from functools import lru_cache

@lru_cache(maxsize=1000)
def call_llm(prompt):
    # Your implementation here
    pass

3. Enable logging:

def call_llm(prompt):
    import logging
    logging.info(f"Prompt: {prompt}")
    response = ...
    logging.info(f"Response: {response}")
    return response

You can also try libraries like litellm

Why not provide an LLM call function?

I believe it is a bad practice to provide LLM-specific implementations in a general framework:

• LLM APIs change frequently. Hardcoding them makes maintenance difficult.
• You may need flexibility to switch vendors, use fine-tuned models, or deploy local LLMs.
• Custom optimizations like prompt caching, request batching, or response streaming may be required.