add map reduce tutorial

2025-03-22 12:44:33 -04:00
parent 7411a9127b
commit eb1c721e00
15 changed files with 420 additions and 52 deletions
@@ -0,0 +1,78 @@
+# Resume Qualification - Map Reduce Example
+
+A PocketFlow example that demonstrates how to implement a Map-Reduce pattern for processing and evaluating resumes.
+
+## Features
+
+- Read and process multiple resume files using a Map-Reduce pattern
+- Evaluate each resume individually using an LLM with structured YAML output
+- Determine if candidates qualify for technical roles based on specific criteria
+- Aggregate results to generate qualification statistics and summaries
+
+## Getting Started
+
+1. Install the required dependencies:
+
+```bash
+pip install -r requirements.txt
+```
+
+2. Set your OpenAI API key as an environment variable:
+
+```bash
+export OPENAI_API_KEY=your_api_key_here
+```
+
+3. Run the application:
+
+```bash
+python main.py
+```
+
+## How It Works
+
+The workflow follows a classic Map-Reduce pattern with three sequential nodes:
+
+```mermaid
+flowchart LR
+    ReadResumes[Map: Read Resumese] --> EvaluateResumes[Batch: Evaluate Resumes]
+    EvaluateResumes --> ReduceResults[Reduce: Aggregate Results]
+```
+
+Here's what each node does:
+
+1. **ReadResumesNode (Map Phase)**: Reads all resume files from the data directory and stores them in the shared data store
+2. **EvaluateResumesNode (Batch Processing)**: Processes each resume individually using an LLM to determine if candidates qualify
+3. **ReduceResultsNode (Reduce Phase)**: Aggregates evaluation results and produces a summary of qualified candidates
+
+## Files
+
+- [`main.py`](./main.py): Main entry point for running the resume qualification workflow
+- [`flow.py`](./flow.py): Defines the flow that connects the nodes
+- [`nodes.py`](./nodes.py): Contains the node classes for each step in the workflow
+- [`utils.py`](./utils.py): Utility functions including the LLM wrapper
+- [`requirements.txt`](./requirements.txt): Lists the required dependencies
+- [`data/`](./data/): Directory containing sample resume files for evaluation
+
+## Example Output
+
+```
+Starting resume qualification processing...
+
+===== Resume Qualification Summary =====
+Total candidates evaluated: 5
+Qualified candidates: 2 (40.0%)
+
+Qualified candidates:
+- Emily Johnson
+- John Smith
+
+Detailed evaluation results:
+✗ Michael Williams (resume3.txt)
+✓ Emily Johnson (resume2.txt)
+✗ Lisa Chen (resume4.txt)
+✗ Robert Taylor (resume5.txt)
+✓ John Smith (resume1.txt)
+
+Resume processing complete!
+```
@@ -0,0 +1,25 @@
+John Smith
+Software Engineer
+
+Education:
+- Master of Computer Science, Stanford University, 2018
+- Bachelor of Computer Science, MIT, 2016
+
+Experience:
+- Senior Software Engineer, Google, 2019-present
+  * Led the development of cloud infrastructure projects
+  * Implemented scalable solutions using Kubernetes and Docker
+  * Reduced system latency by 40% through optimization
+
+- Software Developer, Microsoft, 2016-2019
+  * Worked on Azure cloud services
+  * Built RESTful APIs for enterprise solutions
+
+Skills:
+- Programming: Python, Java, C++, JavaScript
+- Technologies: Docker, Kubernetes, AWS, Azure
+- Tools: Git, Jenkins, Jira
+
+Projects:
+- Developed a recommendation engine that increased user engagement by 25%
+- Created a sentiment analysis tool using NLP techniques 
@@ -0,0 +1,25 @@
+Emily Johnson
+Data Scientist
+
+Education:
+- Ph.D. in Statistics, UC Berkeley, 2020
+- Master of Science in Mathematics, UCLA, 2016
+
+Experience:
+- Data Scientist, Netflix, 2020-present
+  * Developed machine learning models for content recommendation
+  * Implemented A/B testing frameworks to optimize user experience
+  * Collaborated with product teams to define metrics and KPIs
+
+- Data Analyst, Amazon, 2016-2020
+  * Analyzed user behavior patterns to improve conversion rates
+  * Created dashboards and visualizations for executive decision-making
+
+Skills:
+- Programming: R, Python, SQL
+- Machine Learning: TensorFlow, PyTorch, scikit-learn
+- Data Visualization: Tableau, PowerBI, matplotlib
+
+Publications:
+- "Advances in Recommendation Systems" - Journal of Machine Learning, 2021
+- "Statistical Methods for Big Data" - Conference on Data Science, 2019 
@@ -0,0 +1,25 @@
+Michael Williams
+Marketing Manager
+
+Education:
+- MBA, Harvard Business School, 2015
+- Bachelor of Arts in Communications, NYU, 2010
+
+Experience:
+- Marketing Director, Apple, 2018-present
+  * Managed a team of 15 marketing professionals
+  * Developed and executed global marketing campaigns
+  * Increased brand awareness by 30% through digital initiatives
+
+- Marketing Manager, Coca-Cola, 2015-2018
+  * Led product launches across North America
+  * Coordinated with external agencies on advertising campaigns
+
+Skills:
+- Digital Marketing: SEO, SEM, Social Media Marketing
+- Analytics: Google Analytics, Adobe Analytics
+- Tools: HubSpot, Salesforce, Marketo
+
+Achievements:
+- Marketing Excellence Award, 2020
+- Led campaign that won Cannes Lions Award, 2019 
@@ -0,0 +1,28 @@
+Lisa Chen
+Frontend Developer
+
+Education:
+- Bachelor of Fine Arts, Rhode Island School of Design, 2019
+
+Experience:
+- UI/UX Designer, Airbnb, 2020-present
+  * Designed user interfaces for mobile and web applications
+  * Created wireframes and prototypes for new features
+  * Conducted user research and usability testing
+
+- Junior Designer, Freelance, 2019-2020
+  * Worked with small businesses on branding and website design
+  * Developed responsive web designs using HTML, CSS, and JavaScript
+
+Skills:
+- Design: Figma, Sketch, Adobe XD
+- Development: HTML, CSS, JavaScript, React
+- Tools: Git, Zeplin
+
+Portfolio Highlights:
+- Redesigned checkout flow resulting in 15% conversion increase
+- Created custom icon set for mobile application
+- Designed responsive email templates
+
+Certifications:
+- UI/UX Design Certificate, Coursera, 2019 
@@ -0,0 +1,28 @@
+Robert Taylor
+Sales Representative
+
+Education:
+- Bachelor of Business Administration, University of Texas, 2017
+
+Experience:
+- Account Executive, Salesforce, 2019-present
+  * Exceeded sales targets by 25% for three consecutive quarters
+  * Managed a portfolio of 50+ enterprise clients
+  * Developed and implemented strategic account plans
+
+- Sales Associate, Oracle, 2017-2019
+  * Generated new business opportunities through cold calling
+  * Assisted senior sales representatives with client presentations
+
+Skills:
+- CRM Systems: Salesforce, HubSpot
+- Communication: Negotiation, Public Speaking
+- Tools: Microsoft Office Suite, Google Workspace
+
+Achievements:
+- Top Sales Representative Award, Q2 2020
+- President's Club, 2021
+
+Interests:
+- Volunteer sales coach for local small businesses
+- Member of Toastmasters International 
@@ -0,0 +1,15 @@
+from pocketflow import Flow
+from nodes import ReadResumesNode, EvaluateResumesNode, ReduceResultsNode
+
+def create_resume_processing_flow():
+    """Create a map-reduce flow for processing resumes."""
+    # Create nodes
+    read_resumes_node = ReadResumesNode()
+    evaluate_resumes_node = EvaluateResumesNode()
+    reduce_results_node = ReduceResultsNode()
+    
+    # Connect nodes
+    read_resumes_node >> evaluate_resumes_node >> reduce_results_node
+    
+    # Create flow
+    return Flow(start=read_resumes_node)
@@ -0,0 +1,25 @@
+from flow import create_resume_processing_flow
+
+def main():
+    # Initialize shared store
+    shared = {}
+    
+    # Create the resume processing flow
+    resume_flow = create_resume_processing_flow()
+    
+    # Run the flow
+    print("Starting resume qualification processing...")
+    resume_flow.run(shared)
+    
+    # Display final summary information (additional to what's already printed in ReduceResultsNode)
+    if "summary" in shared:
+        print("\nDetailed evaluation results:")
+        for filename, evaluation in shared.get("evaluations", {}).items():
+            qualified = "✓" if evaluation.get("qualifies", False) else "✗"
+            name = evaluation.get("candidate_name", "Unknown")
+            print(f"{qualified} {name} ({filename})")
+    
+    print("\nResume processing complete!")
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,106 @@
+from pocketflow import Node, BatchNode
+from utils import call_llm
+import yaml
+import os
+
+class ReadResumesNode(Node):
+    """Map phase: Read all resumes from the data directory into shared storage."""
+    
+    def exec(self, _):
+        resume_files = {}
+        data_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "data")
+        
+        for filename in os.listdir(data_dir):
+            if filename.endswith(".txt"):
+                file_path = os.path.join(data_dir, filename)
+                with open(file_path, 'r', encoding='utf-8') as file:
+                    resume_files[filename] = file.read()
+        
+        return resume_files
+    
+    def post(self, shared, prep_res, exec_res):
+        shared["resumes"] = exec_res
+        return "default"
+
+
+class EvaluateResumesNode(BatchNode):
+    """Batch processing: Evaluate each resume to determine if the candidate qualifies."""
+    
+    def prep(self, shared):
+        return list(shared["resumes"].items())
+    
+    def exec(self, resume_item):
+        """Evaluate a single resume."""
+        filename, content = resume_item
+        
+        prompt = f"""
+Evaluate the following resume and determine if the candidate qualifies for an advanced technical role.
+Criteria for qualification:
+- At least a bachelor's degree in a relevant field
+- At least 3 years of relevant work experience
+- Strong technical skills relevant to the position
+
+Resume:
+{content}
+
+Return your evaluation in YAML format:
+```yaml
+candidate_name: [Name of the candidate]
+qualifies: [true/false]
+reasons:
+  - [First reason for qualification/disqualification]
+  - [Second reason, if applicable]
+```
+"""
+        response = call_llm(prompt)
+        
+        # Extract YAML content
+        yaml_content = response.split("```yaml")[1].split("```")[0].strip() if "```yaml" in response else response
+        result = yaml.safe_load(yaml_content)
+
+        
+        return (filename, result)
+
+    def post(self, shared, prep_res, exec_res_list):
+        shared["evaluations"] = {filename: result for filename, result in exec_res_list}
+        return "default"
+
+
+class ReduceResultsNode(Node):
+    """Reduce node: Count and print out how many candidates qualify."""
+    
+    def prep(self, shared):
+        return shared["evaluations"]
+    
+    def exec(self, evaluations):
+        qualified_count = 0
+        total_count = len(evaluations)
+        qualified_candidates = []
+        
+        for filename, evaluation in evaluations.items():
+            if evaluation.get("qualifies", False):
+                qualified_count += 1
+                qualified_candidates.append(evaluation.get("candidate_name", "Unknown"))
+        
+        summary = {
+            "total_candidates": total_count,
+            "qualified_count": qualified_count,
+            "qualified_percentage": round(qualified_count / total_count * 100, 1) if total_count > 0 else 0,
+            "qualified_names": qualified_candidates
+        }
+        
+        return summary
+    
+    def post(self, shared, prep_res, exec_res):
+        shared["summary"] = exec_res
+        
+        print("\n===== Resume Qualification Summary =====")
+        print(f"Total candidates evaluated: {exec_res['total_candidates']}")
+        print(f"Qualified candidates: {exec_res['qualified_count']} ({exec_res['qualified_percentage']}%)")
+        
+        if exec_res['qualified_names']:
+            print("\nQualified candidates:")
+            for name in exec_res['qualified_names']:
+                print(f"- {name}")
+        
+        return "default" 
@@ -0,0 +1,3 @@
+pocketflow>=0.0.1
+openai>=1.0.0
+pyyaml>=6.0 
@@ -0,0 +1,14 @@
+import os
+from openai import OpenAI
+
+def call_llm(prompt):    
+    client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "your-api-key"))
+    r = client.chat.completions.create(
+        model="gpt-4o",
+        messages=[{"role": "user", "content": prompt}]
+    )
+    return r.choices[0].message.content
+
+# Example usage
+if __name__ == "__main__":
+    print(call_llm("Tell me a short joke"))