CFP Scout

CFP Scout is an AI-powered agent system that scrapes tech events with open CFPs, filters them using a local language model via Ollama based on user-defined interests, and sends daily email notifications with beautiful formatting. Built with Anthropic's Model Context Protocol (MCP) for modular agent architecture.

🎯 Perfect for: Developers, conference speakers, and tech professionals who want automated CFP discovery without manual searching.

✨ Key Features

• 🤖 AI-Powered Filtering – Ollama local LLM analyzes CFP relevance to your interests
• 📧 Beautiful Email Reports – HTML-formatted emails with relevance scores and CFP details
• 🖥️ External Terminal Display – Launches separate terminal windows for email previews
• ⚙️ Multi-Mode Execution – Traditional, Hybrid, and full MCP agent modes
• ⏰ Production Scheduling – Automated daily execution with systemd and cron support
• 🔗 MCP Integration – Full Anthropic Model Context Protocol implementation
• 📮 Mailgun Support – Professional email delivery without personal passwords

---

🚀 Quick Start (2 minutes)

# 1. Clone and setup
git clone <repository-url>
cd cfp-scout
cp config/env.example .env

# 2. Install dependencies
pip install -r requirements.txt

# 3. Setup Ollama
ollama serve
ollama pull llama3:latest

# 4. Configure environment (edit .env with your settings)
# 5. Run CFP Scout
python cfp_scout.py --run-once

Result: 🎉 CFP email generated and terminal preview launched!

---

Tech Stack

• Language: Python 3.12
• Web Scraping: Selenium with Chrome WebDriver
• Local LLM: Ollama (llama3:latest)
• Agent Protocol: Anthropic's Model Context Protocol (MCP)
• Email: Mailgun API and SMTP with HTML formatting
• Scheduling: Python schedule, systemd, cron
• Data Storage: JSON files for event pipeline stages

---

Configuration

Environment Variables

Edit .env with your settings:

# Ollama Configuration
OLLAMA_HOST=http://localhost:11434
OLLAMA_MODEL=llama3:latest

# User Interests & Filtering Criteria
EVENT_THEMES=AI,machine learning,artificial intelligence,engineering leadership,fintech,developer experience,software architecture,cloud computing,DevOps,data science
EVENT_LOCATIONS=Europe,UK
EVENT_TYPE=Online,Physical

# Email Configuration (choose ONE method)

# Method 1: Mailgun (Recommended - no personal passwords)
MAILGUN_API_KEY=your-mailgun-api-key-here
MAILGUN_DOMAIN=your-domain.mailgun.org
MAILGUN_FROM_EMAIL=CFP Scout <noreply@your-domain.mailgun.org>
TO_EMAIL=recipient@example.com

# Method 2: SMTP (Gmail, Outlook, etc.)
# EMAIL_ADDRESS=your_email@gmail.com
# EMAIL_PASSWORD=your_app_password_here
# SMTP_SERVER=smtp.gmail.com
# SMTP_PORT=587
# TO_EMAIL=recipient@example.com

# Schedule Configuration
SCHEDULE_TIME=08:00
TIMEZONE=Europe/London
EXECUTION_MODE=hybrid

# Display Configuration
SHOW_EXTERNAL_TERMINAL=true

Email Setup

Mailgun (Recommended)

1. Sign up at mailgun.com (free tier: 1,000 emails/month)
2. Get your API key and domain from Mailgun dashboard
3. Configure in .env file
4. Test: python -m cfp_scout.core.email_sender

Gmail/SMTP

1. Enable 2-Factor Authentication on Gmail
2. Create App Password in Google Account Settings
3. Use App Password (not regular password) in .env
4. Test: python -m cfp_scout.core.email_sender

---

Usage

Command Line Interface

# Execution modes
python cfp_scout.py --mode traditional  # Fast, basic
python cfp_scout.py --mode hybrid       # Balanced (default)
python cfp_scout.py --mode mcp          # Full agent mesh

# Operation modes
python cfp_scout.py --run-once          # Single execution
python cfp_scout.py --schedule          # Daily scheduling
python cfp_scout.py --status            # Show status

# Examples
python cfp_scout.py --run-once --mode hybrid
python cfp_scout.py --schedule --mode traditional

# Background execution (survives terminal closing)
nohup python cfp_scout.py --schedule > logs/cfp_scout.log 2>&1 &

Expected Results

✅ Pipeline execution completed successfully
📊 Mode: hybrid, Duration: 15.23s
📈 Events processed: 27
📋 Total executions: 1
📧 Email sent successfully via Mailgun

Performance Metrics:

• Events Found: ~25-30 CFP events per run
• Execution Time: 15-20 seconds
• Memory Usage: <200MB
• Success Rate: 99%+ (depends on website availability)

---

Production Deployment

Native Python Scheduling (Recommended)

# Test the system
python cfp_scout.py --run-once

# Start scheduled execution
python cfp_scout.py --schedule

# Run in background
nohup python cfp_scout.py --schedule > logs/cfp_scout.log 2>&1 &

# Check status
python cfp_scout.py --status
tail -f logs/cfp_scout.log

Systemd Service (Linux Servers)

# Install service
sudo cp scripts/cfp-scout.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable cfp-scout

# Start service
sudo systemctl start cfp-scout
sudo systemctl status cfp-scout

# Monitor logs
sudo journalctl -u cfp-scout -f

Manual Cron (Alternative)

# Add to crontab for daily execution at 8 AM
0 8 * * * cd /path/to/cfp-scout && python3 cfp_scout.py --run-once >> logs/cron.log 2>&1

---

Testing & Debugging

Quick Test Commands

# 1. Quick system test
python cfp_scout.py --run-once

# 2. Test specific modes
python cfp_scout.py --run-once --mode traditional
python cfp_scout.py --run-once --mode hybrid
python cfp_scout.py --run-once --mode mcp

# 3. Test individual components
python -m cfp_scout.core.cfp_filter_agent

python -m cfp_scout.core.scraper

# 4. Run comprehensive test suite
python -m pytest tests/

Performance Benchmarks

# Benchmark different modes
time python cfp_scout.py --run-once --mode traditional  # ~15-17 seconds
time python cfp_scout.py --run-once --mode hybrid       # ~16-18 seconds  
time python cfp_scout.py --run-once --mode mcp          # ~20-25 seconds

Common Issues & Solutions

1. Ollama Not Running

# Error: Connection refused to localhost:11434
# Solution:
ollama serve
ollama pull llama3:latest
curl http://localhost:11434/api/version  # Test connection

2. Chrome/Selenium Issues

# Error: Chrome binary not found
# Solution: Install Chrome browser (ChromeDriver auto-managed)
google-chrome --version

3. Email Configuration Issues

# Test email setup
python -m cfp_scout.core.email_sender

# Mailgun: Check API key and domain in .env
# Gmail: Use App Password, not regular password

4. External Terminal Issues (macOS)

# Test terminal launching
osascript -e 'tell application "Terminal" to do script "echo test"'

System Health Checks

# Check system status
python cfp_scout.py --status

# Verify dependencies
pip freeze | grep -E "(selenium|ollama|requests|schedule)"

# Check Ollama
ollama list
ollama ps

# Test environment loading
python -c "from dotenv import load_dotenv; load_dotenv(); import os; print('✅ Config loaded')"

---

Architecture

CFP Scout uses Production-Ready MCP Architecture with scheduling and monitoring:

┌─────────────────────────────────────────────────────┐
│              Main Orchestrator                      │
│     (Scheduling, Monitoring, Multi-Mode)           │
└─────────────────────┬───────────────────────────────┘
                      │ Execution Control
          ┌───────────┼───────────────────┐
          │           │                   │
          ▼           ▼                   ▼
   Traditional    Hybrid Mode        Pure MCP Mode
     Pipeline    (MCP + Legacy)     (Full Agent Mesh)
          │           │                   │
          └───────────┼───────────────────┘
                      │
         ┌────────────┴────────────┐
         │    CFP Scout MCP Host   │
         │  (Agent Coordination)   │
         └─────────────┬───────────┘
                       │ MCP JSON-RPC 2.0
           ┌───────────┼───────────┐
           │           │           │
           ▼           ▼           ▼
   ┌─────────────┐ ┌───────────┐ ┌─────────────┐ ┌─────────────┐
   │   Scraper   │ │Event Orch.│ │ CFP Filter  │ │Email Sender │
   │ MCP Server  │ │MCP Server │ │ MCP Server  │ │ MCP Server  │
   └─────────────┘ └───────────┘ └─────────────┘ └─────────────┘

Execution Modes

1. Traditional → Direct module imports (fastest, basic)
2. Hybrid → MCP coordination + traditional execution (balanced, recommended)
3. Pure MCP → Complete agent-to-agent communication (future-ready)

---

Project Structure

cfp-scout/
├── 📁 cfp_scout/                       # Main application package
│   ├── 📄 __init__.py                  # Package initialization
│   ├── 📁 core/                        # Core business logic
│   │   ├── 📄 __init__.py
│   │   ├── 📄 event_orchestrator.py    # Central pipeline coordinator
│   │   ├── 📄 cfp_filter_agent.py      # LLM-based event filtering
│   │   ├── 📄 scraper.py               # Web scraping engine
│   │   └── 📄 email_sender.py          # Email notification system
│   ├── 📁 mcp/                         # MCP agent implementations
│   │   ├── 📄 __init__.py
│   │   ├── 📄 mcp_host.py              # MCP ecosystem coordinator
│   │   ├── 📄 scraper_mcp_server.py    # Scraper MCP server
│   │   ├── 📄 cfp_filter_mcp_server.py # Filter MCP server
│   │   ├── 📄 email_sender_mcp_server.py # Email MCP server
│   │   └── 📄 event_orchestrator_mcp_server.py # Orchestrator MCP server
│   ├── 📁 cli/                         # Command-line interface
│   │   ├── 📄 __init__.py
│   │   └── 📄 main.py                  # CLI main module
│   └── 📁 utils/                       # Utility functions
│       └── 📄 __init__.py
├── 📁 tests/                           # Comprehensive test suite
│   ├── 📄 __init__.py
│   ├── 📄 conftest.py                  # Test configuration
│   ├── 📄 test_runner.py               # Test execution coordinator
│   ├── 📁 unit/                        # Unit tests
│   │   ├── 📄 __init__.py
│   │   ├── 📄 test_email_sender.py
│   │   ├── 📄 test_event_mcp.py
│   │   └── 📄 test_scraper.py
│   ├── 📁 integration/                 # Integration tests
│   │   ├── 📄 __init__.py
│   │   ├── 📄 test_async_cfp_filter.py
│   │   ├── 📄 test_cfp_scout.py
│   │   └── 📄 test_mcp_integration.py
│   ├── 📁 security/                    # Security tests
│   │   ├── 📄 __init__.py
│   │   ├── 📄 test_email_sender_security.py
│   │   ├── 📄 test_scraper_security.py
│   │   ├── 📄 test_security_functions.py
│   │   ├── 📄 test_security_minimal.py
│   │   └── 📄 test_security_simple.py
│   ├── 📁 performance/                 # Performance tests
│   │   ├── 📄 __init__.py
│   │   └── 📄 test_performance_demo.py
│   └── 📁 fixtures/                    # Test fixtures and mock data
│       └── 📄 mock_data.py
├── 📁 docs/                            # Documentation
│   ├── 📄 README.md                    # This file
│   ├── 📄 IMPROVEMENTS.md              # Development roadmap
│   ├── 📄 SECURITY_FIXES_SUMMARY.md    # Security documentation
│   ├── 📄 TESTING_GUIDE.md             # Testing procedures
│   └── 📄 claude.md                    # Development guidelines
├── 📁 config/                          # Configuration files
│   └── 📄 env.example                  # Environment template
├── 📁 scripts/                         # Deployment and utility scripts
│   └── 📄 cfp-scout.service            # Systemd service configuration
├── 📁 logs/                            # Runtime logs and data
├── 📁 reports/                         # Generated reports
├── 📄 cfp_scout.py                     # Main entry point
├── 📄 requirements.txt                 # Python dependencies
├── 📄 pytest.ini                      # Test configuration
└── 📄 LICENSE                          # MIT license

File Summary

Core Application Files

File	Purpose	Key Features
cfp_scout.py	Main entry point and CLI interface	• Command-line argument parsing • Execution mode selection (traditional/hybrid/mcp) • Scheduling and status monitoring • Integration with all core modules
cfp_scout/core/event_orchestrator.py	Central pipeline coordinator managing data flow between all agents	• Normalized event data structure • Pipeline stage management • Event deduplication and validation • JSON data storage and retrieval • Statistics tracking
cfp_scout/core/cfp_filter_agent.py	LLM-based event filtering using Ollama for relevance scoring	• Ollama integration for local LLM processing • User interest parsing from environment • Event relevance scoring (0-10 scale) • Concurrent processing support • Score caching for performance
cfp_scout/core/scraper.py	Web scraping engine for CFP events from confs.tech	• Selenium-based JavaScript rendering • Anti-bot detection handling • Content sanitization for security • Retry logic with exponential backoff • Chrome WebDriver management
cfp_scout/core/email_sender.py	Email notification system with HTML formatting	• Mailgun API and SMTP support • HTML email template generation • External terminal preview on macOS • Email address validation • Sensitive data masking for logs
cfp_scout/mcp/mcp_host.py	MCP ecosystem coordinator for agent communication	• Model Context Protocol implementation • Agent lifecycle management • JSON-RPC 2.0 communication • Error handling and recovery • Session management

MCP Agent Servers

File	Purpose	Key Features
cfp_scout/mcp/scraper_mcp_server.py	MCP server wrapper for web scraping functionality	• MCP protocol implementation • Connectivity testing tools • Scraping resource management • Error handling and status reporting
cfp_scout/mcp/cfp_filter_mcp_server.py	MCP server for AI-powered event filtering	• Ollama integration via MCP • Filtering tools and resources • Batch processing support • Configuration management
cfp_scout/mcp/email_sender_mcp_server.py	MCP server for email delivery and notifications	• Email sending tools • Connection testing • Template management • Delivery status tracking
cfp_scout/mcp/event_orchestrator_mcp_server.py	MCP server for pipeline coordination	• Pipeline management tools • Data flow coordination • Statistics and monitoring • Resource management

Configuration & Deployment

File	Purpose	Key Features
requirements.txt	Python dependencies specification	• Core libraries: requests, beautifulsoup4, selenium • MCP framework integration • Testing dependencies: pytest, pytest-mock • Security and performance libraries
config/env.example	Environment variables template	• Ollama configuration • User interest settings • Email provider setup • Scheduling and timezone config
scripts/cfp-scout.service	Systemd service configuration for Linux deployment	• Service definition for systemd • User and group configuration • Restart policies • Environment file integration

Testing & Quality Assurance

File	Purpose	Key Features
tests/integration/test_cfp_scout.py	Comprehensive test suite for main functionality	• End-to-end pipeline testing • Mode-specific test cases • Performance benchmarking • Error scenario validation
tests/integration/test_mcp_integration.py	MCP agent communication and integration tests	• Agent lifecycle testing • Protocol compliance validation • Communication flow testing • Error handling verification
tests/integration/test_async_cfp_filter.py	Asynchronous filtering functionality tests	• Concurrent processing tests • Performance validation • Error handling in async context
tests/security/test_email_sender_security.py	Security-focused email functionality tests	• Email validation testing • Injection prevention • Data masking verification • Authentication security
tests/security/test_scraper_security.py	Security testing for web scraping functionality	• Content sanitization tests • Anti-bot detection handling • Input validation • XSS prevention
tests/performance/test_performance_demo.py	Performance benchmarking and optimization tests	• Execution time measurement • Memory usage monitoring • Concurrent processing validation • Scalability testing
tests/test_runner.py	Test execution coordinator and runner	• Test suite orchestration • Custom test discovery • Result aggregation • Coverage reporting

Documentation & Analysis

File	Purpose	Key Features
docs/README.md	Complete project documentation and usage guide	• Quick start instructions • Architecture overview • Configuration guide • Deployment strategies
docs/IMPROVEMENTS.md	Development roadmap and enhancement tracking	• Feature improvement ideas • Technical debt items • Performance optimization targets • Architecture evolution plans
docs/SECURITY_FIXES_SUMMARY.md	Security enhancement documentation	• Vulnerability assessments • Implemented security fixes • Security best practices • Compliance considerations
docs/TESTING_GUIDE.md	Testing strategy and execution guide	• Test execution procedures • Quality assurance workflows • Performance testing guidelines • CI/CD integration
LICENSE	MIT license for open source usage	• Usage rights and restrictions • Distribution terms • Liability disclaimers

Runtime & Logs

Directory/File	Purpose	Key Features
logs/	Runtime execution logs and data storage	• Application execution logs • Event processing data (JSON) • Debug information • Performance metrics
tests/	Structured test suite organization	• Unit tests for individual components • Integration tests for workflows • Test fixtures and mock data • Configuration for test environments
reports/	Generated reports and analytics	• Execution summaries • Performance analytics • Error reports • System health metrics

This comprehensive file structure enables CFP Scout to operate as a production-ready system with proper separation of concerns, robust testing, and multiple deployment strategies.

---

MCP Integration

CFP Scout implements Anthropic's Model Context Protocol across all agents:

MCP Agent Ecosystem

Main Orchestrator (cfp_scout.py)

• Multi-Mode Execution: Traditional, Hybrid, Pure MCP
• Scheduling: Daily execution with timezone support
• Monitoring: Health checks and performance tracking
• Deployment: Native Python, systemd, cron integration

MCP Servers: Each agent implements full MCP protocol with tools, resources, and prompts:

• Scraper MCP Server: Web scraping with connectivity testing
• Event Orchestrator MCP Server: Pipeline management and statistics
• CFP Filter MCP Server: AI filtering with Ollama integration
• Email Sender MCP Server: Email delivery with connection testing

---

Monitoring & Status

Status Monitoring

# Check orchestrator status
python cfp_scout.py --status

# View logs
tail -f logs/cfp_scout.log

# Check background process
ps aux | grep "python3 cfp_scout.py"

# Example status output:
# execution_mode: hybrid
# schedule_time: 08:00
# timezone: Europe/London
# last_execution: 2025-05-30T18:18:44.694000
# execution_count: 1
# total_events_processed: 27

What Happens During Execution

1. 🌐 Web Scraping: Scrapes CFP events from confs.tech/cfp using Selenium
2. 🔄 Data Processing: Normalizes and deduplicates event data
3. 🤖 AI Filtering: Uses Ollama LLM to score events based on your interests
4. 📧 Email Notification: Sends beautiful HTML emails via Mailgun or SMTP
5. 🖥️ Terminal Preview: Launches external terminal with email content
6. 📝 Logging: Saves detailed logs for monitoring

---

Contributing

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Make your changes and add tests
4. Update documentation (README, docstrings)
5. Commit your changes: git commit -m 'Add amazing feature'
6. Push to branch: git push origin feature/amazing-feature
7. Submit a Pull Request

Guidelines

• Follow modular architecture principles
• Add tests for new functionality
• Update documentation for any changes
• Remove deprecated/unused code
• Ensure MCP compatibility for new agents
• Maintain backward compatibility when possible
• Test all execution modes (traditional, hybrid, mcp)

---

License

This project is licensed under the MIT License. See LICENSE for details.

---

Support

• Issues: Report bugs via GitHub Issues
• Documentation: Check README and code comments
• MCP Resources: Anthropic MCP Documentation
• Ollama Setup: Ollama Installation Guide

---

Built with ❤️ using Anthropic's Model Context Protocol and Ollama for local AI processing

🎉 Production-Ready Native Python Deployment!