dondai1234/master-fetch
Project Overview
Hound (master-fetch / hound-mcp)
A self-hosted MCP server that lets AI agents search, fetch, and crawl the web without API keys.
True Intent (AI Assessment)
Provide AI agents with a free, local, keyless MCP server for web search, fetching, crawling, and content extraction.
README Accuracy
Notes
Code matches the README's claims: 6 MCP tools, 10 search backends, Cloudflare Turnstile solver, HTTP-only fallback, self-repair CLI, and ONNX reranker are all present in source. The repo name (`master-fetch`) differs from the shipped product name (`Hound` / `hound-mcp`), which could confuse first-time viewers but is disclosed openly. Scrapes public search engines from the user's IP — a limitation the README states honestly.
This is a Python-based MCP (Model Context Protocol) server, published to PyPI as `hound-mcp`, that exposes web-research capabilities as tools an AI agent can call. It provides six MCP tools: `smart_fetch` (HTTP fetch with browser escalation), `smart_crawl` (same-domain best-first crawler), `smart_search` (keyless meta-search across 10 backends), `screenshot`, `cache_clear`, and `version`. The codebase is substantial (~30 source modules) and well-tested (~750 tests across ~40 test files).
The problem it targets is the friction and cost of giving LLM agents reliable web access. Commercial alternatives (Exa, Brave Search API, ScraperAPI, Firecrawl) require API keys, accounts, and per-request billing, and hosted browser services introduce third-party data exposure. Hound bundles anti-bot fetching (via patchright + a ported Cloudflare Turnstile solver), content extraction (trafilatura + markdownify), PDF/OCR handling, and metasearch scraping into one local process so an agent can do research at $0 with no external accounts.
The intended users are developers integrating web research into AI coding agents or assistants that speak MCP — Claude Code, Cursor, OpenCode, Pi, etc. Install is `pip install hound-mcp[all] && playwright install chromium`, then the MCP client points at the `hound` command. There is also a Pi-agent TypeScript extension shipped under `pi-extension/` for that specific ecosystem.
Internally, the architecture is modular: `server.py` implements the MCP interface; `fetcher.py` handles primp-based HTTP; `browser.py` wraps patchright for stealth browsing with a Cloudflare solver; `search.py` + `search_engines.py` + `search_metasearch.py` scrape 10 engines in parallel with circuit-breaker cooldowns and cross-backend consensus ranking; `reranker.py` uses a local ONNX cross-encoder for semantic re-ranking; `cache.py` provides a SQLite/WAL cache with TTL; `extractor.py`, `trafilatura_extractor.py`, `pdf_extractor.py`, and `ocr.py` handle content extraction; `updater.py` + `cli.py` implement a self-healing update/repair flow. Version 11.0.0 removed the `scrapling` dependency entirely and reimplemented its logic in-house.
Notable positives: extensive test coverage, graceful degradation to HTTP-only mode when browser deps can't install (e.g. Termux/aarch64), input validation (`security.py` validates CSS selectors), and a lazy-import strategy in `__init__.py` to keep cold start under one second. No obvious red flags — the code performs the tasks the README advertises, and web scraping / TLS impersonation via `primp` is a normal (if legally grey) approach for keyless search.
| Languages | Python, TypeScript, HTML |
| Runtime | Python 3.11+ |
| Framework | MCP (Model Context Protocol) SDK |
| Database | SQLite (via aiosqlite) for local cache |
| Package Manager | pip / hatchling; npm for the Pi extension |
| Key Dependencies | mcp, pydantic, trafilatura, primp, httpx, patchright, playwright, onnxruntime, pdfplumber, rapidocr, markdownify |
| Build Tool | Hatchling |
| Test Framework | pytest (with pytest-asyncio, pytest-mock) |