Web Search Backends for Hermes Agent: Firecrawl, SearXNG, Tavily & More

TLDR: Hermes Agent ships two model-callable web tools, web_search and web_extract, configured through one backend selection in hermes tools or config.yaml. Firecrawl is the labeled default backend. With no credentials and nothing selected, a keyless ring (Exa, Parallel, Firecrawl, Keenable) serves requests, so web tools work out of the box. See the official docs for the authoritative reference.

Introduction: one config surface, two tools

Hermes Agent includes two model-callable web tools backed by multiple providers: web_search (ranked results) and web_extract (readable content from URLs). Both are configured through a single backend selection, made with hermes tools or directly in config.yaml (docs). Search-only backends such as SearXNG, Brave, and DDGS need a separate extract-capable provider for web_extract. A fresh install with no credentials still gets working web tools through the keyless ring described below.

How This Guide Was Built

This guide is based on the official docs page Hermes Web Search. It was not run hands-on: no commands were executed and no outputs were observed on a live install. Treat the documentation as the authoritative source, and consult it for any detail beyond this summary.

The Backends at a Glance

The table below reproduces the backend list from the official docs:

Provider	Env Var	Search	Extract	Free tier
Firecrawl (default)	FIRECRAWL_API_KEY (optional, keyless when selected)	✔	✔	500 credits/mo
SearXNG	SEARXNG_URL	✔	—	Free (self-hosted)
Brave Search (free tier)	BRAVE_SEARCH_API_KEY	✔	—	2 000 queries/mo
DDGS (DuckDuckGo)	none	✔	—	Free
Exa	EXA_API_KEY (optional)	✔	✔	Keyless ring member; 1 000 searches/mo with key
Parallel	PARALLEL_API_KEY (optional)	✔	✔	Keyless ring member; paid with key
Tavily	TAVILY_API_KEY (optional)	✔	✔	Opt-in keyless when selected
Perplexity	PERPLEXITY_API_KEY	✔	✔ (query-relevant snippets)	Paid
Keenable	KEENABLE_API_KEY (optional)	✔	✔	Keyless ring member; paid with key
xAI (Grok)	XAI_API_KEY or hermes auth add xai-oauth	✔	—	Paid (SuperGrok or per-token)
OpenAI Native (Codex)	hermes auth add openai-codex	✔	—	Requires a ChatGPT/Codex subscription

Firecrawl is the labeled default backend. Only when no credentials are set and nothing is selected do requests rotate across the keyless free tiers described below. Brave Search, DDGS, xAI, and OpenAI Native are search-only, so pair them with an extract-capable provider when you also need web_extract. DDGS uses the ddgs Python package under the hood.

Quick Setup

Run hermes tools, open Web Search & Extract, and pick a provider. The wizard asks for any required URL or API key and writes it to your config. Confirm the result with hermes setup, which prints the detected backend, for example Web Search & Extract (searxng).

Nous subscribers get managed web search through the Tool Gateway with no API key. New installs can run hermes setup --portal to log in and enable all gateway tools at once; existing installs can enable only web through hermes tools.

Configuration: one backend or a search/extract split

Each capability accepts the same value set: firecrawl | searxng | brave-free | ddgs | tavily | perplexity | keenable | exa | parallel | xai | nous, where nous selects the managed Tool Gateway.

# ~/.hermes/config.yaml
web:
  backend: "searxng"

To split the two capabilities, set them separately. This example uses SearXNG (free) for search and Firecrawl for extraction:

# ~/.hermes/config.yaml
web:
  search_backend: "searxng"     # used by web_search
  extract_backend: "firecrawl"  # used by web_extract

Priority per capability is: (1) web.search_backend or web.extract_backend, (2) the shared web.backend, (3) auto-detection from environment variables. Once a shared selection exists, adding a key to .env does not reroute web traffic. A per-capability key affects only its own capability.

Auto-Detection Order

If no shared backend has ever been selected, Hermes picks the first available backend based on which credentials are set:

Credential present	Auto-selected backend
TAVILY_API_KEY	tavily
PERPLEXITY_API_KEY	perplexity
EXA_API_KEY	exa
PARALLEL_API_KEY	parallel
FIRECRAWL_API_KEY or FIRECRAWL_API_URL	firecrawl
Nous Tool Gateway ready (Portal subscription)	managed web search via the Tool Gateway
SEARXNG_URL	searxng
BRAVE_SEARCH_API_KEY	brave-free
ddgs package importable	ddgs
Nous identity present (any account, including guests)	managed web search (web_search only)
Nothing set	keyless ring: exa / parallel / firecrawl / keenable (round-robin)

xAI is not in the auto-detection chain. You must opt in with web.backend: "xai".

The Keyless Free-Tier Ring

A fresh install with no web credentials works out of the box. Requests rotate round-robin across Exa, Parallel, Firecrawl, and Keenable. When a vendor rate-limits a request, the request retries on the next vendor in the ring. A search that a vendor refuses with a 401, 402, 403, or 5xx response retries the same way. The ring is strictly a last resort: any configured backend or present API key takes precedence. Requests carry no user identifiers, only a random per-process session ID that rotates on restart. To turn the tier off, set web.keyless_fallback: false.

In hermes tools, Exa, Parallel, and Keenable each appear as a Free (keyless) row and a Paid (API key) row. Choosing Free pins that vendor’s anonymous endpoint, and choosing Paid pins the keyed path. The choice is stored as web.provider_tier.<name>: free|paid. If you leave it unset, a present key means paid and no key means the keyless ring. Free tiers are rate-limited under burst load, but sustained normal use works. Pinning one vendor stops the rotation, and the ring then serves only as failover on throttles.

A one-shot keyless rescue also applies to keyed backends. If your chosen backend fails a single call, that call retries on the keyless ring, and the result notes rescued_from and backend_error. The failover is never sticky, so the next call tries your chosen backend again. Disable it with web.keyless_rescue: false, which is also off whenever keyless_fallback is off.

Provider Walkthroughs

Firecrawl (default)

Firecrawl offers full-featured search and extraction and is the recommended choice for most users. Get a key at firecrawl.dev; the free tier includes 500 credits per month. To self-host, set FIRECRAWL_API_URL (for example http://localhost:3002). When that variable is set, the API key is optional, and you can disable server auth with USE_DB_AUTHENTICATION=false.

SearXNG (free, self-hosted)

SearXNG is a privacy-respecting metasearch engine that aggregates 70+ engines. It needs no API key and is search-only. Self-host it with Docker Compose on port 8888 (container port 8080) using the searxng/searxng:latest image. JSON output is disabled by default, so copy settings.yml out of the container, add search: formats: [html, json], copy it back, and restart. Then verify:

curl -s "http://localhost:8888/search?q=test&format=json" | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'{len(d[\"results\"])} results')"

A 403 response means JSON is still disabled. Set SEARXNG_URL=http://localhost:8888 in ~/.hermes/.env, then set web.search_backend: "searxng". Public instances are listed at searx.space; filter for JSON-enabled ones. Their rate limits and uptime vary, so self-host for production use. Pair SearXNG with an extract backend, as shown in the configuration section.

Tavily

Tavily is an AI-optimised search and extract API. Select it in hermes tools or set web.backend: tavily to use it keyless, which is rate-limited. For higher limits, set TAVILY_API_KEY from app.tavily.com. The Tavily keyless docs cover the keyless option.

Perplexity

Perplexity’s Search API returns ranked, date-stamped results. For web_extract, it returns query-relevant snippets with elisions marked … rather than full pages. Get a key at perplexity.ai/account/api and read the quickstart. For full-page extraction, set Firecrawl, Exa, or Parallel as web.extract_backend.

Exa and Parallel

Exa offers neural search with semantic understanding. Get a key at exa.ai; the free tier includes 1 000 searches per month with a key. Parallel offers AI-native search and extraction. Get access at parallel.ai.

Brave Search (free tier)

BRAVE_SEARCH_API_KEY enables Brave Search with a 2 000 queries-per-month free tier. It is search-only; select it with web.backend: "brave-free", and pair it with an extract backend if you need web_extract.

DDGS

DDGS needs no key and uses the ddgs package. Hermes can lazy-install it on first use. If you prefer to install it yourself from a PM-prepared source checkout, run the package-management sync for ddgs, then reactivate the checkout and restart Hermes.

Keenable

Keenable is a keyless ring member. Set KEENABLE_API_KEY for the paid tier. The official docs do not include separate setup steps for it.

xAI (Grok) and OpenAI Native: search-only, with a trust caveat

xAI routes web_search through Grok’s server-side search tool on the Responses API. Authenticate with XAI_API_KEY or hermes auth add xai-oauth, then set web.backend: "xai". Optional knobs sit under web: xai::

# ~/.hermes/config.yaml
web:
  backend: "xai"
  xai:
    model: grok-build-0.1        # default
    allowed_domains: [arxiv.org] # optional, max 5; mutually exclusive with excluded_domains
    timeout: 90                  # seconds (default)

On a 401 with OAuth credentials, the provider performs one forced token refresh and retries. Environment-variable credentials (XAI_API_KEY) skip that retry. The trust caveat matters: unlike index-backed providers such as Brave, Tavily, and Exa, xAI is an LLM that chooses the URLs and writes the titles and descriptions. A crafted query could in principle steer it toward attacker-chosen URLs, so validate returned URLs before fetching, especially when the query came from untrusted input.

OpenAI Native declares the provider-executed web_search tool on the Codex Responses endpoint. Set web.search_backend: "openai-native" and authenticate with hermes auth add openai-codex; it has no API key of its own. The built-in works only on that endpoint, and it covers search only, so pair it with an extract backend through web.extract_backend. Selecting it replaces the client-side web_search function rather than adding to it. The same trust caveat applies, because the model drives the search and folds the results into its own answer.

How web_extract handles long pages

Backends return raw page markdown, which can be very large. web_extract applies a deterministic character budget and does no LLM summarization. The default budget is 15,000 characters. Over-budget pages are cut to a head-and-tail window (about 75% head, 25% tail) on markdown line boundaries, with a [TRUNCATED] footer that names the stored file and the exact read_file call to page through the middle. Configure the budget with web.extract_char_limit (default 15000, clamped 2000 to 500000), or pass char_limit on a single call. The per-provider timeout is web.extract_timeout (default 120 seconds; 0 disables it). For JavaScript-heavy pages, use browser_exec with new_tab(url) and wait_for_load(), then read the live DOM with page_info() or js().

Result Caching

Repeated web calls within a short window are served from cache. The web_search memo is per process and matches the same query and provider. web_extract stores full text under ~/.hermes/cache/web/, shared across the CLI, gateway, cron, and subagents. Concurrent identical searches are coalesced into one backend request. Only successful responses are cached. Keyless-rescue responses are never cached, and URLs on your security.website_blocklist are never served from cache.

Local and private URLs bypass the extract cache entirely. That covers localhost, 127.0.0.1, *.local, single-label LAN hosts, and private or link-local IP ranges. For public staging or tunnel hosts, list them in web.cache_exempt_hosts. Caching is controlled by web.cache_enabled (default true) and web.cache_ttl_minutes (default 20, clamped 1 to 1440). For live data such as scores or prices, lower the TTL or disable the cache.

Verify Your Setup

Run hermes setup. It reports the detected backend, for example ✅ Web Search & Extract (searxng). From a source checkout, activate the environment with source ./activate and run python -m tools.web_tools. That prints the active backend and its status.

Troubleshooting

When using SearXNG, if web_search returns {"success": false}, first check that the JSON endpoint responds. An HTTP 403 means JSON is disabled: add json to formats in settings.yml and restart. A connection error usually means the container is not running, so check with docker ps | grep searxng. If web_extract reports a “search-only backend”, set web.extract_backend to an extract-capable provider such as firecrawl, tavily, perplexity, keenable, exa, or parallel. If SearXNG returns zero results or a public instance rate-limits you, try a different query or a different searx.space instance, or self-host. A [TRUNCATED] footer is expected; raise web.extract_char_limit or pass a larger char_limit.

Optional Skill: searxng-search

Install it with hermes skills install official/research/searxng-search. The skill teaches the agent to call the SearXNG JSON API directly through curl or Python, filter by category (general, news, science), handle pagination and errors, and fall back gracefully when SearXNG is unreachable.

FAQ

Q: Can I use different providers for search and extract? A: Yes. Set web.search_backend and web.extract_backend independently, for example SearXNG for search and Firecrawl for extraction.

Q: Does Hermes need any credential for web tools to work? A: No. With no credentials and nothing selected, the keyless ring described above provides working web_search and web_extract.

Q: How do I disable the keyless fallback? A: Set web.keyless_fallback: false. To disable only the one-shot rescue for keyed backends, set web.keyless_rescue: false.

Q: Is xAI used automatically if I have XAI_API_KEY? A: No. xAI is not in the auto-detection chain, so you must opt in with web.backend: "xai".

Q: Where is the authoritative reference? A: The official docs page is the primary source for this guide.