Skip to content

feat(agent): hybrid BM25 search_wiki tool for query/chat agent - #234

Open
sebastianbraun25 wants to merge 1 commit into
VectifyAI:mainfrom
sebastianbraun25:feat/issue-233-hybrid-search
Open

feat(agent): hybrid BM25 search_wiki tool for query/chat agent#234
sebastianbraun25 wants to merge 1 commit into
VectifyAI:mainfrom
sebastianbraun25:feat/issue-233-hybrid-search

Conversation

@sebastianbraun25

Copy link
Copy Markdown

Problem

openkb query/openkb chat navigate the wiki purely via index.md one-line summaries plus
LLM-directed page reads (read_file, get_page_content). One-line summaries cannot surface facts
buried deep in a page body (a specific figure, a niche keyword, an exact term). As a wiki grows,
the agent increasingly misses pages whose index.md summary doesn't mention the exact term the
user asked about, even though the page body contains the answer.

Solution / Changes

  • 新建 module openkb/fulltext_index.py: a dependency-free BM25 (Robertson/Sparck-Jones) index
    over concepts/, entities/, and summaries/ wiki pages (PAGE_CONTENT_DIRS), rebuilt
    in-memory on construction — cheap at the wiki sizes this pattern targets, no on-disk cache or
    incremental update needed. Exposes 维基FullTextIndex.search(query, top_k) returning ranked
    搜索Hit(path, title, score, snippet) results.
  • openkb/agent/tools.py: new plain function search_wiki(query, wiki_root, top_k=5) formatting
    ranked hits as [[wikilink]] — title (score) / snippet text for the agent.
  • openkb/agent/query.py: wires search_wiki in as a fourth tool (read_file,
    get_page_content, search_wiki, get_image) in build_query_agent — and therefore also in
    build_chat_agent, which extends the query agent's tools. Instructions updated with a new
    search-strategy step describing it as a hybrid, additive fallback: use alongside, not
    instead of, index.md navigation, so recall can only improve, never regress.
  • No new dependency: BM25 is small enough in pure Python that a search-library dependency (e.g.
    Whoosh) isn't warranted, and openkb's dependencies are pinned exactly and vetted deliberately
    (see pyproject.toml).
  • README: one paragraph documenting the hybrid retrieval behavior under "Query & Chat".
  • Backward compatible: existing tools/instructions unchanged aside from the new step; no config,
    CLI flag, or schema changes.

Testing

  • tests/test_fulltext_index.py (new): tokenization/BM25 ranking, PAGE_CONTENT_DIRS-only
    scoping, top_k, snippet extraction, title fallback, empty-index/empty-query edge cases.
  • tests/test_agent_tools.py (new Test搜索维基 class): search_wiki formatting, no-match
    message, top_k.
  • tests/test_query.py (updated): build_query_agent now exposes four tools including
    search_wiki.
  • ruff check, ruff format --check, mypy openkb, pytest all green (pre-existing,
    environment-specific failures unrelated to this change are unaffected).

问题

Resolves #233

Adds a dependency-free BM25 full-text index (openkb/fulltext_index.py) over concepts/entities/summaries pages, exposed as a new search_wiki tool alongside index.md-driven navigation in build_query_agent. Additive hybrid retrieval: surfaces pages whose one-line index summary omits a buried detail, without replacing existing navigation. Resolves VectifyAI#233.
注册 for free to join this conversation on GitHub. Already have an account? 登录 to comment

标签

None yet

项目

None yet

Development

Successfully merging this pull request may close these issues.

feat(agent): hybrid BM25 full-text search tool for query/chat agent

1 participant