LangGraph, Hybrid RAG, and Signature Engine: A Universal Graph for Streaming Data
Moonshot AI
An architecture of an asynchronous graph based on LangGraph is presented, combining LLM, hybrid RAG (vector search + BM25), and signatures (YARA, Sigma) for processing streaming data. The system was tested on cybersecurity logs but can be applied to review analysis, spam filtering, and document moderation.
A developer has introduced an architectural pattern that combines an LLM, hybrid RAG, and the YARA and Sigma signature engines in a single asynchronous graph on LangGraph. The log stream is read by Vector, split by a Lua script into chunks of 250 lines with a 20-line overlap, then passed through Kafka to a consumer with retry logic and a DLQ. The graph contains two parallel branches: an AI branch with filtering, log grouping by Agent 1, description generation by Description Agent, and RAG Agent 2, which searches for MITRE ATT&CK techniques; a deterministic branch with parse_logs, YARA, and Sigma scanning. All four streams merge into Agent 3, which produces a final report: confirmed incidents go into the final report, unconfirmed ones go into a "Requires manual review" block. RAG is implemented as hybrid search: vector (ChromaDB) with weight 0.6 and BM25 with weight 0.4, followed by LLM re-ranking. YARA uses the native yara-python compiler, Sigma uses a custom rule-based engine. The system has been tested on cybersecurity logs but is universal.
Source: Habr — хаб ИИ —
original