Cluster Mathematics: Understanding Smart Data Clustering with Our Log Anomaly Detection System. Part 1
Andrey Basov, head of technical support at MWS, describes the architecture of a new system for analyzing product health that detects anomalies in logs. The system uses LLM-based vectorization, PCA dimensionality reduction, and dynamic clustering with Bayesian adaptation and MST metrics. This is the first part of a series explaining the underlying mathematics and techniques.
Andrey Basov, head of the technical support team for the corporate products and services stream at MWS, describes a new system for analyzing product health that he and his colleagues developed after reworking their previous log anomaly detection approach. The system is a microservice architecture with an event-driven and incremental processing design, comprising components such as ml-log-anomaly-cluster-manager-detector for log reception, vectorization, clustering, and cluster state updates; ml-log-anomaly-cluster-manager-retrain for periodic cluster rebuilding; ml-log-anomaly-cluster-manager-cleaner for archiving and deleting outdated data; and ml-log-anomaly-cluster-manager-exporter for exporting results to monitoring systems like Grafana and OpenSearch. The services communicate through shared data stores: Qdrant stores log vectors (768-dimensional BGEM3 embeddings) and cluster metadata (centroids, boundaries, Bayesian parameters, MST metrics, flags), while PostgreSQL manages configuration, process state, and historical data. The data flow involves retrieving logs from OpenSearch, normalizing, deduplicating, vectorizing via LLM (BGEM3), dimensionality reduction with adaptive PCA, then clustering using cosine distance to find the nearest cluster; if the distance exceeds the adaptive boundary, the point is marked as an anomaly and stored for nightly processing. HDBSCAN is used to build new clusters from anomalies during nightly rebuilds. The system's mathematics are integrated into the cluster's object model, making each cluster a self-tuning agent with internal state, statistics, name, and behavior, enabling dynamic boundary adjustment, Bayesian threshold adaptation, centroid velocity and acceleration monitoring, MST-based density estimation, and incremental aggregate updates. The article also covers fundamental linear algebra concepts: vectors, vector operations, scalar product, and Euclidean norm, which are essential for understanding the clustering methods.
- Сокращения
- LLM = Large Language Model — большая языковая модель
- PCA = Principal Component Analysis — анализ главных компонент
- MST = Minimum Spanning Tree — минимальное остовное дерево
- HDBSCAN = Hierarchical Density-Based Spatial Clustering of Applications with Noise — иерархическая кластеризация на основе плотности с шумом
- PCA = Principal Component Analysis — анализ главных компонент
Source: Хабр — Data Mining —
original