Redinators

Description: Redis powered anomaly detection in HDFS logs

Our use case is HDFS log anomaly detection using traditional machine learning along with Redis Vector similarity (During model training) and Redis VL DB for inference

A trained ML ensemble model is used for anomaly detection which contains the vector similarity as one of the ensemble models

During inference, we use kafka broker, consumer and a spark job that creates embeddings of log entries and stores them in RedisVL Database. The anomaly detection service loads the trained ensemble model and retrieves logs embeddings from Redis VL in sub millisecond latency

Finally redis cache is used to store the predictions of anomalous log events so if a duplicate log line gets ingested, it avoids recomputation

We have also built an MCP server for basic querying with natural language