Why Micro F1=0.94 May Hide a Useless Classifier: A Multi-Label Problem Breakdown

The author develops AI moderation for Discord based on a ruBERT tiny2 model, fine-tuned for multi-label classification with 15 labels (SAFE, TOXIC, ADVERTISEMENT, SPAM, SCAM, THREAT, EVASION, FLOOD, and others). The dataset contains about 1 million labeled examples. After training, the model achieved test micro F1=0.9358 and macro F1=0.8396 — a difference of nearly 0.1 signaled the need to check metrics for each class individually. It turned out that rare classes (e.g., EVASION with 35,000 samples) drown in the mass of frequent ones (SAFE with 400,000), and micro F1 does not reflect errors on them. To combat imbalance, pos_weight was used in BCEWithLogitsLoss. Detailed analysis revealed that TOXIC has recall≈0.78, meaning the model misses a noticeable portion of toxic messages. The FLOOD and IMAGE_SCAM classes are almost absent in the test split, but their processing is delegated to other system components, not the text classifier. The author also notes that selecting the best checkpoint based on training loss leads to overfitting: the optimal variant was found using macro F1, not loss. The main takeaway: the KISS principle applies to implementation, but not to the task — class imbalance, metric differences, separate precision/recall for each class, and error costs cannot be ignored. The next step is to calibrate thresholds for each class individually.