The Dark Side of LLMs: How and Why They Are Turned into Weapons (and What to Do About It)

Jailbreaking is a type of prompt injection where an attacker assigns the LLM a role that overrides its safety training. For example, a prompt called SWILL was shared on forums, making DeepSeek believe it was a different model without censorship, after which it readily provided advice on destroying fingerprints and counterfeiting currency, although it still refused to discuss the Tiananmen Square incident. Jailbreaks work because LLM safety is not a separate module but part of the model's learned behavior; a jailbreak creates a strong contextual signal that shifts the probability distribution toward harmful outputs. Beyond jailbreaks, dedicated black-hat LLMs like WormGPT, EvilGPT, WolfGPT, and DarkBERT are trained specifically for cybercrime, though many are defunct or scams. Defenses include architectural separation of system and user prompts, input/output filters like AVI (Aligned/Agreement Validation Interface) or Llama Guard, adversarial fine-tuning, constitutional AI alignment, internal activation analysis, RAG-based protection, and red teaming. To gather real-time data on jailbreak techniques, Lakera launched the Gandalf project, an interactive game where participants hack simulated AI applications to extract secrets or force harmful behavior, with a leaderboard to boost engagement.