DeepSeek

Latest AI news, models and releases from DeepSeek. ['DeepSeek', 'DeepSeek 3.1', 'DeepSeek-4-pro', 'DeepSeek API', 'deepseek-coder-v2', 'DeepSeek-Coder-V2-Lite', 'DeepSeek-GRM', 'DeepSeek large model', 'DeepSeekMath-V2', 'DeepSeekMoE', 'DeepSeek-Prover-V1.5-Base', 'DeepSeek-Prover-V2', 'DeepSeek-Prover-V2-671B', 'DeepSeek-Prover-V2-7B', 'DeepSeek R1', 'DeepSeek-R1', 'DeepSeek R1-0528', 'DeepSeek-R1-0528', 'DeepSeek-R1-671B', 'DeepSeek-R1-Distilled-Llama-70B', 'DeepSeek-R1-Distilled-Qwen-32B', 'DeepSeek-R1-Distill-Qwen-32B', 'DeepSeek-R1-Zero', 'DeepSeek R2', 'DeepSeek-R2', 'DeepSeek Sparse Attention (DSA)', 'DeepSeek V1', 'DeepSeek V2', 'DeepSeek-V2', 'DeepSeek-V2.5', 'DeepSeek V3', 'DeepSeek-V3', 'DeepSeek V3.1', 'DeepSeek V3.1-Terminus', 'DeepSeek v3.2', 'DeepSeek V3.2', 'DeepSeek-V3.2', 'DeepSeek V3.2-Exp', 'DeepSeek-V3-Base', 'DeepSeek V3/R1']

Open Source 🇺🇸

Gigatoken: BPE Tokenizer in Rust Encodes Text at 24.53 GB/s, Up to 989x Faster Than HuggingFace Tokenizers

Gigatoken, a BPE tokenizer written in Rust by Stanford PhD student Marcel Rød, achieves text encoding speeds of 24.53 GB/s on a 144-core AMD EPYC system, outperforming HuggingFace tokenizers by 989x and OpenAI's tiktoken by 681x. The library supports 23 tokenizer families and attributes its speed to a hand-optimized pretokenizer using SWAR SIMD techniques and pretoken caching.

OpenAIOpenAI Hugging FaceHugging Face Google/DeepMindGoogle/DeepMind MetaMeta Alibaba/QwenAlibaba/Qwen DeepSeekDeepSeek Moonshot AIMoonshot AI NVIDIANVIDIA
MarkTechPost24.07 · 01:03
Fresh news