ChartNet from MIT: Teaching AI to Understand Charts with a Million-Scale Dataset
IBM
Alibaba/Qwen
Google/DeepMind
OpenAI
Hugging Face
MIT and IBM Research introduced ChartNet, a multimodal dataset of 1.7 million examples for training AI to understand charts. Small models fine-tuned on ChartNet compete with or outperform large models like GPT-4o on chart understanding tasks.
Researchers from MIT and IBM Research created ChartNet, a large-scale multimodal dataset for chart understanding. It contains 1.7 million examples generated from 150,000 unique charts by recovering Python code and applying augmentations to create diverse visualizations using six libraries (Matplotlib, Seaborn, Plotly, Vega-Altair, Pygal, Plotnine) and 24 chart types. The dataset supports multiple tasks: chart-to-code, chart-to-table, chart-to-text, and chart QA with chain-of-thought reasoning. Quality control includes automatic filtering (77% code execution success) and manual verification of 3,157 random charts. Fine-tuning small VLMs (e.g., Granite-Vision-3.3-2B) on ChartNet yields results competitive with or exceeding GPT-4o on several benchmarks. The dataset also includes a safety subset of 7,600 examples for sensitive topics.
- Сокращения
- VLM = Vision-Language Model — визуально-языковая модель
- LLM = Large Language Model — большая языковая модель
- QA = Question Answering — вопросно-ответная система
- CoT = Chain of Thought — цепочка рассуждений
Source: Habr — хаб NLP —
original