Reading the Next Evolution of Large Language Models (LLMs) Through the "Densing Law"

Introduction

At the forefront of AI research, new metrics and laws aimed at balancing efficiency and performance keep appearing. The paper we focus on this time, "Densing Law of LLMs" (2024), proposes a novel evaluation criterion called "capability density" and reveals the "Densing Law," which captures the pace of LLM evolution as exponential. Based on this law, we take a closer look at the future of LLMs.

Link: Densing Law of LLMs
https://arxiv.org/pdf/2412.04315

---

Capability density: an evaluation criterion for a new era

What is capability density?
Capability density is an evaluation metric that uses a model's "effective parameter size." Specifically, it is calculated by dividing the model's performance by its actual number of parameters. This simple concept leads to important implications such as the following:

• Evaluating performance and efficiency together: Goes beyond simple performance comparisons and makes it possible to evaluate efficiency with costs and resources taken into account.
• Guidance for model design: Enables the development of highly efficient models even under resource constraints.

---

The exponential evolution shown by the "Densing Law"

The paper proposes the "Densing Law," under which capability density doubles roughly every 3.3 months. This pace suggests that model efficiency is evolving exponentially, and it points to several interesting prospects:

1. A future of fewer parameters
   The number of parameters needed falls rapidly while past performance is maintained. For example, it is highly likely that ChatGPT-level performance will be achieved with less than half the parameters.

2. Lower inference costs
   With the arrival of highly efficient models, computing costs fall and energy efficiency improves. This is expected to accelerate the spread of open-source models.

Research evidence and validation

The validity of this law has been demonstrated through an analysis of capability density across 29 open-source models. Across diverse benchmarks such as MMLU and HumanEval, models with higher capability density were also confirmed to have the advantage.

---

The future of open-source models

From the perspective of the "Densing Law," the following developments are expected:

1. Open-source LLMs that rival commercial models
   By the end of 2024, open-source models with performance comparable to ChatGPT GPT-4 Turbo may appear.

2. Better cost efficiency
   Running open-source LLMs delivers substantial cost savings compared with commercial models. On the other hand, operational infrastructure and technical challenges also need to be considered.

3. Faster evolution
   Active research and development in the open-source community will likely drive rapid improvement of LLMs.

The landscape of open-source LLMs

Summary

The future shown by the "Densing Law" is groundbreaking as a new indicator of AI model evolution. Based on it, an era is predicted in which open-source models reach performance close to that of commercial models. What this brings may be not just technological progress but the democratization of AI itself. For engineers and companies, this is a trend that cannot be overlooked.