The more lottery tickets you buy, the higher your chances of winning, but spending more than you win is obviously not a wise strategy. Something similar happens in AI powered by deep learning: we know that the larger a neural network is (i.e., the more parameters it has), the better it can learn the task we set for it.
However, the strategy of making it infinitely large during training is not only impossible but also extremely inefficient. Scientists have tried to imitate the way biological brains learn, which is highly resource-efficient, by providing machines with a gradual training process that starts with simpler examples and progresses to more complex ones—a model known as "curriculum learning."
Surprisingly, however, they found that this seemingly sensible strategy is irrelevant for overparameterized (very large) networks.
A study in the Journal of Statistical Mechanics: Theory and Experiment sought to understand why this "failure" occurs, suggesting that these overparameterized networks are so "rich" that they tend to learn by following a path based more on quantity (of resources) than quality (input organized by increasing difficulty).
This might actually be good news, as it suggests that by carefully adjusting the initial size of the network, curriculum learning could still be a viable strategy, potentially promising for creating more resource-efficient, and therefore less energy-consuming, neural networks.
To read more, click here.