Lifelong learning is a goal not just for humans. Intelligent systems also need to continually learn and adapt. The problem: When machine learning models are repeatedly confronted with new data or changing tasks, they, like humans, tend to forget what they previously learned. However, the extent of this is so severe that experts refer to it as "catastrophic forgetting." As a result, it is still common in many fields to periodically retrain models from scratch. This retraining is extremely inefficient and unsustainable, comparable to humans needing to relearn how to walk just to keep up with the latest political developments.
Collaborative Learning: How Small Models Can Achieve Big Results Together
"The high costs of retraining and the immense computational demands make machine learning models a luxury that often only large industrial companies can afford," explains CISPA researcher Sebastian Stich. Yet, fields like medicine could significantly benefit from their capabilities. Approaches to address this issue have been around for a while. One way to save resources is to move away from training one large and complex model toward training several smaller models that share their knowledge rather than their training data. "This is referred to as federated learning, a type of collaborative learning. Existing methods focus on using distributed training data to ultimately train a large model. We are pursuing a different approach: we want small, independent models to collaborate effectively," Stich says.
Better Knowledge Sharing, Continuous Learning, and Meaningful Forgetting
His plan includes three core objectives: "We aim to use improved training algorithms to make models better at adapting to new data and forgetting outdated knowledge. Additionally, we will work on enabling models trained on different devices and with various data sources to share their knowledge effectively. Each of these models develops a type of specialized expertise based on the data it was trained on and can be used for specific tasks. Lastly, we want to enable these small expert models to collaboratively tackle complex tasks," Stich explains.
The result would be a more sustainable, adaptable, and equitable form of machine learning that benefits not only large corporations but also smaller players. "Our research could also be extremely useful in medicine. Patient data is both scarce and sensitive. If local hospitals can work with small models, it helps protect data, conserve resources, and still allow models to improve collaboratively, ultimately providing better support, for instance, in cancer diagnostics," Stich says.
The EU Supports Innovative and Independent Research
In 2024, a total of 328 researchers from 25 countries across Europe will receive a Consolidator Grant. Sixteen of these prestigious grants are awarded in the field of computer science and informatics. The EU, through the ERC Grants, supports outstanding researchers in establishing an independent research team, enabling them to implement their most promising research ideas. Receiving an ERC Consolidator Grant is an honor for Stich as well as for his peers. "The fact that the EU believes in my research project means a lot to me. This support is not only recognition of our previous work but also an incentive to continue developing innovative solutions to the challenges of machine learning. I am convinced that our research can make a significant contribution to a more sustainable and equitable technological future," Stich says.
About the ERC
The ERC, set up by the European Union in 2007, is the premier European funding organisation for excellent frontier research. It funds creative researchers of any nationality and age, to run projects based across Europe. The ERC offers four core grant schemes: Starting Grants, Consolidator Grants, Advanced Grants and Synergy Grants. With its additional Proof of Concept Grant scheme, the ERC helps grantees to bridge the gap between their pioneering research and early phases of its commercialisation. The ERC is led by an independent governing body, the Scientific Council. Since November 2021, Maria Leptin is the President of the ERC. The overall ERC budget from 2021 to 2027 is more than €16 billion, as part of the Horizon Europe programme, under the responsibility of European Commissioner for Startups, Research and Innovation, Ekaterina Zaharieva.
