Self-supervised learning is the future of AI

Despite the huge contributions of deep learning to the field of artificial intelligence, there’s something very wrong with it: It requires huge amounts of data. This is one thing that both the pioneers and critics of deep learning agree on. In fact, deep learning didn’t emerge as the leading AI technique until a few years ago because of the limited availability of useful data and the shortage of computing power to process that data.

Reducing the data-dependency of deep learning is currently among the top priorities of AI researchers.

In his keynote speech at the AAAI conference, computer scientist Yann LeCun discussed the limits of current deep learning techniques and presented the blueprint for “self-supervised learning,” his roadmap to solve deep learning’s data problem. LeCun is one of the godfathers of deep learning and the inventor of convolutional neural networks (CNN), one of the key elements that have spurred a revolution in artificial intelligence in the past decade.

Self-supervised learning is one of several plans to create data-efficient artificial intelligence systems. At this point, it’s really hard to predict which technique will succeed in creating the next AI revolution (or if we’ll end up adopting a totally different strategy). But here’s what we know about LeCun’s masterplan.

A clarification on the limits of deep learning

The 💜 of EU tech

The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!

First, LeCun clarified that what is often referred to as the limitations of deep learning is, in fact, a limit of supervised learning. Supervised learning is the category of machine learning algorithms that require annotated training data. For instance, if you want to create an image classification model, you must train it on a vast number of images that have been labeled with their proper class.

“[Deep learning] is not supervised learning. It’s not just neural networks. It’s basically the idea of building a system by assembling parameterized modules into a computation graph,” LeCun said in his AAAI speech. “You don’t directly program the system. You define the architecture and you adjust those parameters. There can be billions.”

Deep learning can be applied to different learning paradigms, LeCun added, including supervised learning, reinforcement learning, as well as unsupervised or self-supervised learning.

But the confusion surrounding deep learning and supervised learning is not without reason. For the moment, the majority of deep learning algorithms that have found their way into practical applications are based on supervised learning models, which says a lot about the current shortcomings of AI systems. Image classifiers, facial recognition systems, speech recognition systems, and many of the other AI applications we use every day have been trained on millions of labeled examples.

Reinforcement learning and unsupervised learning, the other categories of learning algorithms, have so far found very limited applications.

Where does deep learning stand today?

Supervised deep learning has given us plenty of very useful applications, especially in fields such as computer vision and some areas of natural language processing. Deep learning is playing an increasingly important role in sensitive applications, such as cancer detection. It is also proving to be extremely useful in areas where the scale of the problem is beyond being addressed with human efforts, such as—with some caveats—reviewing the huge amount of content being posted on social media every day.

“If you take deep learning from Facebook, Instagram, YouTube, etc., those companies crumble,” LeCun says. “They are completely built around it.”

But as mentioned, supervised learning is only applicable where there’s enough quality data and the data can capture the entirety of possible scenarios. As soon as trained deep learning models face novel examples that differ from their training examples, they start to behave in unpredictable ways. In some cases, showing an object from a slightly different angle might be enough to confound a neural network into mistaking it with something else.

Deep reinforcement learning has shown remarkable results in games and simulation. In the past few years, reinforcement learning has conquered many games that were previously thought to off-limits for artificial intelligence. AI programs have already decimated human world champions at StarCraft 2, Dota, and the ancient Chinese board game Go.

But the way these AI programs learn to solve problems is drastically different from that of humans. Basically, a reinforcement learning agent starts with a blank slate and is only provided with a basic set of actions it can perform in its environment. The AI is then left on its own to learn through trial-and-error how to generate the most rewards (e.g., win more games).

This model works when the problem space is simple and you have enough compute power to run as many trial-and-error sessions as possible. In most cases, reinforcement learning agents take an insane amount of sessions to master games. The huge costs have limited reinforcement learning research to research labs owned or funded by wealthy tech companies.

Reinforcement learning systems are very bad at transfer learning. A bot that plays StarCraft 2 at grandmaster level needs to be trained from scratch if it wants to play Warcraft 3. In fact, even small changes to the StarCraft game environment can immensely degrade the performance of the AI. In contrast, humans are very good at extracting abstract concepts from one game and transferring it to another game.

Reinforcement learning really shows its limits when it wants to learn to solve real-world problems that can’t be simulated accurately. “What if you want to train a car to drive itself? And it’s very hard to simulate this accurately,” LeCun said, adding that if we wanted to do it in real life, “we would have to destroy many cars.” And unlike simulated environments, real life doesn’t allow you to run experiments in fast forward, and parallel experiments, when possible, would result in even greater costs.

The three challenges of deep learning

LeCun breaks down the challenges of deep learning into three areas.

First, we need to develop AI systems that learn with fewer samples or fewer trials. “My suggestion is to use unsupervised learning, or I prefer to call it self-supervised learning because the algorithms we use are really akin to supervised learning, which is basically learning to fill in the blanks,” LeCun says. “Basically, it’s the idea of learning to represent the world before learning a task. This is what babies and animals do. We run about the world, we learn how it works before we learn any task. Once we have good representations of the world, learning a task requires few trials and few samples.”

Babies develop concepts of gravity, dimensions, and object persistence in the first few months after their birth. While there’s debate on how much of these capabilities are hardwired into the brain and how much of it is learned, what is for sure is that we develop many of our abilities simply by observing the world around us.

The second challenge is creating deep learning systems that can reason. Current deep learning systems are notoriously bad at reasoning and abstraction, which is why they need huge amounts of data to learn simple tasks.

“The question is, how do we go beyond feed-forward computation and system 1? How do we make reasoning compatible with gradient-based learning? How do we make reasoning differentiable? That’s the bottom line,” LeCun said.

System 1 is the kind of learning tasks that don’t require active thinking, such as navigating a known area or making small calculations. System 2 is the more active kind of thinking, which requires reasoning. Symbolic artificial intelligence, the classic approach to AI, has proven to be much better at reasoning and abstraction.

But LeCun doesn’t suggest returning to symbolic AI or to hybrid artificial intelligence systems, as other scientists have suggested. His vision for the future of AI is much more in line with that of Yoshua Bengio, another deep learning pioneer, who introduced the concept of system 2 deep learning at NeurIPS 2019 and further discussed it at AAAI 2020. LeCun, however, did admit that “nobody has a completely good answer” to which approach will enable deep learning systems to reason.

The third challenge is to create deep learning systems that can lean and plan complex action sequences, and decompose tasks into subtasks. Deep learning systems are good at providing end-to-end solutions to problems but very bad at breaking them down into specific interpretable and modifiable steps. There have been advances in creating learning-based AI systems that can decompose images, speech, and text. Capsule networks, invented by Geoffry Hinton, address some of these challenges.

But learning to reason about complex tasks is beyond today’s AI. “We have no idea how to do this,” LeCun admits.

Self-supervised learning

The idea behind self-supervised learning is to develop a deep learning system that can learn to fill in the blanks.

“You show a system a piece of input, a text, a video, even an image, you suppress a piece of it, mask it, and you train a neural net or your favorite class or model to predict the piece that’s missing. It could be the future of a video or the words missing in a text,” LeCun says.

The closest we have to self-supervised learning systems are Transformers, an architecture that has proven very successful in natural language processing. Transformers don’t require labeled data. They are trained on large corpora of unstructured text such as Wikipedia articles. And they’ve proven to be much better than their predecessors at generating text, engaging in conversation, and answering questions. (But they are still very far from really understanding human language.)

Transformers have become very popular and are the underlying technology for nearly all state-of-the-art language models, including Google’s BERT, Facebook’s RoBERTa, OpenAI’s GPT2, and Google’s Meena chatbot.

More recently, AI researchers have proven that transformers can perform integration and solve differential equations, problems that require symbol manipulation. This might be a hint that the evolution of transformers might enable neural networks to move beyond pattern recognition and statistical approximation tasks.

So far, transformers have proven their worth in dealing with discreet data such as words and mathematical symbols. “It’s easy to train a system like this because there is some uncertainty about which word could be missing but we can represent this uncertainty with a giant vector of probabilities over the entire dictionary, and so it’s not a problem,” LeCun says.

But the success of Transformers has not transferred to the domain of visual data. “It turns out to be much more difficult to represent uncertainty and prediction in images and video than it is in text because it’s not discrete. We can produce distributions over all the words in the dictionary. We don’t know how to represent distributions over all possible video frames,” LeCun says.

For each video segment, there are countless possible futures. This makes it very hard for an AI system to predict a single outcome, say the next few frames in a video. The neural network ends up calculating the average of possible outcomes, which results in blurry output.

“This is the main technical problem we have to solve if we want to apply self-supervised learning to a wide variety of modalities like video,” LeCun says.

LeCun’s favored method to approach supervised learning is what he calls “latent variable energy-based models.” The key idea is to introduce a latent variable Z which computes the compatibility between a variable X (the current frame in a video) and a prediction Y (the future of the video) and selects the outcome with the best compatibility score. In his speech, LeCun further elaborates on energy-based models and other approaches to self-supervised learning.

The future of deep learning is not supervised

“I think self-supervised learning is the future. This is what’s going to allow to our AI systems, deep learning system to go to the next level, perhaps learn enough background knowledge about the world by observation, so that some sort of common sense may emerge,” LeCun said in his speech at the AAAI Conference.

One of the key benefits of self-supervised learning is the immense gain in the amount of information outputted by the AI. In reinforcement learning, training the AI system is performed at scalar level; the model receives a single numerical value as reward or punishment for its actions. In supervised learning, the AI system predicts a category or a numerical value for each input.

In self-supervised learning, the output improves to a whole image or set of images. “It’s a lot more information. To learn the same amount of knowledge about the world, you will require fewer samples,” LeCun says.

We must still figure out how the uncertainty problem works, but when the solution emerges, we will have unlocked a key component of the future of AI.

“If artificial intelligence is a cake, self-supervised learning is the bulk of the cake,” LeCun says. “The next revolution in AI will not be supervised, nor purely reinforced.”

This story is republished from TechTalks, the blog that explores how technology is solving problems… and creating new ones. Like them on Facebook here and follow them down here: