What is transfer learning and how is it used to pre-train models for new tasks?

Transfer learning is an approach where a pre-trained model is applied to a new, related task. This method helps save time and resources while enhancing the performance of machine learning models. There are various strategies for adapting pre-trained models to fit a new task.

What is transfer learning?

Transfer learning is a method from machine learning where a trained model is optimized for a new, similar task. Rather than training a new model from scratch for a specific task, the existing knowledge is used. Through minor adjustments, the pre-trained model is adapted to the new task, enabling it to handle different features. This approach saves time and resources, as it requires much smaller datasets for training, making it both more efficient and powerful.

How does transfer learning work?

Transfer learning involves taking a model that has already been fully trained for a specific task and applying it to a new, similar task. This method is especially effective when working with unstructured data, such as images or videos. For example, a model trained to recognize images of cars can be adapted to identify trucks, as many features, such as wheels, doors, and overall shape, are shared between the two categories.

Selecting a trained model

As a starting point, you need a pre-trained model, which is created by training on a large dataset with labeled examples. The model learns to recognize patterns and relationships in the data, allowing it to perform the intended task. In machine learning, this process involves layers that are interconnected and used to perform calculations. The more layers a model has, the more complex patterns it can recognize.

In transfer learning, you choose a model that has already successfully completed this process. It’s important to closely examine the source task of the existing model. The more similar it is to the new task, the easier it will be to adapt the model for the new application.

Reconfiguring and training the model

The second step is to configure the pre-trained model for the new task. There are generally two practical methods for this, and you can choose the one that best suits your needs.

In the first method, the last layer of the trained output model is replaced. This layer, also called the output layer, acts as the final classification unit, determining whether a given input matches the trained parameters. For example, this layer might decide whether an image represents a car. In many cases, you can remove this layer and replace it with a new one that is tailored to your specific application. In our example, the new layer would be designed to identify trucks instead of cars.

Alternatively, with transfer learning, it’s possible to freeze the previous parameters and add new layers instead. These new layers are specifically designed to align with the new task and are integrated into the model. The adapted model is then trained with a much smaller dataset containing the relevant examples. During this training, the model recognizes patterns and relationships while leveraging the knowledge gained from the original training.

Checking progress

The final step must be carried out in any case: You can only train the AI for the new task through conscientious monitoring and, if necessary, adjustments to the training material and possibly the new shifts. If the parameters are adjusted during training, the accuracy will also increase and the model will learn to meet the new requirements.

What are the different strategies?

There are different strategies for the use of transfer learning. Which one is right for you depends primarily on the desired purpose. These are some approaches:

• Feature extraction: In feature extraction, you use the pre-trained model to extract basic features, such as textures, while the new layers are designed to recognize more specific features. This approach is useful when the source and target tasks have significant overlap.
• Inductive transfer learning: In this case, the source and target domains are the same, but the tasks differ. This method allows new functions to be trained quicker, as the model can leverage knowledge from the source task to improve learning in the target task.
• Transductive transfer learning: In this strategy, the knowledge gained from the source task is transferred directly to specific instances of the new task, for example in order to be able to classify them better. This approach is promising if the source and target tasks have comparatively few similarities.
• Unsupervised transfer learning: Here, the source and target domains are similar, but the tasks differ. However, no labeled data is provided. Instead, the model learns the differences and similarities of the unlabeled data, enabling it to generalize and make predictions based on this information.
• Multitasking: In this approach, a model simultaneously performs multiple tasks that are not identical but are related to each other. This enables shared knowledge.
• Prediction: In this form of transfer learning, the model is supposed to fill in certain missing aspects of the data itself. For example, words within a sentence are predicted. The results are then improved through fine-tuning.
• Zero-shot and Few-shot: This is also a form of transfer learning in the field of generative AI, in which knowledge from a source is to be transferred to a target if there are only a few overlaps (Few-Shot) or no overlaps at all (Zero-Shot) between the two. The method is used when only very little training data is available.
• Disentanglement: For this approach, data is split into different factors. The model can then consider and manipulate style and content separately, for example.

What areas of application does transfer learning have?

There are numerous potential areas of application for transfer learning. The method offers significant savings in cost, time, and resources, making it highly advantageous. The most important applications to date include:

• image recognition
• speech recognition
• object localization
• diagnostics in the healthcare sector

In the future, however, transfer learning is likely to be applied in many other areas as well.