Finetune BERT on Google Colab: A Simple Tutorial

In this blog post, you will learn how you can finetune BERT on Google Colab and use the trained model for your NLP task. You will also learn how you can observe the performance of the model in the training phase.

Why BERT and Google Colab?

BERT is a language model and can thus be used for predicting the next word in a sentence. Furthermore, BERT can be used for automatic summarization, text classification and many more downstream tasks. Google Colab provides you with a cloud-based environment on which you can train your machine learning models on a GPU. The downside is that your data is uploaded to the Google cloud. Google Colab gives you the opportunity to finetune BERT.

Implementation

Here, I will show some code snippets which are relevant to the training part of BERT in Google Colab. Feel free to share your implementations and questions in the comment section. The first step would be to load BERT (of some of the flavours of BERT). Here, I will use the Dutch BERT (a.k.a. BERTje):

Other flavours of BERT can be found via HuggingFace. Then, you can load your datasets:

I stored mine in Google Drive since it is public data and used them in Google Colab by mounting my drive. After setting up your model, you can specify the training arguments:

And then, you can observe your model during training.

Observe your model

In order to observe your model, you can use Tensorboard in Google Colab which provides you with losses and metrics during the training. You can activate Tensorflow in Google Colab with the following code:

%load_ext tensorboard
%tensorboard --logdir logs

By executing the code, you get an embedded version of Tensorboard, which is quite useful! Here you can see an example of my training loss during the training phase:

How can I use my model?

You can download the latest model checkpoint to your local machine which you can then use for further inference. I would be interested in where you are using your model for. Please leave your use cases and questions in the comments down below. Have fun using your model!