Data Science

A lot of computations are easily parallelizable.

Training deep learning models requires a lot of data. It is not uncommon to train models on millions of images or gigabytes of text data to achieve good results. Most real-world problems don't have that amount of labeled data ready, and not all companies or individuals who want to train a model can afford to hire people to label data for them.

For many years this has been very challenging. Fortunately, it has been solved for image based models a couple of years ago and recently also for NLP. One approach that helps train models with limited labeled data is called transfer learning.

The idea is that once a model is trained on a large dataset for a specific task (e.g., classifying houses vs. planes), the model has learned certain features of the data that can be reused for another task. Such features could be how to detect edges or textures in images. If these features are useful for another task, then we can train the model on new data without requiring as many labels as if we were training it from scratch.

For transfer learning in NLP Jeremy Howard and Sebastian Ruder came up with a similar approach called ULMFiT (Universal Language Model Fine-tuning for Text Classification) for texts. The central theme of the approach is language modeling.

In language modeling the goal is to predict the next word based on the previous word in a text.

The steps include:

1. Language model (wiki): A language model is trained on a large dataset. Wikipedia is a common choice for this task as it includes many topics, and the text is of high quality. This step usually takes the most time on the order of days. In this step, the model learns the general structure of language.

2. Language model (domain): The language model trained on Wikipedia might be missing some aspects of the domain we are interested in. If we want to do sentiment classification, Wikipedia does not offer much insight since Wikipedia articles are generally of neutral sentiment. Therefore, we continue training the language model on the text we are interested in. This step still takes several hours.

3. Classifier (domain): Now that the language model works well on the text we are interested in, it is time to build a classifier. We do this by adapting the output of the network to yield classes instead of words. This step only takes a couple of minutes to an hour to complete.

The power of this approach is that you only need little labeled data for the last step and only need to go through the expensive first step once. Even the second step can be reused on the dataset if you, for example, build a sentiment classifier and additionally a topic classifier. This can be done in minutes and allows us to achieve great results with little time and resources.

untar_data()

bd = Batch size

The batch size bs specifies ho many samples the modelis optimised for at each step.

xxunk = A word that is not in the dictionary

xxbos and xxeos = Identify the beginning and the end of a string. So if the first entry in the encoding vector is 1 this means that the token is xxunk. If the third entry is 1 then the token is xxbos.

xxmaj = Signifies that the first letter of the following word is capitalised.

A Deep Learning text classifier model named long shoort-term memory network. This is a neural network with a feedback loop. That means that when fed a sequence of tokens, it feeds back its output for the next prediction. With this the model has a mchanism remembering the past inputz. This is especially useful when dealing with sequential data such as texts, where the sequence of words and characters carries important meaning.

Loading the on Wikipedia pretrained language model likely to the fastai library. The dataset it will be trained on is passed to in the end.

The learning rate

With the lr_find() function, we can explore how the loss function behaves with regards to the value of the learning rate:

learn.lr_find()

In the graph we want to find the spot where the loss function decreases the steepest with the largest learning rate.

To predict the next word based on a sequence of words.

Generate text examples (in this case movie reviews.

from fastai.vision import *

img_dl = ImageDownloader(data_path, dataset name)

Creating a new class for the image downloader.

Look at examples of the downladed images with its classes.

Downloading image dataset.

To load an image dataset according to fastai library.

We apply a few tricks when we load the data:

1. A trick often used in image classification is data augmentation. This means one creates more data by manipulating existing data. For images this means that the images can be rotated, flipped cropped etc. This generally improves the performance of the classifier and is setup with the get_transforms function.
2. We split the data 80/20 into train and validation data.
3. We crop the images to a size of 224 pixels.

tfms = get_transforms()
data = ImageDataBunch.from_folder(path, ds_tfms=tfms, valid_pct=0.2, size=224)

Peek at the data:

data.show_batch(rows=3, figsize=(8, 8)) 

1. Load a pretrained model (e.g. resnet34, resnet50, etc.)

2. Find the optimal learning rate

3. Fit the head of the network

4. Unfreeze all layers and fine-tune

5. Evaluate the results

It is a convolutinal neural network with 3 layers. There are also larger networks with up to 150 layers but this model usually takes the least effort to train  on and get good results.

Load pretrained image model

With the confusin matrix

fast.ai's ImageCleaner for data cleaning.

• precision = TP / (TP+FP)
• recall = TP / (TP+FN)

Note: Very hard (impossible) to get a model with perfect precision AND recall.

Principal Compnent Analysis.

PCA aims at finding new axis that contain most of the variance in the dataset.

Note: PCA is sensitive to the scaling of the features --> larger scaling <--> larger variance

The study of shapes

One way to study shapes is to look at holes: The whole form of a figure does not matter as long as there is a hole in it, it is the same.

Count-encodings are the sum of the word encodings.

Advantage: some information on the sewuence is conserved

Disadvantage: Vocabulary can get very large

In deep learning many neuron layers are stacked on top of each other -> y becomes X of next layer

Backpropagatin uses the chain rule to calculate the gradients over the whole network.

• Feedforward networks: Information flows forward only
• Recurrent neural network: Neuron has a feedback loop
• Convolutional neural network: Network "scans" through input instead of consuming all input at once