Recognizing and Localizing Endangered Right Whales with Extremely Deep Neural Networks
In this post I’ll share my experience and explain my approach for the Kaggle Right Whale challenge. I managed to finish in 2nd place.
1. Background
Right whale is an endangered species with fewer than 500 left in the Atlantic Ocean. As part of an ongoing preservation effort, experienced marine scientists track them across the ocean to understand their behaviors, and monitor their health condition. The current process is quite time-consuming and laborious. It starts with photographing these whales during aerial surveys, selecting and importing the photos into a catalog, and finally the photos are compared against the known whales inside the catalog by trained researchers. Each right whale has unique callosity pattern on the whale head (see digram below). You can find more details at the competition page. The goal of the competition was to develop an automated process to identify the whales from aerial photos.
Image quality varies quite a bit because they were possibly taken in different years with different camera equipments. Note that some images were overexposed and some were underexposed. But in general, I found it very difficult to identify the whale myself with even using the highest quality images.
Here are 4 pairs of right whales, can you guess which ones are the same and which ones are not?
(Answers can be found from the URL of the images)
One thing that I didn’t notice from the images was how big these whales actually are. They can grow to 50 feet, weigh up to 170,000 lbs, and has a life span of typically 50 years. You can learn more fascinating facts about right whales from the NOAA website
2. Dataset
This dataset was special in 2 main ways from the perspective of machine learning.
2.1 Dataset distribution
It was non-straightforward to split the dataset into training and validation set. There were only 4237 images for 427 right whales. Most importantly the number of images per whales varies hugely, as can be seen from the below histogram. There were 24 whales that came with only 1 image in the dataset!
A reasonable local validation set was essential to evaluate how the model will perform on the testing set and estimation of public / private score on Kaggle.
The usual approach is to perform a stratified split so that the training and validation label distribution were similar. To handle the whales with single photo, we can either a) put those images just in the training set (overestimating); or b) just in the validation set (underestimate). Note that putting the whales with 1 image into validation set would result in the classifier not be able to predict those whales at all!
However due to a noob mistake, I ended up doing a random split. To makes things worse, different classifiers were trained with a different split. I noticed this issue about 3 weeks before the deadline, and I decided not to fix them because I thought it was too late.
Ironically none of model trained before that point ended up in the final submission. Lesson learned: it is never too late to makes thing right, just like in life.
2.2 Extremely fine-grained classification
Unlike many commonly cited classification tasks which is to classify images into different species (bird, tree leaves, dogs in ImageNet), this task is to classify images of the same species into different individuals. So the classifier must pick up the fine details of the callosity patterns regardless of image perspective and exposure etc.
Fortunately the academia has actually done immense work about recognition within a species – Homo sapiens. Realizing the similarity of recognition whale and human would become a source of many of my ideas.
3. Software and Hardware
Software
All code was written in Python. The neural networks were trained using Lasagne, Nolearn and cuDNN. scikit-image, pandas and scikit-learn were used for image processing, data processing and final ensembling respectively. I also made use of iPython / Jupyter notebook to sanity check my result and and ipywidgets to quickly browse through the images. Most of the charts in the blog post were made using matplotlib and seaborn.
Hardware
Most of the models were trained on GTX 980Ti and GTX 670 on a local Ubuntu machine. I made use of AWS EC2 near the very end of the competition, which will be explained further in Section 5.2
Who needs a heater when your machine is crunching numbers all the time!
4. Approaches
All my approaches were based on deep convolutional neural network (CNN), as I initially believed that human is no match to machine in extracting image feature. However it turned out that machines are not quite there yet. Understanding neural network performance bottleneck proved to be very important.
Below are 3 approaches I attempted in chronological order
4.1 Baseline Naive Approach
After deciding to participate in this competition, the first thing I did was to establish a baseline classifier with CNN.
Virtually no image processing is performed, except to resize them to 256x256 and stored as numpy memmap. I did not even perform zero mean unit variance normalization. Also the aspect ratio is not preserved during the resize. The target of the model was the whale name encoded to 0 and 447 as integer. The network architecture was based on OxfordNet.
OxfordNet (or VGGNet) was the winner of the 2014 ImageNet challenge. It contained a series of stacks of small 3x3 convolutional filters immediately followed by max-pooling. The network usually ended with a few stacks of fully connected layers. See original paper for details.
The network was trained with heavy data augmentation, including rotation, translation, shearing and scaling. I also added “brightness” augmentation to account for underexposed and overexposed images. I found that color permutation did not help, which made sense because there was not much color variation in the images.
Very leaky rectified linear unit (VLeakyReLU) was used for all of the models.
This naive approach yielded a validation score of just ~5.8 (logloss, lower the better) which was barely better than a random guess. This surprised me because I expected the network to be able to focus on the whale given the non-cluttered background. My hypothesis for the low score was that the whale labels did not provide a strong enough training signal in this relatively small dataset.
To prove my hypothesis, I looked at the saliency map of the neural network that is analogous to eye tracking. This was done by sliding a black box around the image and keeping track of the probability changes.
The saliency map suggested that the network was “looking at” the ocean waves instead of the whale head to identify the whale.
I further experimented with larger image sizes (e.g. 512x512) but found image size did not accuracy.
4.2 Bounding Box Localization
To help the classifier locating the whale head and hence improve the score, I added a localizer before the classifier.
A localization CNN took the original photos as input and output a bounding box around the whale head, and the classifier was fed the cropped image.
This is made possible thanks to the annotations by Vinh Nguyen posted to the competition forum.
4.2.1 Localizer
I treated the localization problem as a regression problem, so that the objective of the localizer CNN is to minimize the mean squared error (MSE) between the predicted and actual bounding box. The bounding boxes were represented by x, y, width and height and were normalized into (0, 1) by dividing with the image size.
Similar to the baseline model, the network structure is based on OxfordNet. Data augmentation was applied to the images as well. To calculate the bounding box of the transformed image, I created a boolean mask denoting the bounding box, applied the transformation to this mask, and extracted the normalized bounding box from the mask. See diagram below for details.