machine learning

Using Single Shot Detection to detect birds (Episode two)

In the previous article, I reached mAP 0.739 for VOC2007. After about two weeks, I add more tricks to reach mAP 0.740.
The most important trick is escalating the expand-scale of augmentation which is made from this patch. Increase the scale range could help the model to detect a smaller object. Moreover, to detect more hidden bird， I enhanced the RandomBrightness() and add ToGray() to let the model detect some black-white objects (I don’t man pandas). By using a confidence threshold of 0.4, I get these images which seems kind of promising:

I also tried learning rate warm up. But it can’t boost the performance. The explanation may be: warm up learning rate may cause overfit for the model.
After used and only used CUB-200-2011 dataset, I still got very bad performance for bird detection which seems like a mystery. I will go on my test to find out why.

Debugging the problem of ‘nan’ value in training

Previously, I was using CUB-200 dataset to train my object detection model. But after I used CUB-200-2011 dataset instead, the training loss became ‘nan’.

iter 10 || Loss: 17.9996 || timer: 0.2171 sec.
iter 20 || Loss: nan || timer: 0.2145 sec.
iter 30 || Loss: nan || timer: 0.2145 sec.
...

I tried to reduce the learning rate, change optimizer from SGD to Adam, and use different types of initializer for parameters. None of these solved the problem. Then I realized it would be a hard job to find the cause of the problem. Thus I began to print the value of ‘loss’, then the values of ‘loss_location’ and ‘loss_confidence’. Finally, I noticed that ‘loss_location’ firstly became ‘nan’ because of the value of $\hat{g}_j^w$ in the equation below (from paper) is ‘nan’:

‘loss_location’ from paper ‘SSD: Single Shot MultiBox Detector’

After checked the implementation in the ‘layers/box_utils.py’ of code:

def encode(matched, priors, variances):
...
    # dist b/t match center and prior's center
    g_cxcy = (matched[:, :2] + matched[:, 2:])/2 - priors[:, :2]
    # encode variance
    g_cxcy /= (variances[0] * priors[:, 2:])
    # match wh / prior wh
    g_wh = (matched[:, 2:] - matched[:, :2]) / priors[:, 2:]
    g_wh = torch.log(g_wh) / variances[1]
    # return target for smooth_l1_loss
    return torch.cat([g_cxcy, g_wh], 1)  # [num_priors,4]

I realized the (matched[:, 2:] – matched[:, :2]) has got a negative value which never happend when using CUB-200 dataset.
Now it’s time to carefully check the data pipeline for CUB-200-2011 dataset. I reviewed the bounding box file line by line and found out that the format of it is not (Xmin, Ymin, Xmax, Ymax), but (Xmin, Ymin, Width, Height)! Let’s show the images for an incorrect bounding box and correct one:

Parse bounding box by format (Xmin, Ymin, Xmax, Ymax) which is wrong
bird

Parse bounding box by format (Xmin, Ymin, Width, Height) which is correct

After changed the parsing method for the bounding boxes of CUB-200-2011 dataset, my training process runs successfully at last.
The lesson I learned from this problem is that dataset should be seriously reviewed before using.

Using Single Shot Detection to detect birds (Episode one)

SSD (Single Shot Detection) is a type of one-stage object detection neural network which uses multi-scale feature maps for detecting. I forked the code from ssd.pytorch, and added some small modifications for my bird-detection task.
I have tried some different types of rectifier function at first, such as ELU and RRelu. But they only reduced the mAP (mean Average Precision). I also tried to change the hyperparameters about augmentation. But it still didn’t work. Only after I enabled the batch normalization by this patch, the mAP has been boosted significantly (from 0.658 to 0.739).
The effect looks like:

Image 1.

Image 2.

But actually, we don’t need all types of annotated objects. We only need annotated bird images. Hence I change the code to train the model with only bird images in VOC2007 and VOC2012. Unexpectedly, the mAP is extremely low, and the model can’t even detect all 16 bird heads in the above [Image 2].
Why using bird images only will hurt the effect? There are might be two reasons: first, a too small number of bird images (only 1000 in VOC2007 and VOC2012); second, not enough augmentations.
To prove my hypothesis, I found CUB-200, a much larger dataset for bird images (about 6000). After training by this dataset, the effect is unsatisfied also: it can’t detect all three birds in [Image 1]. I need more experiments to find the reason.

Some tips about PyTorch and Python

1. ‘()’ may mean tuple or nothing.

len(("birds"))  # the inner '()' means nothing
len(("birds",))  # the inner '()' means tulple because of the comma

The result is:

5
1

2. Unlike TensorFlow’s static graph, PyTorch could run neural network just as the code. This means a lot of conveniences. The first advantage, we could print out any tensor in our program, no matter in prediction or training. Second, just adding ‘time.time()’ in code, could help us profiling every step of training.
3. Follow the example of NVIDIA’s apex, I wrote a prefetcher to let PyTorch loading data and computing parallelly. But in my test, the ‘data_prefetcher’ actually hurt the performance of training. The reason may be my model (VGG16) is not dense enough, thus computing cost less time than loading data.

Summaries for Kaggle’s competition ‘Histopathologic Cancer Detection’

Firstly, I want to thank for Alex Donchuk‘s advice in discussion of competition ‘Histopathologic Cancer Detection‘. His advice really helped me a lot.
1. Alex used the ‘SEE-ResNeXt50’. Instead, I used the standard ‘ResNeXt50’. Maybe this is the reason why my score ‘0.9716’ in public leaderboard is not as good as Alex’s. After the competition, I did spend some time to read the paper about ‘SE-ResNeXt50’. It’s really a simple and interesting idea about optimizing the architecture of the neural network. Maybe I can use this model on my next Kaggle competition.
2. In this competition, I split the training dataset into ten folds and train three different models on different train/eval splits. After ensembled these three models, it could get a nice score. Seems Bagging is a good method on practical application.
3. After training model to a ‘so far so good’ f1-score by using SGD with ReduceOnPlateu in Keras, I use this model as the ‘base model’ for following fine-tuning. By ensemble all high-score finetuning models, I eventually get the best score. This strategy comes from the Snapshot Ensembles.
4. By the way, ReduceOnPlateu is really useful when using SGD as the optimizer.

Experiencing TensorCore on RTX 2080 Ti

My colleague’s bare metal PC with three-fans-RTX-2080-Ti

My previous colleague Jian Mei has bought a new GPU – RTX 2080 Ti for training bird images of http://dongniao.net/. After he installed all the power supply and GPU on his computer, I began to run my MobileNetV2 model on it. Unsurprisingly, the performance doesn’t boost significantly: training speed increase from 60 samples/sec to about 150 samples/sec.
The most possible reason for the poor performance is the TensorCore.

To use the full power of TensorCore, or my colleague’s RTX 2080 TI, only following the guide <Mixed Precision Training> is not enough. I directly used the complete code example from Nvidia’s Github.
By using the ResNeXt50 model from the example, the TensorCore do promote the performance of training:

	Float32	Float16 (TensorCore)
Performance(samples/sec)	40	79

In the document of Nvidia, it reports 20 times performance enhancement by TensorCore. But in our RTX 2080 Ti, it only gained 2 times performance. Actually, the mainstream neural network models, such as ResNet/Densenet/Nasnet, couldn’t use up a highend GPU of Nvidia, since its too strong coumputation power for floating point. To produce the best results of TensorCore, I need to try more complicated and dense model continously.

Using XGBoost to predict large sparse data

For using XGBoost to predict, I wrote code like this:

test = load_npz('test.npz')
test = csr_matrix(test, dtype = 'float32')
xgb_model = xgb.Booster({'n_jobs': -1})
xgb_model.load_model(MODEL_PATH)
result = xgb_model.predict(test)

But it reported error:

  File "/usr/lib/python3.7/site-packages/scipy/sparse/base.py", line 689, in __getattr__
    raise AttributeError(attr + " not found")
AttributeError: feature_names not found

Seems csr_matrix in SciPy is not supported by XGBoost. Maybe I need to transfer sparse data to dense:

result = xgb_model.predict(test.todense())

But it still reported:

  File "/usr/lib/python3.7/site-packages/scipy/sparse/base.py", line 1187, in _process_toarray_args
    return np.zeros(self.shape, dtype=self.dtype, order=order)
MemoryError

The ‘test’ data is too big so it cann’t even be transfered to dense data!
XGBoost doesn’t support the sparse format, and my sparse data cannot be changed to dense. Then what should I do?
Actually, the solution is incredible simple — just use XGBoost’s DMatrix!

result = xgb_model.predict(xgb.DMatrix(test))

Some summaries for Kaggle’s competition ‘Humpback Whale Identification’

This time, I only spent one month on competition “Humpback Whale Identification”. But still, get a little step forward than previous competitions. Here are my summaries:
1. Do review ‘kernels’ in competition page, this will teach me a lot of information and new technology. By using Siamese Network rather than classic model, I eventually beat overfit problems. Thanks for suggestions from the ‘kernel’ page of competition.
2. Bravely use cutting-edge model, such as ResNeXt50 / Densenet121. They are more powerful and easy to use.
3. Do use fine-tuning. Don’t train model from scratch every time!
4. Ensemble learning is really powerful. I have used three different models to ensemble the final result.
There are also some tips for future challenge (may be correct, may be wrong):
1. albumentations is handful library for image augmentations
2. Cosine-decay-learning-rate performs worse than Exponential-decay-learning-rate
3. LeakyRelu doesn’t work significantly better than Relu
4. Bigger image size may not lead to higher accuracy

Using ResNeXt in Keras 2.2.4

To use ResNeXt50, I wrote my code as the API documentation for Keras:

keras.applications.resnext.ResNeXt50(...)

But it reported errors:

AttributeError: module 'keras.applications' has no attribute 'resnext'

That’s weird. The code doesn’t work as documentation said.
So I checked the code of Keras-2.2.4 (the version in my computer), and noticed that this version of code use ‘keras_applications’ instead of ‘keras.applications’.
Then I changed my code:

keras_applications.resnext.ResNeXt50((input_tensor = pinp, include_top = False, weights = 'imagenet')

But it reported another error:

Using TensorFlow backend.
Traceback (most recent call last):
  File "ktrain.py", line 292, in 
    main()
  File "ktrain.py", line 277, in main
    model, orig_model, branch_model, head_model = build_model(args)
  File "ktrain.py", line 210, in build_model
    branch_model = resnet_model(args, img_shape)
  File "ktrain.py", line 164, in resnet_model
    base_model = keras_applications.resnext.ResNeXt50(input_tensor = pinp, include_top = False, weights = 'imagenet')
  File "/usr/lib/python3.6/site-packages/keras_applications/resnet_common.py", line 555, in ResNeXt50
    **kwargs)
  File "/usr/lib/python3.6/site-packages/keras_applications/resnet_common.py", line 348, in ResNet
    data_format=backend.image_data_format(),
AttributeError: 'NoneType' object has no attribute 'image_data_format'

Witout choice, I had to check code of ‘/usr/lib/python3.6/site-packages/keras_applications/resnet_common.py’ too. Finally, I realise the ResNeXt50() function need three more arguments:

keras_applications.resnext.ResNeXt50(
        input_tensor = pinp, include_top = False, weights = 'imagenet',
        backend = keras.backend, layers = keras.layers, models = keras.models, utils = keras.utils)

Now the program could run ResNeXt50 model correctly. This github issue explained the detail: the ‘keras_applications’ could be used both for Keras and Tensorflow, so it needs to pass library details into model function.

Some tips about using Keras

1. How to use part of a model

    model = load_model(sys.argv[1], custom_objects = {'fmeasure': fmeasure})
    branch_model = model.get_layer(name = 'model_1')
    img_embed = Model(inputs = [branch_model.get_input_at(0)],
                      outputs = [branch_model.get_output_at(0)])

The ‘img_embed’ model is part of ‘branch_model’. We should realise that ‘Model()’ is a heavy cpu-cost function so it need to be create only once and then could be used many times.
2. How to save a model when using ‘multi_gpu_model’

orig_model = Model(x, y)
model = multi_gpu_model(orig_model, gpus = 2)
model.compile(optim, loss='binary_crossentropy', metrics = ['acc'])
....
model.fit_generator()
orig_model.save(PATH)

We should reserve original model. And only by using it, we can save the model to file.