{ "cells": [ { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "## Fine-Tuning a Pretrained VGG16 Convolutional Network" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "\n", "\n", "The VGG16 network, developed by the Visual Geometry Group at Oxford, was trained on the large-scale ImageNet dataset, consisting of 1.4 million labeled images from 1,000 different categories. Most of these images are of animals or other everyday objects, including many different breeds of cats and dogs.\n", "\n", "[K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition (2014)](https://arxiv.org/abs/1409.1556) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Preliminaries" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import random\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "plt.rcParams[\"figure.figsize\"] = (2.5,2.5) # set default figure size" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "# define some utility functions\n", "\n", "# plots validation data if available\n", "def plot_history(history):\n", " loss_values = history.history['loss']\n", " accuracy_values = history.history['accuracy']\n", " validation = 'val_loss' in history.history\n", " if validation:\n", " val_loss_values = history.history['val_loss']\n", " val_accuracy_values = history.history['val_accuracy']\n", " epoch_nums = range(1, len(loss_values)+1)\n", " plt.figure(figsize=(12,4)) # width, height in inches\n", " plt.subplot(1, 2, 1)\n", " if validation:\n", " plt.plot(epoch_nums, loss_values, 'r', label=\"Training loss\")\n", " plt.plot(epoch_nums, val_loss_values, 'r--', label=\"Validation loss\")\n", " plt.title(\"Training/validation loss\")\n", " plt.legend()\n", " else:\n", " plt.plot(epoch_nums, loss_values, 'r', label=\"Training loss\")\n", " plt.title(\"Training loss\")\n", " plt.xlabel(\"Epochs\")\n", " plt.ylabel(\"Loss\")\n", " plt.subplot(1, 2, 2)\n", " if validation:\n", " plt.plot(epoch_nums, accuracy_values, 'b', label='Training accuracy')\n", " plt.plot(epoch_nums, val_accuracy_values, 'b--', label='Validation accuracy')\n", " plt.title(\"Training/validation accuracy\")\n", " plt.legend()\n", " else:\n", " plt.plot(epoch_nums, accuracy_values, 'b', label='Training accuracy')\n", " plt.title(\"Training accuracy\")\n", " plt.xlabel(\"Epochs\")\n", " plt.ylabel(\"Accuracy\")\n", " plt.ylim(0, 1)\n", " plt.show()\n", "\n", "def show_random_image(images):\n", " i = random.randrange(len(images))\n", " print(f\"image {i}\")\n", " plt.imshow(images[i])\n", "\n", "def show_random_selection(images):\n", " indices = range(len(images))\n", " plt.figure(figsize=(15,12)) # (width, height) in inches\n", " rows, columns = 5, 6\n", " for k in range(1, columns*rows+1):\n", " i = random.choice(indices)\n", " plt.subplot(rows, columns, k)\n", " plt.axis('off')\n", " plt.imshow(images[i])\n", "\n", "from keras.models import Model\n", "from keras.applications.imagenet_utils import preprocess_input\n", "\n", "def show_features(network, image, layer_name, features=range(20), cmap='gray', cols=5):\n", " # features: a number like 0 or a sequence like [0,2,4] or range(10)\n", " if network.name == 'vgg16':\n", " vgg16 = network\n", " else:\n", " # extract embedded VGG16 network\n", " vgg16 = network.get_layer('vgg16')\n", " vgg16_layer_names = [layer.name for layer in vgg16.layers]\n", " if layer_name not in vgg16_layer_names:\n", " print(f\"No such VGG16 layer: {layer_name}\")\n", " return\n", " # generate feature maps for layer_name\n", " input_tensor = vgg16.input\n", " output_tensor = vgg16.get_layer(layer_name).output\n", " model = Model(inputs=input_tensor, outputs=output_tensor)\n", " input_batch = np.array([preprocess_input(image)])\n", " output_batch = np.array(model(input_batch))\n", " output = output_batch[0]\n", " h, w, d = output.shape\n", " # display feature maps\n", " if type(features) is int:\n", " features = [features]\n", " rows = len(features) // cols\n", " if len(features) > rows*cols:\n", " rows += 1\n", " fig = plt.figure(figsize=(3*cols,3*rows)) # (width, height) in inches\n", " k = 1\n", " for feature in features:\n", " if 0 <= feature < d:\n", " fig.add_subplot(rows, cols, k)\n", " k += 1\n", " plt.imshow(output[:,:,feature], cmap=cmap)\n", " plt.title(f\"feature {feature}\")\n", " plt.axis('off')\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Building and Training a New Network Based on VGG16\n", "\n", "The advantage of using a large pretrained network like VGG16 is that it has already been trained on a large dataset of real-world images (> 1.4 million in the case of ImageNet). The network's convolutional layers have learned to recognize many low-level visual features shared by almost all visual scenes, such as edges separating light and dark regions, line segments of specific orientations, corners, intersections, visual textures, etc.\n", "\n", "These general elements are common to many types of visual recognition tasks, such as distinguishing cats from dogs. Rather than re-training a large network from scratch on each new visual recognition task, a better approach is to re-use a pretrained network by harnessing and \"fine tuning\" its convolutional layers according to the specific task at hand.\n", "\n", "We will build a new network to classify cats vs. dogs, based on the pretrained VGG16 network. For this task, we will use a larger dataset of 2000 color images of cats and dogs of size 150 × 150. First, let's download the data:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#!curl -O science.slc.edu/jmarshall/bioai/data/cats_dogs_2000_150x150.npz" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "npz = np.load('cats_dogs_2000_150x150.npz')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "list(npz.keys())" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "images, labels = npz['images'], npz['labels']" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "images.shape" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(images.dtype, images.min(), images.max())" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "labels.shape" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "labels" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "labels[950:1050]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cats, dogs = images[0:1000], images[1000:2000]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plt.imshow(cats[0]);" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plt.imshow(dogs[0]);" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "show_random_selection(dogs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The Convolutional Base\n", "\n", "We will use only the VGG16's convolutional layers. The parameter `include_top=False` omits the classification layers of the VGG16 network. Because the number of convolutional weights in a ConvNet does not depend on the input image size, our input images are not restricted to being 224 × 224. The output shape of the VGG16 module is unspecified, as are the sizes of the convolutional and pooling layers. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "from keras.applications import VGG16" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "vgg16_base = VGG16(weights='imagenet', include_top=False)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "vgg16_base.summary(print_fn=print)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "show_features(vgg16_base, dogs[0], 'block1_conv1', features=range(10))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "show_features(vgg16_base, dogs[0], 'block5_pool', features=range(10))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will fine-tune the last few convolutional layers of the VGG16 base while simultaneously training new classification layers on top of it for classifying cats and dogs. We will follow these steps:\n", "\n", "1. Add the classification network on top of the pretrained convolutional base.\n", "\n", "2. Freeze the convolutional base weights.\n", "\n", "3. Train the classification weights.\n", "\n", "4. Unfreeze some of the convolutional layers.\n", "\n", "5. Jointly train the unfrozen convolutional layers and the classification layers." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Step 0: Prepare the data\n", "\n", "The first 1000 images are of cats (label=0), and the last 1000 are of dogs (label=1)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "images.shape" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "labels[950:1050]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "np.random.permutation(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Randomly shuffle the data and divide it into training, validation, and test sets." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "shuffled_indices = np.random.permutation(2000)\n", "images = images[shuffled_indices]\n", "labels = labels[shuffled_indices]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "labels[950:1050]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 1000 training images\n", "train_images = images[0:1000]\n", "train_targets = labels[0:1000].astype('float32') # targets must be probabilities\n", "\n", "# 500 validation images\n", "val_images = images[1000:1500]\n", "val_targets = labels[1000:1500].astype('float32')\n", "\n", "# 500 test images\n", "test_images = images[1500:2000]\n", "test_targets = labels[1500:2000].astype('float32')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Step 1: Add the classification network on top of the pretrained VGG16 base\n", "\n", "We will use a fully-connected hidden layer of 256 rectified linear (ReLU) units, followed by a \"Dropout\" layer to improve the network's generalization ability. Since our network will be doing two-way classification, we will use a single sigmoid output unit." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from keras.models import Sequential\n", "from keras.layers import Input, Lambda, Flatten, Dense, Dropout\n", "from keras.applications.imagenet_utils import preprocess_input\n", "\n", "def build_network():\n", " network = Sequential()\n", " network.add(Input(shape=(150,150,3)))\n", " network.add(Lambda(preprocess_input))\n", " network.add(VGG16(weights='imagenet', include_top=False))\n", " network.add(Flatten())\n", " network.add(Dense(256, activation='relu'))\n", " network.add(Dropout(0.5)) # improves the performance of the network\n", " network.add(Dense(1, activation='sigmoid'))\n", " return network" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "network = build_network()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "network.summary(print_fn=print)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Dropout layers\n", "\n", "Dropout layers introduce stochastic noise into the training process by temporarily \"zeroing-out\" a randomly-chosen subset of the weights on each training cycle. The particular set of zeroed-out weights changes on each cycle. Once training is finished, dropout no longer occurs, and the full set of trained weights are used to compute the network's response to each input. The dropout parameter (0.5 in the network above) specifies the fraction of network weights that are zeroed-out on each training step.\n", "\n", "Srivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov, \"Dropout: A Simple Way to Prevent Neural Networks from Overfitting\", *Journal of Machine Learning Research* **15**, 2014, pp. 1929-1958.\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Step 2: Freeze the convolutional base weights" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "vgg16_base = network.layers[1]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "vgg16_base.trainable" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "vgg16_base.summary(print_fn=print) # note the number of trainable parameters listed at the bottom" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "vgg16_base.trainable = False" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "vgg16_base.trainable" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "vgg16_base.summary(print_fn=print) # note the number of trainable parameters listed at the bottom" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "network.summary(print_fn=print) # note the number of trainable parameters listed at the bottom" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Step 3: Train the classification weights" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "network.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "history = network.fit(train_images, train_targets,\n", " validation_data=(val_images, val_targets),\n", " epochs=5)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_history(history)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "network.evaluate(val_images, val_targets)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "network.evaluate(test_images, test_targets)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Step 4: Unfreeze some of the convolutional layers of the VGG16 base\n", "\n", "We will unfreeze the weights in the layers named `block5_conv1`, `block5_conv2`, and `block5_conv3`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "vgg16_base.trainable = True" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for layer in vgg16_base.layers:\n", " if layer.name in ('block5_conv1', 'block5_conv2', 'block5_conv3'):\n", " layer.trainable = True\n", " else:\n", " layer.trainable = False" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**IMPORTANT:** Always re-compile the network after freezing or unfreezing weights!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "network.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for layer in vgg16_base.layers:\n", " print(f'{layer.name}: {layer.trainable}')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "vgg16_base.summary(print_fn=print) # note the number of trainable parameters listed at the bottom" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(2359808*3, \"trainable parameters in the VGG16 base\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "network.summary(print_fn=print)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(7079424 + 2097408 + 257, \"trainable parameters in the whole network\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Step 5: Jointly train the unfrozen convolutional layers and the classification layers" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from keras.optimizers import Adam\n", "\n", "network.compile(loss='binary_crossentropy',\n", " optimizer=Adam(learning_rate=0.00001), # use a smaller learning rate\n", " metrics=['accuracy'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "history = network.fit(train_images, train_targets,\n", " validation_data=(val_images, val_targets),\n", " epochs=3)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "network.evaluate(val_images, val_targets)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "network.evaluate(test_images, test_targets)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Pick a new image at random and classify it:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "new_image = random.choice(test_images)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plt.imshow(new_image);" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "new_output = network.predict(np.array([new_image])) # a batch with one image" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "new_output.shape" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(new_output)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def cat_or_dog(image):\n", " plt.imshow(image)\n", " output = network.predict(np.array([image])) # a batch with one image\n", " value = output[0][0]\n", " print('MEOW!') if value < 0.5 else print('WOOF!')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cat_or_dog(new_image)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def cat_or_dog(image):\n", " plt.axis('off') # added\n", " plt.imshow(image)\n", " output = network.predict(np.array([image]), verbose=0) # added verbose=0\n", " value = output[0][0]\n", " print('MEOW!') if value < 0.5 else print('WOOF!')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cat_or_dog(new_image)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cat_or_dog(random.choice(test_images))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "image = random.choice(test_images)\n", "plt.imshow(image);\n", "show_features(network, image, 'block1_conv1')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "show_features(network, image, 'block3_conv1')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.12" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }