{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "wZic934F6RrP" }, "source": [ "## DeepDream with VGG16" ] }, { "cell_type": "markdown", "metadata": { "jupyter": { "source_hidden": true }, "tags": [] }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": { "id": "8eiCisTQ6RrQ", "tags": [] }, "source": [ "The DeepDream algorithm is almost identical to the filter-visualization technique we explored for individual filters, in which we used gradient ascent to find an input image that maximizes the response of a particular filter within a layer. But there are three key differences:\n", "\n", "* DeepDream finds an image that maximizes the activation of an entire set of layers rather than a specific filter within a layer, thus mixing together visualizations of large numbers of features at once.\n", "* Instead of starting with a noisy blank image, we start with an existing image.\n", "* Input images are processed at different scales, called *octaves*." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "cRcjf87Q6RrS", "tags": [] }, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import tensorflow as tf\n", "import keras" ] }, { "cell_type": "markdown", "metadata": { "id": "qQ3PB2_s6RrS", "tags": [] }, "source": [ "We'll load just the pretrained VGG16 convolutional base, without the classification layers, so that we can feed images of any size into the network." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "N_RSENFa6RrS", "tags": [] }, "outputs": [], "source": [ "from keras.applications import VGG16\n", "vgg16 = VGG16(weights='imagenet', include_top=False)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "UoNqW3yc6RrT", "outputId": "8f327288-7a2c-4b15-d8db-0faead5af8d8", "scrolled": true, "tags": [] }, "outputs": [], "source": [ "vgg16.summary(print_fn=print)" ] }, { "cell_type": "markdown", "metadata": { "id": "6NIzGnzr6RrU", "jp-MarkdownHeadingCollapsed": true }, "source": [ "### Utility Functions for Processing Images" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "xypbNItz6RrU", "tags": [] }, "outputs": [], "source": [ "def load_image_from_server(filename):\n", " url = \"https://science.slc.edu/jmarshall/bioai/images/\" + filename\n", " path_to_image = keras.utils.get_file(filename, origin=url)\n", " print(\"Image saved as \" + path_to_image)\n", " jpeg_img = keras.utils.load_img(path_to_image)\n", " return np.array(jpeg_img)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 269 }, "id": "AiYHnmiM6RrU", "outputId": "ff3eb317-bea4-4370-fd4c-455085b8e226", "tags": [] }, "outputs": [], "source": [ "elephants = load_image_from_server('elephants.jpg')\n", "jellyfish = load_image_from_server('jellyfish.jpg')\n", "flamingos = load_image_from_server('flamingos.jpg')\n", "tiger = load_image_from_server('tiger.jpg')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plt.imshow(elephants);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `pre_process` function converts an image of ints in the range [0, 255] into a single-image batch of floats in the range [-1.0, 1.0]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "54arYaBE6RrV", "tags": [] }, "outputs": [], "source": [ "from keras.applications.inception_v3 import preprocess_input\n", "\n", "def pre_process(image):\n", " return np.array([preprocess_input(image)])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "img = elephants" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "print(img.shape, img.dtype, np.min(img), np.max(img))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "img2 = pre_process(img)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "print(img2.shape, img2.dtype, np.min(img2), np.max(img2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `post_process` function converts a single-image batch of floats into a displayable image of ints in the range [0, 255]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "1diaBdWM6RrX", "tags": [] }, "outputs": [], "source": [ "def post_process(image_batch):\n", " batch_size, height, width, depth = image_batch.shape\n", " image = image_batch.reshape((height, width, depth)) # reshapes batch as a single image\n", " image = (image / 2 + 0.5) * 255 # rescales the range [-1.0, 1.0] to [0.0, 255.0]\n", " image = image.clip(0, 255).astype('uint8') # converts all values to ints in the range [0, 255]\n", " return image" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "img3 = post_process(img2)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "print(img3.shape, img3.dtype, np.min(img3), np.max(img3))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "plt.imshow(img3);" ] }, { "cell_type": "markdown", "metadata": { "id": "z_ZP8x0x6RrY", "jp-MarkdownHeadingCollapsed": true }, "source": [ "### Gradient Ascent in Input Image Space" ] }, { "cell_type": "markdown", "metadata": { "id": "O_rb3jCX6RrZ", "tags": [] }, "source": [ "This dictionary specifies which layers of the VGG16 network will be used to construct the dream image, as well as their relative weightings:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "QV7KBnms6RrZ", "tags": [] }, "outputs": [], "source": [ "layer_contributions = {\n", " 'block1_conv1': 1.0,\n", " 'block2_conv1': 1.0,\n", " 'block3_conv1': 2.0,\n", " 'block4_conv1': 3.0,\n", " 'block5_conv1': 1.0,\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is the output tensor from the `block1_conv1` layer, which has 64 filters:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "vgg16.get_layer('block1_conv1').output" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's build a dictionary that maps the layer names in `layer_contributions` to their associated output feature tensors:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "layer_features = {}\n", "for name in layer_contributions:\n", " output_tensor = vgg16.get_layer(name).output\n", " layer_features[name] = output_tensor" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or, more succinctly:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "layer_features = {name: vgg16.get_layer(name).output for name in layer_contributions}" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "layer_features" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `feature_extractor` returns the activation values of each of the above layers in response to an input image, as a single dictionary:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "vgg16.input" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "feature_extractor = keras.Model(inputs=vgg16.input, outputs=layer_features)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "feature_extractor(vgg16.input)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is an example of applying our feature extractor to an input image. The input must be a **batch** containing a single image:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "elephants.shape" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "image_batch = np.array([elephants])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "image_batch.shape" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "layer_features = feature_extractor(image_batch)\n", "activations = layer_features['block2_conv1']" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "activations.shape" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(np.min(activations), np.max(activations))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "keras.ops.mean(keras.ops.square(activations))" ] }, { "cell_type": "markdown", "metadata": { "id": "zz2GnHOZ6RrZ", "tags": [] }, "source": [ "Here is the function we will optimize, which assumes that `layer_contributions` has been defined:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "jupyter": { "source_hidden": true }, "tags": [] }, "outputs": [], "source": [ "def compute_response(image_batch, feature_extractor): # image_batch contains a single image\n", " layer_features = feature_extractor(image_batch)\n", " response = tf.zeros(shape=())\n", " for layer_name, coeff in layer_contributions.items():\n", " activations = layer_features[layer_name]\n", " response += coeff * keras.ops.mean(keras.ops.square(activations))\n", " return response" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This function returns the combined ***response*** of the layers in ``layer_contributions`` as a scalar tensor. This is similar to a *loss* value, except that instead of minimizing the loss, the gradient ascent process will seek to maximize the response." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "compute_response(vgg16.input, feature_extractor)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `gradient_ascent_step` function computes the gradient of the response value with respect to the input image (represented as a batch) and then updates the image. We'll use the `@tf.function` decorator to compile the function to make it faster:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "jupyter": { "source_hidden": true }, "tags": [] }, "outputs": [], "source": [ "@tf.function\n", "def gradient_ascent_step(image_batch, feature_extractor, step_size):\n", " with tf.GradientTape() as tape:\n", " tape.watch(image_batch)\n", " response = compute_response(image_batch, feature_extractor)\n", " grads = tape.gradient(response, image_batch) # grads = d_response/d_image_batch\n", " # normalize the gradient (safely)\n", " grads /= keras.ops.maximum(1e-7, keras.ops.mean(keras.ops.abs(grads)))\n", " image_batch += step_size * grads\n", " return response, image_batch" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This function performs gradient ascent for the specified number of iterations:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "jupyter": { "source_hidden": true }, "tags": [] }, "outputs": [], "source": [ "def gradient_ascent_loop(image_batch, feature_extractor, cycles, step_size, quiet):\n", " for i in range(1, cycles+1):\n", " response, image_batch = gradient_ascent_step(image_batch, feature_extractor, step_size)\n", " if not quiet:\n", " print(f\"...value on cycle {i}: {response:.2f}\")\n", " return image_batch" ] }, { "cell_type": "markdown", "metadata": { "id": "uSM3Kt4v6Rra", "jp-MarkdownHeadingCollapsed": true }, "source": [ "### The DeepDream Algorithm" ] }, { "cell_type": "markdown", "metadata": { "id": "IvRW75zZ6Rra", "tags": [] }, "source": [ "" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "jupyter": { "source_hidden": true }, "tags": [] }, "outputs": [], "source": [ "def size_sequence(height, width, num_octaves=3, scale_factor=1.4):\n", " sizes = [(height, width)]\n", " for i in range(1, num_octaves):\n", " height, width = int(height/scale_factor), int(width/scale_factor)\n", " sizes.append((height, width))\n", " return sizes[::-1] # reverses the list" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "size_sequence(467, 699)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "size_sequence(350, 350)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "size_sequence(350, 350, num_octaves=5)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "size_sequence(350, 350, num_octaves=5, scale_factor=1.6)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "size_sequence(350, 350, num_octaves=5, scale_factor=2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To avoid losing a lot of image detail after each successive scale-up (resulting in increasingly blurry or pixelated images), we can use a simple trick: after each scale-up, we’ll re-inject the lost details back into the image, which is possible because we know what the original image should look like at the larger scale. Given a small image size *S* and a larger image size *L*, we can compute the difference between the original image resized to size *L* and the original resized to size *S*. This difference quantifies the details lost when going from *S* to *L*." ] }, { "cell_type": "markdown", "metadata": { "id": "HMEOvE276Rrc", "tags": [] }, "source": [ "#### Parameters\n", "
\n",
    "cycles          number of iterations of gradient ascent\n",
    "step_size       gradient ascent step size\n",
    "num_octaves     number of dream/upscale cycles\n",
    "scale_factor    upscale factor\n",
    "quiet           controls level of output\n",
    "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def deep_dream(image, cycles=20, step_size=0.01, num_octaves=3, scale_factor=1.4, quiet=False):\n", " if not(type(image) == np.ndarray and image.dtype == 'uint8'):\n", " print(\"Image must be an int array in the range [0,255]\")\n", " return\n", "\n", " # build the feature extractor based on the layers specified in layer_contributions\n", "\n", " # create the sequence of image sizes to process\n", "\n", " # create a sequence of rescaled images from the original image\n", "\n", " # start with a copy of the first rescaled image in the sequence\n", " \n", " # process each octave: dream, upscale, reinject lost details...\n", "\n", " # display the dream\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "layer_contributions" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "deep_dream(elephants)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "layer_contributions = {\n", " 'block1_conv1': 1.0,\n", " 'block2_conv1': 2.0,\n", "}\n", "deep_dream(elephants, cycles=15)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "layer_contributions = {\n", " 'block2_conv1': 1.0,\n", " 'block3_conv1': 2.0,\n", "}\n", "deep_dream(elephants, cycles=15, quiet=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "id": "2AiVfhxR6Rre", "outputId": "52489b52-c79a-4d5e-dac2-b7bea994865b", "tags": [] }, "outputs": [], "source": [ "layer_contributions = {\n", " 'block1_conv1': 1.0,\n", " 'block2_conv1': 1.0,\n", " 'block3_conv1': 2.0,\n", " 'block4_conv1': 3.0,\n", " 'block5_conv1': 1.0,\n", "}\n", "deep_dream(elephants, cycles=5, quiet=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "accelerator": "GPU", "colab": { "collapsed_sections": [], "name": "DeepDream.ipynb", "provenance": [] }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.12" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }