Blog

The latest from Google Research

Exploring Neural Networks with Activation Atlases

Wednesday, March 6, 2019

Posted by Shan Carter, Software Engineer, Google AIde factoOpenAIExploring Neural Networks with Activation Atlasesjupyter notebooks

A detail view of an activation atlas from one of the layers of the InceptionV1 vision classification network. It reveals many of the visual detectors that the network uses to classify images, such as different types of fruit-like textures, honeycomb patterns and fabric-like textures.

Inceptionv1ImageNetcarbonarasnorkelfrying panUMAP

Left: A randomized set of one million images is fed through the network, collecting one random spatial activation per image. Center: The activations are fed through UMAP to reduce them to two dimensions. They are then plotted, with similar activations placed near each other. Right: We then draw a grid, average the activations that fall within a cell, and run feature inversion on the averaged activation.

one

An overview of an activation atlas for one of the many layers (mixed4c) within Inception v1. It is about halfway through the network.

In this detail, we can see detectors for different types of leaves and plants.

Here we can see different detectors for water, lakes and sandbars.

Here we see different types of buildings and bridges.

In an early layer, mixed4a, there is a vague "mammalian" area.

By the next layer in the network, mixed4b, animals and people have been disentangled, with some fruit and food emerging in the middle.

By layer mixed4c these concepts are further refined and differentiated into small "peninsulas".

Left: This early layer is very nonspecific in comparison to the others. Center: By the middle layer, the images definitely resemble leaves, but they could be any type of plant. Right: By the last layer the images are very specific to cabbage, leaves curved into rounded balls.

You can see how sand and water are distinct concepts in a middle layer, mixed4c (left and center), both with strong attributions to the classification of "sandbar". Contrast this with a later layer (right), mixed5b, where the two ideas seem to be fused into one activation.

Here we can more clearly see what the network is focusing on to classify a "red fox". There are pointy ears, white snouts surrounded by red fur, and wooded or snowy backgrounds.

Here we can see the many different scales and angles of detectors for "tile roof".

For "ibex", we see detectors for horns and brown fur, but also environments where we might find such animals, like rocky hillsides.

Like the detectors for tile roof, "artichoke" also has many different sizes of detectors for the texture of an artichoke, but we also get some purple flower detectors. These are presumably detecting the blossoms of an artichoke plant.