Google AI open-sourced the code for researchers and developers for more contributions in this area
Google AI open-sourced mood board search, a new ML-powered tool for subjective or conceptual queries over images
Mood board search helps users define conceptual and subjective queries like peaceful and beautiful over images. Using deep learning in computer vision enabled engineers and researchers to provide different functionalities such as similar image search, object detection, tagging, etc. One of the main challenges in this area is how to define and query images with conceptual intent. Mood board search helps people train and personalise the deep learning model in a way that they see the world. The following snapshot shows how an artist sees the world by categorising images into different artistic concepts.
In mood board search, researchers used pre-trained computer vision models like GoogLeNet and MobileNet, and a machine learning approach called Concept Activation Vectors (CAVs).
CAV is a technique to measure how a trained model is sensitive to the concept presented by the user. The following picture shows how CAV or tested CAV is working.
As an example, a deep learning model trained to classify images as zebra or not zebra. We want to quantify how important the stripe concept is for the classifier. We can answer the question by simply running TCAV and getting the score. CAV is one of the general techniques for the explainability of deep learning models. As mentioned in the blog post :
In Mood Board Search, we use CAVs to find a model’s sensitivity to a mood board created by the user. In other words, each mood board creates a CAV — a direction in embedding space — and the tool searches an image dataset, surfacing images that are the closest match to the CAV. However, the tool takes it one step further by segmenting each image in the dataset in 15 different ways to uncover as many relevant compositions as possible.
Working with the mood board search GUI is straightforward. As it is explained in the blog post, “To get started, simply drag and drop a small number of images that represent the idea you want to convey. Mood Board Search returns the best results when the images share a consistent visual quality, so results are more likely to be relevant with mood boards that share visual similarities in colour, pattern, texture, or composition.”
Some of the design studios like morrama got excited about this tool and tweeted, “Amazing work, can’t wait to be able to play with it.”
Google AI open-sourced the code for researchers and developers for more contributions in this area. Also, there is an experimental app by design invention studio Nord Projects, which uses mood board search.
