Blog

The latest from Google Research

Building SMILY, a Human-Centric, Similar-Image Search Tool for Pathology

Friday, July 19, 2019

Posted by Narayan Hegde, Software Engineer, Google Health and Carrie J. Cai, Research Scientist, Google Researchdiabetic eye diseasemetastatic breast cancerwhathowanatomic pathologycontent-based image retrievalreverse image searchGoogle Imagessimilar image searchSimilar Image Search for Histopathology: SMILYNature Partner Journal (npj) Digital Medicine“Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making” herehonorable mention2019 ACM CHI Conference on Human Factors in Computing SystemsSMILY DesignembeddingThe Cancer Genome Atlas

Schematic of the steps in building the SMILY database and the process by which input image patches are used to perform the similar image search.

Example of selecting a small region in a slide and using SMILY to retrieve similar images. SMILY efficiently searches a database of billions of cropped images in a few seconds. Because pathology images can be viewed at different magnifications (zoom levels), SMILY automatically searches images at the same magnification as the input image.

Second example of using SMILY, this time searching for a lobular carcinoma, a specific subtype of breast cancer.

Refinement tools for SMILYlooks similariterative diagnosissecond paperrefine-by-region refine-by-examplerefine-by-concept

beyond

AcknowledgementsThis work would not have been possible without Jason D. Hipp, Yun Liu, Emily Reif, Daniel Smilkov, Michael Terry, Craig H. Mermel, Martin C. Stumpe and members of Google Health and PAIR. Preprints of the two papers are available here and here.