Blog

The latest news from Google AI

Using Deep Learning to Improve Usability on Mobile Devices

Tuesday, April 2, 2019

Posted by Yang Li, Research Scientist, Google AIgraphical user interfacesfalse affordancesCHI'19Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning

Predicting Tappability with Deep Learningthe blue color and underline of a linktypelocationsizecolorwordstypelocation

Heatmaps displaying the accuracy of tappable and non-tappable elements by location, where warmer colors represent areas of higher accuracy. Users labeled non-tappable elements more accurately towards the upper center of the interface, and tappable elements towards the bottom center of the interface.

sizecolorswordword deep neural networklocationwords typesizeconvolutional neural networkEvaluation of the Model

The scatterplot of the tappability probability predicted by the model (the Y axis) versus the consistency in the human user labels (the X axis) for each element in the consistency dataset.

precisionrecallAcknowledgementsThis research was a joint work of Amanda Swangson, summer intern at Google, and Yang Li, a Research Scientist in Deep Learning and Human Computer Interaction.