The future of Android: An up-close look at Google Now on Tap

When it comes to the evolution of Android, change usually seems to fall into one of three main areas.

First and most immediately noticeable are the broad and sweeping visual overhauls — like what we’ve seen with Android Lollipop and its accompanying Material Design standard.

On a less obvious but no less significant level are the under-the-hood improvements, such as the efforts to boost battery life and performance in recent Android releases.

Then there’s the third category — the one that’s trickiest to pinpoint when it first appears. That category is the addition of a seemingly simple service or feature that has the potential to dramatically change the nature of the operating system and how we use it. Items in this category are rare, but they can end up being the most transformative of all.

Google Now on Tap is one of those items.

Google Now on Tap, introduced in the middle of a giant string of announcements at Google’s I/O developers’ conference last week, is an expanded version of the Google Now system present on Android since 2012. Back when Now first launched alongside the Android 4.1 Jelly Bean release, I referred to it as the platform’s “new secret weapon.”

Make no mistake about it: Now on Tap is no less pivotal. In fact, it makes the original Now launch look like little more than a stepping stone to the fleshed-out product we’re about to experience.

Here’s a detailed look at what Now on Tap is all about, how exactly it works, and how it’ll change the way you interact with your Android devices — including some new nuggets of info we hadn’t previously heard.

What is Google Now on Tap?

In essence, Google Now on Tap takes the power of Google Now and makes it a core part of the Android operating system. Instead of just being a standalone app that you open sometimes — and maybe see in your notifications once in a while — Now on Tap turns Google Now into a connective tissue that ties everything on your device together with potent Google intelligence.

How does Now on Tap work?

Starting with this fall’s Android “M” release, Now on Tap will pull up anytime you tap and hold your device’s Home button. Instead of being a full-screen app-like experience, it’ll appear in a card-like interface that sits atop whatever app you’re using — and it’ll provide contextual info related to what’s on your screen at any given moment.

Okay — so what exactly does Now on Tap do, then?

As Google explains it, tapping and holding the Home key tells Google Now to use the context on your screen to give you more information about whatever you’re seeing. That allows you to get extra knowledge about something without having to stop what you’re doing, go to Google or your Web browser, and manually search for what you want.

Real-world examples are really the best way to understand it. Say, for instance, you’re looking at a text message from a friend suggesting a particular restaurant for lunch. You could tap and hold your Home key, and Now on Tap would instantly bring up a card showing you info about the restaurant your buddy mentioned — ratings and reviews, a quick link for directions, and links to get even more info by jumping over to a relevant app like Yelp or OpenTable.

Google

Similarly, if you’re reading an email where a friend mentions a movie title, calling up Now on Tap would give you a card with reviews of the movie along with links to watch the trailer and get more info from apps like IMDB or Flixster.

Google

The system can also use contextual info on your screen to answer a question. If you’re listening to a song, for example, you could simply say: “Okay, Google, who’s the lead singer?” — and Now on Tap would pop up a card with the answer. Or if you’re looking at a place in Yelp, you could say: “Okay, Google, how far is it from here?” and Now on Tap would figure out what you mean and show you a map with the info.

Google

Oh dears! Does this mean Google is going to be watching everything I do?!

Nope — not at all. First of all, Google has confirmed to me that Now on Tap scans your screen only when you press and hold the Home button to activate the feature — never before, and never in any ongoing or continuous manner. It sees what you’re doing only if and when you tell it to look.

Beyond that, Now on Tap will be a fully opt-in feature. You’ll have to opt in to use Google Now, just like you do already, and you’ll then have to opt in a second time to use the system-wide Now on Tap system. If you don’t opt in, Now on Tap will never run.

Got it. But how exactly does Now on Tap do all this stuff, anyway?

According to Google, Now on Tap uses a system known as Android’s View Hierarchy. For the non-geek-speakers among us, it basically scans the words you’re seeing on your screen, identifies the most relevant terms, pastes them into a Google search, and then comes back with the info it thinks you need in a nicely structured card format.

And — theoretically, at least — it does all of that in a fraction of a second.

So Google Now on Tap can only “see” and interpret plain text?

For the moment, it’ll focus on text — but Google has hinted that the system could eventually scan and recognize images as well and provide you with contextual info about pictures you’re seeing. Think Google Goggles kind of magic — identifying famous works of art, landmarks, bottles of wine, and so forth. But that’s all speculative, and I’m told recognition beyond basic text is likely not going to be present right away.

Will Now on Tap work from every app, or will developers have to actively support it in order for it to function?

Now on Tap will work out of the box with any app — no special coding or compatibility required.

That being said, Google tells me developers will be able to take extra steps to mark up their apps and make them even more effective with the system. A game developer, for instance, might code in information that’d let Now on Tap know what level a player is in at any given moment. Then, if the player were to press and hold her device’s Home key, Now on Tap could pull up content and links relevant to that specific level in the partial-screen card.

In order for an app to appear as a link within a Now on Tap card, meanwhile, — like IMDB and Flixster did in our example a minute ago — its developer would need to have implemented app indexing so the system can find the relevant data and point users to the right place. App indexing is nothing new, though, and most apps that should support it already do.

All right, wise guy: If Now on Tap works everywhere, what’s to stop it from seeing sensitive info — like something from a banking app?

It’s a good question, and one I posed to Google as well. First of all, remember that Now on Tap sees what’s on your screen only when you activate it — so it’s not going to be suddenly snooping at your financial info and transmitting it via the local tinfoil network. 

But beyond that, a Google spokesperson tells me the company is looking into ways in which it could allow apps to mark certain types of info as sensitive and thus not compatible with or accessible to Now on Tap at all, under any circumstances — even if you were to go out of your way to activate the system while such info is on-screen. The feature is still being developed and the specifics aren’t yet concrete, but I’m told the issue is absolutely something engineers are considering.

What about data storage? Does Google maintain any type of history of the info it scans from my Now on Tap searches? 

Nope. Google assures me the data is processed and immediately discarded and that no sort of history is ever stored.

What happens if Now on Tap detects competing info on the screen — say, if I’m looking at an email that mentions a movie title and a restaurant name?

It’ll give you two cards — one for each relevant bit of info. Easy enough, eh?

What happens if I activate Now on Tap and it can’t find anything on my screen with an obvious card to accompany it?

I wondered the same thing (my, you ask excellent questions!). The answer, for now, is that Google is still thinking this through — and it’s still up in the air. The ladies and gents working on the feature figure folks will typically tap and hold the Home key when there’s an obvious path and area of info to be obtained. But we all know someone is gonna test the system in all sorts of weird scenarios just to see what happens (hey — why’s everyone looking at me all of a sudden?!).

My first thought was that Now on Tap could just show you plain search results in such instances, much like what happens when you ask Google Now or even Google itself a question for which it has no simple answer. But the Now architects may have a better solution: Right now, the team is thinking about whether there might be basic helpful things the system could show on a card when there’s no specific info connected to what you’re doing.

If all else fails, a photo of a tuba-playing puffin in a top hat might also be quite enchanting. (Just my two cents.)

Crikey! That’s a beautiful bird. But one last question: Is Google Now on Tap going to replace Google Now as we know it — or will the full-screen app-like interface also still exist?

This one has another big fat “TBD” stamp attached to it, so things may change as we get closer to the launch — but as of now, I’m told that tapping and holding the Home key is being positioned as the universal gesture to get to Now across the entire Android platform. And where you perform that gesture will determine what exactly transpires.

If you tap and hold the Home key from an app, you’ll get the new Now on Tap cards. If you do it from the home screen, you’ll likely get the full Now UI (similar to how you do today).

Consistency is the overarching goal, but different implementations may make sense for different scenarios.

The big picture

However all those details end up shaking out, one thing is clear: Google Now is about to become the glue that holds Android together. The post-Andy-Rubin Android has been all about making the operating system less of an island and more of a consistent and connected part of the grander Google ecosystem.

We’ve seen lots of little steps nudging the software in that direction. Google Now on Tap may be the thread that brings it all together — the powerful piece that transforms a house of apps into a search-centric universe with Google at its core.

UPDATE: It’s here! And, well, it’s almost amazing. See my hands-on analysis for more.