DANNY SULLIVAN: It’s a real pleasure to be here and doing a panel for the Churchill Club We’ve been talking about doing some kind of a session about Search for just over about a year And when they got closer and we came up with some ideas– this panel really came about from Stephen Levy’s excellent book In the Plex If you haven’t read it, my short review is go read it Not now, but maybe in about an hour and a half And engineer, David Bailey, is quoted as being assigned to work in an office at Google And he’s supposed to be working in this office with Amit Singal, Ben Gomes, and Matt Cutts And he says, it’s definitely the cool kids office And that really struck me when I read that, because I’d been to this office many times over the years I typically would go into Google and have a day’s worth of briefings, and then at the end of the day I’d usually end up talking to Matt on webmaster issues And we would be in his office and it would kind of turn into this Search bowl session, which is a lot of fun to hear all of them interacting about, well maybe the results should be this way or this particular issue So this kind of session is kind of bringing that office to life and some of the things that go on with it If I had to describe the areas that each of these men oversee, when it comes to Search, Amit would be the brains, and Ben would be the looks, and Matt would be the brawn [LAUGHTER] So I hope you’re happy with your choices in life [LAUGHTER] Amit oversees the ranking algorithm at Google How Google decides what content should be shown in response to searches that happen Ben overseas features that help you search better, as well as the user interface look that lets you interact with Google Search And Matt’s the bouncer He’s the chief of police The person in charge of keeping the people who would spam and pollute and harm Google Search results and bring disorder there under control And they’re each going to tell you a little bit more about themselves We’ll start with Amit because he’s the baby of the group, having been at Google for only ten and a half years AMIT SINGAL: Thank you, Danny So what Danny’s referring to is the fact that of the three Googlers you are seeing up here, I joined Google last These two were already there, and I’ve been here ten and a half years in a twelve and a half-year-old Google So that’s the composition of the office We have been there for ten and a half years together, eleven and a half, and somewhere in their for Matt, and we have worked together since the day I arrived I have had a fairly long academic background before I arrived at Google I got a Masters in Search and then a PhD in Search Yes, there is such a thing And this is back when it was the sleepy field that librarians used to study And what computer scientists thought that wasn’t really the prestigious thing to do operating systems or compilers were This is all geek talk And I ended up getting a PhD in Search and went to AT&T Bell Labs, which became AT&T Labs, to pursue an academic career, and published a lot of papers and so on about Search And then in 1999, my good friend Krishna Bharat, who is father of Google News, in a conference over beers told me that he’s going to this company court Google in 1999 And Krishna was single then, and just to give you some color on the conversation And I, and my lovely wife, [? Shilpa ?] sitting there, we had one child and the second one on the way And I said, Krishna, Google what? Like, you are single You can afford to do this Google shmoogle’s all going to die I have a family to feed I work for AT&T. [LAUGHTER] And next year, I was here, working with Krishna and these wonderful people And the rest is, indeed, history Ben BEN GOMES: So I think common thread is Krishna, because I went to high school with Krishna We both had chemistry labs at home and that was our common love, actually, doing chemistry experiments Now this is something that both of us, being brown-skinned boys, don’t do in this country anymore [LAUGHTER] That was our common thread That our common interest from a long time ago and both of us went into computer science He told me he’d interviewed at Google, and a lot of sharp people there, and seemed like a fun place to work I was like, sure Why no I could be there, too So I interviewed there I got the job And I remember my boss at Sun asked me, he said, do you think Google is ever going to be one tenth the size of Alta Vista? [LAUGHTER] I was like, I am not sure, but you know it looks like a fun place to work And his boss made fun of me in the group meeting saying, they bring to work for a company that has an exclamation mark in their name, because Google, if you’ll remember, used to

have an exclamation mark Both of them now work at Google [LAUGHTER] And the scuttlebutt in the valley at the time was Google had a secret plan, because it clearly wasn’t Search These are smart guys there’s no business in Search And so I remember when I first joined Google, I asked Urs, who was my boss at the time– Urs Holzle, the VP of Engineering– so what really is Google’s plan? And when he told me, it’s Search, I was so disappointed [LAUGHTER] It turned out to work out OK though MATT CUTTS: Hey, everybody My name’s Matt and I was actually a computer graphics guy I was working on my PhD at the University of North Carolina at Chapel Hill– yeah, woohoo Represent A couple go Tar Heels– and got a job offer to go work for Google in 1999 And my girlfriend at the time said, well that sounds like a really good opportunity, but I’m not willing to move all the way across the country without a wedding ring So we actually eloped and drove across the country Took a quick honeymoon Showed up and the very first project that I got a Google came about because my manager stopped at my cubicle and said, Matt, how do you feel about porn? [LAUGHTER] And I said, well it depends Why are you asking, exactly? [LAUGHTER] And so they wanted me to write a familY-safe version of Google, which is called Safe Search So I wrote the first version of that And in the process of doing that, I found out that there are bad people on the Internet [LAUGHTER] And so for the last 10 or 11 years, I’ve been dealing with bad people on the Internet And of course, there’s a huge number of good people as well, but that’s been a lot of what I work on DANNY SULLIVAN: And Matt has a corn field that he can wish those bad people into You don’t want to be one of them We’re going to dive into some questions Just a really bit of background just to set the stage on some of the technical stuff It won’t be that much of a technical issue Search engines, the essential pieces of them, there is a crawler that goes out and find pages from all over the web and stores the content that this crawler finds in what we call an index It’s like a big book of the web The crawler’s constantly updating this index, keeping all the pages fresh Some of them it visits a lot, maybe every few hours, maybe every few minutes depending on how frequent that page Some it goes back as it needs to Your personal blog, maybe it only needs to drop by once a month When all that stuff is put into this index and we come along, we do our searches, that’s where the search algorithm kicks in And it goes and it flips through that giant book, if you will, and it finds all the pages that it thinks are most relevant and should be shown in response to a search Now, for me, and for others, when you talk about what an ideal search engine should be– and this is how the big search engines are generally operated– they are akin to being a newspaper You have editorial content that is supposed to represent what somebody feels as a fair representation Some of the best stuff that they can put together that is being shown to you independent of any advertising influences or any sponsorship that’s going on out there You can get ads but they go off to the side, just like you might get in a regular newspaper If you follow through with the newspaper metaphor, search engines have a much more incredibly hard job than a newspaper The newspaper will assemble this content each day They’re in their office building Maybe they’re getting phone calls that come in They’re getting story tips or whatever For a search engine, it’s as if they’re in the middle of Times Square and they are surrounded by people shouting things at them saying, I got a great story for you I got a great story for you And some of those are great ideas, but the people don’t shout very loudly And you need to hear better and bring them forward Some of them shout very loud and they’re terrible ideas And some people are just purposely being misleading And in the midst of all this, you have people who are literally just dumping garbage all around where you’re working Just junk all over the place So go out there and come up with some good search listings off of that With that kind of background, all these things going on, people actively trying to mislead you, people who don’t even understand that when they’ve rendered their entire web site built out of, say, flash, it was kind of hard to read How do you figure out who to trust, who not to trust? How do you go about that core part of the job, coming up with the right listings? AMIT SINGAL: That’s a great summary of how search engines operate, Danny Imagine you have a book with billions and billions and billions and billions of pages You do have an index like you have at the end of a book, where it would say which word appears where And that’s the index, a very similar index that most search engines, including ourselves, built Now when you type a query into Google, you give us a few words and we use that index to find the few most relevant

pages for your two, two and a half, three word query This is clearly a deeply scientific process It’s incredibly hard as a science Has been studied as an academic field for over 40 years And at its heart, there are simple heuristics Simple heuristics like if you said Danny as your query, then a page that contains the word Danny many times is probably more relevant than the page that contains Danny just once That’s the first simple principle of search And then there are other principals like words like the are not very important so don’t use them as boldly as you would use a word like Danny And you take such simple principles of how language behaves, and you build algorithms based on those principles in an environment that’s the modern web, and that’s with the basic description of a search algorithm Now clearly, in the real world, everyone wants to come up number for the query Danny Sullivan And therefore, they give us pages which say Danny Sullivan one thousand times, where as Danny is so polite He only uses his name one or twice on his website So in the real world, all these assumptions, all these principles are challenged like we are fighting through all this complexity to get to Danny’s page, which we do get our there So that’s roughly how it works So we have taken simple principles of how language behaves and coded it in into algorithms to return to you what we believe are the most relevant pages on the web That’s the words of the web, according to our algorithms MATT CUTTS: And in the same way that all good things in moderation, according to the ancient proverb, the first time or two you see Danny Sullivan, that’s great But if you see it a whole bunch then it starts to turn a lot worse And so you have to realize that the web is like a giant torture test. People will do crazy things on the web Any program you write to parse a web page will break by the time you get web pages that are long, that are complicated, people don’t close their tables, and also malicious or deceptive people And so you have to worry about not just ambiguity, like there’s a Danny Sullivan who’s a race car driver, turns out, but also just people who want to rank for everything And so it is a really difficult problem But through those sorts of principles, those computer programs, those algorithms, rather than having a single person saying I think this ought to be number one or I like this result for that It’s much more scalable to have the computer programs and say, OK They can work 24/7 They don’t have to sleep It tends to work a lot better in different languages All the sorts of things BEN GOMES: And I think one of the early insights that Google had was that the link structure of the web was going to be really useful for this And so the page rank algorithm, which was at the heart of Google so early is such search was actually fundamentally leveraging the link structure of the web to actually find out which is the real Danny Sullivan, in addition to all the other things that we were doing AMIT SINGAL: So you can imagine when you’re going on the web and someone says, click here to go to for Danny Sullivan’s home page That link– hyperlink as we call it– is a tiny recommendation for Danny And we take these people’s voices which are encoded on these web pages as links and we have built an algorithm to actually harness the power of the web These voices of authors who had built this web to build whats Google today DANNY SULLIVAN: Now the links especially were the big claim to fame when Google came along People had been using links, but you really took it up to a new level And suddenly you could find this needle in the million page haystack that was happening And over this period, if you go back to say 2000, it’s largely what I’d say is the Google decade You were this unquestioned leader when it came to search That was the gold standard And then towards the end of last year and through this year, it’s been really amazing the amount of attention that’s been focused on search quality We had the sunglasses merchant who the New York Times profiled because he was convinced that because he was such a rotten merchant that all the bad reviews were giving him links that helped him do better We had complaints that people were saying that, wow I’ve written this content somebody has simply copied it, or scraped it, and put it on their own website and now they’re out ranking me I’m the originator You’re not finding my original content There were concerns that what’s been dubbed content farms– and won’t get into the debate about what those are– but people were concerned that there were these content farm things and they were flooding Google with low quality listings And then suddenly it seemed like there was this turn that oh, well Google is just terrible and all your results are junk And you’ve reacted to that with a series of updates and a

series of changes The merchant prompted this unprecedented five-day shift in your algorithm that I’d never seen happen so quickly You’ve had other updates on the scrapers Probably the most talked about thing had been this Panda update Google often gives nicknames to when they make these algorithm changes The most recent algorithm change they had was nicknamed Panda So you’re making these changes How do you decide what’s broken, how to fix it, and assess whether or not your fixes are working? AMIT SINGAL: This is a very good question And the heart of Danny’s question is how do you decide if I should change my algorithm in this direction and is that the right change or not? And what we do at Google is that have a few principles on which we operate this search team And the first and the foremost principle that we have put in place is do what’s best for the user And once you use that principle and develop science then to measure what’s best for the user to the best to scientists’ ability– I’ve been a scientist in this field for 20 years Ben and Matt have worked here for almost a dozen now So what they have done internally is developed very rigorous scientific methodology to evaluate if turning of an algorithm right is a good thing or turning it left is a good thing, and by how much is it going to do a good thing for the users or not And it would be really good to set up the context of the team How we operate Let me just use a very, very simple example of how search progresses And I’m going to use a dumb example to just prove a point Suppose one day an engineer walks up to me and says, Amit I’m going to promote all pages that have pink background Matt says, no Blue backgrounds better, OK? [LAUGHTER] What does this engineer do with this idea? The engineer goes and writes an algorithm that incorporates his or her idea, which either promotes pink pages or promotes blue pages, depending on who won the argument in our office That’s usually Matt And then that algorithm is unleashed in a sandbox, which we have of the entire web inside our offices Not really in our offices They are in data centers, but only available to our engineers No users are subjected to that sandbox And the engineer runs this blue-page promotion idea in the sandbox So the new algorithm is blue page ranked higher, other page raked lower Now you have the new algorithm and the current algorithm The two ranking systems. We take these two ranking systems and put them through amazingly rigorous testing Rigorous testing involves things like we will take hundreds and thousands of queries from our past logs, run the old algorithm, run the new algorithm We will show it to an independent human being outside who doesn’t even know what algorithm is being tested And sometimes we do this on them so that they don’t develop a favorite, left or right OK? And computers keep track of whether left was the new or the right was the new And statistically after you have run thousands of queries through this left- right blind test, first number emerges that says, no, promoting blue pages is not such a great idea, Matt MATT CUTTS: Sorry It was a thought AMIT SINGAL: So, there we go, right? Now that’s the first test you have to pass Once you have passed that level of testing, we take a tiny subset up over live traffic and we take this algorithm and we put it outside in a real data center where users are doing their queries And we take a very, very tiny subset of over live traffic coming into that data center and subject it to this new algorithm, if it was good The blue one didn’t make it to live testing And if users are now liking the new algorithm, where their liking is described by they are clicking much higher in the ranks in that relevant documents relevant documents are ranking higher, and they are spending less time searching through results and click on multiple results– they find the right result quickly with speed– then the new algorithm’s better

And with these tests in mind and this much statistics behind every change and we make about 500 changes to our algorithm every year, and we run 20,000 plus such tests every year With this level of rigorous scientific testing, we put a statistical report together put by a statistician, not limited to the engineer or his or her team That report is brought to a group of people, all three of us included, to make a decision whether this algorithm should be launched to the world or not And in that committee that you may have about in Stephen’s books and articles, is where a whole lot of debate happens It’s not just these two numbers that will force you to launch something There are many more considerations that go into launching a system like, is it good for the web ecosystem at large? Would it benefit authors? Would it benefit high-quality content? Would it keep our system simple so that we can maintain it much longer? And with all these considerations put together, that committee says, yes, we should launch this or not Hey, what did I miss? MATT CUTTS: Very good description And there’s a couple things you might not realize One is that there is no way to change a computer program, to change how Google ranks and scores things, that’s going to be perfect That’s going to make every single query better Because we get over a billion queries a day And if you get hundreds of millions of queries, you could make 500 of them great and only one worse, but you’re never going to make all of them better So there’s always this tension There’s always this trade off where you’re trying to have the goals of the best results for users, the best quality, helping the web ecosystem And then there has to be room for intuition and experience So, for example, I work on spam That’s people who tried to cheat And it’s definitely the case that users often click on spam because they see something that looks enticing Oh, this is what I looked for And so we’ve seen examples where one person will click on the same spammer eight times in a row And you just want to– how can you do that? After seven times you couldn’t tell this was not good stuff? And so by the looking at just the raw clicks are just the raw statistics it might look like this is a horrible change So you have to take those factors into account But absolutely, the statistics and those launch reports carry so much weight because they’ve built up this intuition about what are good changes and what are not as good for users BEN GOMES: And some ways this is essential for us moving faster Because if you don’t know that you’re making progress, you can constantly end up second guessing yourself and so on So being very rigorous is essential to our speed of moving fast. So we are constantly sure that we are doing the best thing we can for users at any given point And we go through the same kind of rigor for our interface changes, too, because those are sometimes even more complex to evaluate because there are more moving parts in the system So the live experimentation and so on There we augment it with user studies where we have a usability lab when we watch people use the product and make sure that it actually works in practice by watching users But they also go to the same level of rigor for all the interface changes that we make AMIT SINGAL: So that’s a long way of saying we are in the process of scientifically determining what’s the best thing for the user This measurement in itself is a science We are in the cutting edge of that science And that science is evolving as well But to the best of science’s ability to predict whether it change is good for a user or not We use that science to make changes to Google DANNY SULLIVAN: I’ll come back in a bit to the determining of the relevancy and if you’ve got it right But Matt, I want to pick up on what you mentioned on the spamming You are in this hostile environment What’s the conference that you have? It’s like information retrieval Information retrieval, which was a science, where people would be like, how do we get our stuff out of the lexis nexis database nobody’s mentioning. and then you have like these hostile information retrieval– MATT CUTTS: Yes There’s an entire conference called adversarial information retrieval It’s where the goals change because the people are trying to cheat and deceive you DANNY SULLIVAN: So you’re in this environment How do you measure up what’s wrong? How do you prevent getting the wrong person? You just said that you’ve got people who will maybe click eight times on a spam thing You have to use some intuition But then some people make click eight times on the right thing MATT CUTTS: Sure DANNY SULLIVAN: So how do you do that? MATT CUTTS: Absolutely So we try to provide very clear guidelines If you search on Google for quality guidelines, we actually have instructions for publishers and webmasters about the sorts of things that are good and the sorts of things that are not as good And hopefully they make sense because we want to judge the same page that a user sees And so, by that principle, you shouldn’t hide white text on a white background, you shouldn’t show us a page about cartoons and then show hardcore porn to users, things that you would think would be intuitive

And so what’s good and bad, what’s the curse, is that once you know how to see spam, you will always see spam in any system You’ll look at the cheaters, you’ll find the people You’ll recognize the people who are trying to game the system And the nice thing is that most of the time it’s very clear Spammers tend to be lazy, and they tend to go all the way out and try to get as much traffic as possible as quickly as possible And that leaves some footprints in some ways that you can spot And what’s tricky is when people get more towards that gray zone So, for example, we have this category of stuff that we call web spam And we have very clear guidelines And some of the stuff that happened in the last year regarding content farms was stuff that you might consider just outside of the guidelines They didn’t necessarily keyword stuff They didn’t necessarily do something horribly bad for users, but it was still really low-quality content that regular people would complain about And so by just stepping barely outside that zone, it fell between the cracks for a little while And the nice thing about Google is that by being in the same office, you can turn around and you can say, hey is this your job or is this my job? OK, we’ll tackle this OK And that has worked very well At Google is also the concept of a war room We try not to go for big war-like metaphors, but that one actually dates back a long time ago And whenever you have a crisis you say, OK Get everybody in the same room And that makes such a difference for collaboration, such a difference for teamwork because if somebody is a minute away, you might walk three or four times a day to check in with them But they are in the same room or you can see through the glass walls, and our offices have glass walls at Google, you can see if they’re at their desk You can walk right over to them You can see whether they’re looking unhappy looking at their computer And so you can say, hey, is something broken? Do we need to fix something? And that really does make a big difference in productivity as well DANNY SULLIVAN: Mercury news had had this article about, hey I’ve been penalized by Google and I’ve had this big network and I didn’t even know it happened And we’ve gone back and forth on this before I’m like why don’t you just tell everybody if they have a penalty You guys will tell people if they have some penalties, but you won’t tell everybody So why not just say, hey You know what? You’re doing something bad and I will report it to you in our Google webmaster central system MATT CUTTS: Absolutely So Amit talked very well about algorithmic search and the vast majority of what happens is all involving computer programs. My team sometimes has to take manual action Because if you type in your name and you get off-topic porn, so you write an angry email to Google and say, I would not like this porn result showing up for my name I’ve never been in a porn film in my life And we write back and we say, well it’s going to take us six to nine months and we think we might have an algorithm that might help, that’s pretty discouraging So my group is one of the very few where we actually are willing to take manual action And so we’ve been trying a communication experiment this past year where if we have taken manual action, you can do is known as a reconsideration request, so it’s basically an appeal And we will tell you whether we have taken manual action against your site are not Now if an algorithm is ranking your site lower, well I’m sorry With over 200 million domains there’s no way we can talk with every single web master or publisher one-on-one Literally everyone at Google would have to do customer support There would be no one left to actually run the computer programs and write new algorithms. So we think that that’s a relatively good compromise in that if there has been manual action you can now start to get information about that You can say, here’s what’s different Here’s what’s new Please let me back into Google I’ve taken off the hidden text or whatever’s involved DANNY SULLIVAN: And if you’re logging in it’s become more broad, more of the thing But there’s still some things you just we’re not going to tell you MATT CUTTS: Well previously we hadn’t revealed everything we knew because there are some really bad guys out there Al Gore has had his web site– DANNY SULLIVAN: Al Gore’s bad MATT CUTTS: Al Gore’s not bad, but his web site has been [LAUGHTER] Donald Trump has had his website hacked And so there are a lot of really malicious people out there that will install malware, viruses, spyware, trojans, whatever you want to call them, stuff you don’t want on your computer And you don’t want to clue those folks in So it is a tension But we’ve absolutely been moving more towards transparency As much communication as we can figure out how to do In fact some Google employees in this room have worked on trying to improve that process And have really done a great job of it DANNY SULLIVAN: Now people not doing well in Google have sparked complaints And in fact, people not doing well in search engines have sparked complaints since before we even had search engines And there’s been various things that have gone– the most common complaint that I’ve heard in my time has been, well you’re not ranking me well because you’re trying to get me to buy an ad But lately, now it seems to be that the reason that you’re not ranking people is because they’re all competitors to you And that to preserve the Google monopoly, you’re blocking them off So what’s the deal? You guys are wiping off competitors? MATT CUTTS: So that the nice thing about working in search quality is we don’t worry about ads or revenue at all We have a very clear mission of doing what’s best for the user

So that’s not in our area or scope of worrying about at all AMIT SINGAL: And so there’s a clear church and state separation between search and adds No matter how much money an advertiser pays Google, and that kind of goes into our revenue They cannot improve their ranking OK That’s fundamentally how it works And then the question that Danny poses is hey, how are you now putting this stuff up? Because I’m competing with you And you have demoted me This is stuff, your stuff That’s what they call it And I go back to our first principle Do what’s best for the user Our job is to give users the answers to their queries What they ask for is what we need to answer Now in a most simplistic form, if the user types the query two plus two Should we return a list of pages that have the words two plus two on it? Or should we say, four What would you expect? If you are writing a search engine, what would you do? Take the next example When someone says the query 1600 Amphitheater Parkway, Mountain View California Would you not show them a map pinpointing exactly what they’re looking for? Our job is to answer user queries And that’s what we do Everything that we do– someone types GOOG They’re looking for what’s the stock price of our company today, at this point And we return that value right out there When you work with our first principle that it’s all about the user, and our job is to answer your queries Everything else falls in place BEN GOMES: I think we think about it in terms of the time it takes you from the time you enter your query to the time you get the information you need And it’s going to be a lot faster for you to see the number four over there when you typed in two plus two So that is our goal To get to the information, the answer you need as quickly as possible And that guides– we believe that’s what’s best for the user And that guides our decision making MATT CUTTS: And if you go to the very extreme, if someone comes to Google, and types in poison control You really want them to get the phone number for poison control as quickly as possible So you want to get them that answer Whatever it is they’re looking for DANNY SULLIVAN: So that gives an answer Some of the direct answers And then you get into this issue of people saying, well, I just did a shopping search And instead of you showing me, listing a bunch of shopping search engines, you’re sending me into Google Shopping or Google Product Search So now you’re just trying to keep yourself there AMIT SINGAL: This definition is actually somewhat absurd If you look at Google Product Search, that you’re talking about Danny It takes pages out there on the web It’s just a search index system It organizes that information better related to your task If your task is to figure out how much does something cost? How well is it rated? Should I buy this? Is this merchant high quality? And Google Product Search takes information that’s out there on the web and merchants can feed it to us for free For free There’s no charge to anyone who is in Google Product Search They can feed us their prices, their availability, and so on And at the same time, they can tell us what web pages they’re selling that item on So Google Product Search is just a different lens or a different interface search that’s far more effective for query [? and ?] [UNINTELLIGIBLE] And yes indeed we send users to that interface From there, they can do their research, and go to that merchant to complete their transactions BEN GOMES: In the end they are still going to Amazon or whatever that merchant is AMIT SINGAL: I think you can squint at it hard enough, and say this is Google’s own stuff But the truth is, it’s all the pages out there on the web And merchants out there on the web, feeding us information for free MATT CUTTS: Well and Danny, you had actually made a really neat graphic a few years ago Because at one point, you might do a search for Tom Cruise And then if you want a picture of Tom Cruise, you had to click on a tab, images And if someone types in sunset, or daffodils, or roses, you might have learned over time that people actually want pictures of a sunset, or a daffodil, or roses And so Danny had made up a cool graphic that was like Google in 2015 with tabs all over the place Look for people, look for whatever And really what people want is they just want to type something in and get something useful back out And they don’t want to have which of the 32 different options of lenses do you want to search through? DANNY SULLIVAN: Now if you go to see the purest form of a

search engine, to me, it’s I’ve done the search I’ve clicked And I’ve gone outbound And so from the search engine to a destination And that the search engine itself is not a destination And when you talk about like with shopping search When I’ve heard these arguments, most of them don’t hold up with me because I think ah yes, Google sent you from Google to Google Shopping Wherein you still left Google Shopping and went to a destination merchant But you get these tricky issues where Google actually hosts content and indeed becomes a destination Cases like Google Books Or Google Places, where you are aggregating and consolidating a lot of information So if I do a search, rather than going outbound to the merchant, I may go to Places And perhaps the best example of this is YouTube Where I do a search, maybe I’m going to get a YouTube video that’s coming up, and you are a destination So there’s inherent conflicts in that How do you deal with that? How do you deal with those conflicts? AMIT SINGAL: So we deal with our conflicts with the same first principle that we have ever had Test, test, test, experiment, scientifically test, and make sure all your changes are good for the users And once you have that principle in place, and you’re designing your result’s page, because our job is to return a results page that’s really, really valuable for the query You’re designing your results’ page and testing it extensively Then for the query evolution of dance, you would see yes, evolution of dance site And yes, you will see some YouTube videos as well Because that’s what the users are looking for So we keep going back to our first principle When ever a conflict arises, we go back to our first principle: is this good for our users? If it is, we’ll do it MATT CUTTS: And at the same time, it makes perfect sense I remember AltaVista used to be where you’d search for a person’s name like Jeff Dean And it would say buy Jeff Dean on eBay Right? And you couldn’t really buy people on eBay So it wasn’t a very good thing for the user experience If we were always showing something– like every single time we showed a result from Google Books, that would be really annoying So we know it’s not in Google’s best interests to annoy users, to do things that are bad for users Because then they get turned off There’s plenty of other places to go and get information on line DANNY SULLIVAN: But as these other things have expanded, have you ever thought, I wish we didn’t have that? Because my life would be simpler Because then I wouldn’t have to deal with these kinds of questions AMIT SINGAL: We like your questioning Danny They’re fine But fundamentally, right, every time we build something– let’s say Maps It was the most innovative product when we launched Google Maps The first product that allowed you to scroll the map in using Ajax technology at the time Everyone else followed suit To the degree that now, if you land at a page which has an embedded image of a map, you try scrolling it right there So we feel proud to build these innovative products And when we build these innovative products then giving users answers right on the result page is absolutely the right thing to do So clearly our job is going back to our first principle Giving users answers to what they asked for And sometimes we have to lean upon innovative products like Maps to answer a query BEN GOMES: I think the core part of our competency is actually ranking a variety of these different information sources I mean it started with PDF’s When on the first time, we started crawling PDF’s in 2000 PDF’s are long documents And so they can begin to dominate all the results And so as soon as we started crawling PDF’s, we saw results full of PDF’s So we had to deal with this issue of [UNINTELLIGIBLE] content there And I remember I was working on crawling and [UNINTELLIGIBLE], and also ranking And then I realized, well we just hired a world expert on ranking Maybe I should ask him And he has not yet even come to Mountain View I remember calling him up to say, Amit, what you did your thesis on this stuff What do you do? And so I remember talking to him about this AMIT SINGAL: So Ben calls me And he says, hey, we just started crawling PDF’s And they’re long documents They have the word over and over again For every query I’m seeing, just PDF’s For the query IBM, I’m seeing PDF, PDF, PDF, PDF Not IBM.com And you have written a dissertation on how to deal with varying document lengths in such systems. And I had joined Google in its New York office back in 2000 And Ben was in California This was around late 2000 He calls me up We were driving back home from Jersey City from having dinner at a friend’s house with our two kids in the back Right? My wife was driving the van And I’m talking to him over the phone, solving, giving him formulas that basically solved the document length problem And at the same time like trying to keep the kids quiet

You know, take it easy guys It’ll be OK, fine Right To the degree that we didn’t pay attention to what was happening to the car, and we ran out of gas and had to call a tow truck MATT CUTTS: So maybe focus matters a lot Whenever you’re trying to figure out these apples and oranges and how to blend DANNY SULLIVAN: Matt, you alluded to this earlier But you don’t manually control the rankings If there’s results that are showing up, it is all down to the algorithm With the exception of a tiny little experiment, we won’t get into right now But you just haven’t done that So things show up And there have been times when– the best example I think was one of the first examples where you would do a search for Jew And you got this site called Jew Watch that came up It was a hate site And people were like, how can you allow this? And in the end, you left it up And this goes on all the time There are these things here that a lot of people would have a consensus of saying, this sucks You shouldn’t be listing that Why don’t you take it out? So why don’t you take it out? AMIT SINGAL: Let me take that Right Danny’s referring to a principle that we have held dear to us in our search team Which is that we would not manually promote, demote, or remove results Even if our judgment is saying our algorithms are doing the wrong thing The extreme case in point was query Jew When someone did that query, it made all of us tremendously sad to see our algorithms fail by putting an anti-semetic site at number one We were in pain We were really, really hurt by this This is not what our principles are as individuals, as a team, as a company And people said, why don’t you just blacklist that site? And get rid of it? However, we said no It’s our algorithms going wrong And we will find a solution to this algorithmic problem by algorithms. Because this judgment was so clear-cut in this case, that everyone’s instinct would be to go blacklist that site But in the real world, not everything is such black and white There are lots of slippery slopes There have been many other queries Where various interested parties have wanted us to shoot results out of Google’s search engine One such query is Scientology Where we have both perspectives return in our top two or three results And clearly one group doesn’t want the other perspective to be there If we start intervening manually, first of all, we would make such arbitrary judgments Number two, we may paper over one hole in our algorithm, which would not fix similar problems in many more queries that our algorithm may be causing, and in many more languages that we don’t even read So our principle that we have held near and dear to our heart is we let our algorithms reflect the voice of the web And sometimes our algorithms will be imperfect And we work hard to make them better and better But our subjective judgment, though mostly right, is not always the right thing And that principle has actually served us well over the last 10 years of us doing that And that is reflected in our users coming back to Google and liking what we do BEN GOMES: I think that’s a principle that’s come down from Larry and Sergey In fact, with the query “Jew,” shortly after the problem arose, Sergey said, we will not be changing this And then the web changed, and actually the ranking changed around And he came back, and he yelled at us like, why did you change? And he’s like, no, we didn’t change anything The web did change, briefly, and then it flipped back again But over time, the algorithms have gradually improved to get rid of many or most of those sorts of problems And the side effect of that actually, like you referred to briefly, is that we have been very good in languages we don’t even speak, because we’ve relied so much on algorithmic approaches Because otherwise, we’d have to do the same thing in every language in every country that we’re in, and that’s an impossible task A consequence of taking this algorithmic approach is that we can then be excellent in small languages, in languages that otherwise are not considered maybe that important And it just works across the world AMIT SINGAL: Yeah, a normal company would set their priorities based on revenue in a language We don’t pay attention to that We say, our job is to serve every query everywhere, in every language, whether it’s in Swahili or in Hindi or in English, of course And this algorithmic approach allows us to dish the best

results for English But the same algorithm, since languages are not that different in general, when you dive deeper– there are lots of differences that I’ll not go into right now– but that algorithmic approach, then, any algorithmic improvement made in English not only serves English It serves Japanese, Swahili, Hindi, you name it And the entire search system in this world gets better And we feel proud to have had this principle DANNY SULLIVAN: I’m sweating, because I’ve got five questions, and 10 minutes to get them in So we’ll go into lightning round You’ve got, also, listings that have impacts on people Rick Santorum would probably be very happy if you could make an algorithm change for searches on his name But you could argue, well, he’s a public figure, did certain things, upset other people Maybe that’s just what happens But you’ve got people who are not public figures, who are not known in any way, do searches on, say, their names And they go, I can’t believe this thing’s coming up, and it needs to go away This is terrible So why not let them take that stuff out? MATT CUTS: It’s a tough call, because it’s a slippery slope, right? Whenever your faced with a he said, she said sort of situation, you really don’t have the time or the staffing or the judgment be able to make the right call in every single situation So normally, we say, look, Google is attempting to be a reflection of the web Right? If there’s someone who’s important in real life, we want to reflect that on the web And if someone is libeling you in real life, you can either send them a cease and desist, you could get a court case There’s lots of ways to take care of that And online, there’s lots of great ways to get your message out You can start a blog You can tweet You can be on Facebook So there’s many, many ways where you can have other search results that can show up, rather than just this one negative search result But at the same time, I would say we feel the responsibility Because we hear complaints, we hear people who are unhappy And it’s safe to say, with so many users and so many queries, there’s always someone who’s unhappy Or there’s always someone who you can point out a really bad search result And so, I can’t speak for these guys, but a lot of the times, I go to bed at night thinking about how can I fix that problem Right? Or you spend that time in the shower thinking, OK, how is that going to get fixed But as long as you have hundreds of people behind the curtain who are thinking about how to improve search quality on all those different dimensions, then in general, it tends to go pretty well AMIT SINGAL: Let me just briefly add, we have spent a substantial amount of our careers devoted to search, [INAUDIBLE] and we think day in and day out about search and how to make it better If someone has seen some content about them that they don’t like, we hurt But then, there is that case when you are searching on a doctor, who you learn had his license revoked three times Surely that doctor doesn’t like that result But our responsibility’s bigger than that And that’s the principle we operate at MATT CUTS: Because it always comes back to the user What is in the best interests of the user? DANNY SULLIVAN: Now, I talked earlier that you’ve had these attacks on relevancy, and we’ve got to improve things And you’ve made these sorts of changes But one of the changes, and it’s not even new, you’ve done it for a couple years, to try to increase the relevancy of results has been to personalize things So where someone is based out of, if you’re doing a search here versus down South, you’re going to see maybe slightly different results Your search history, the kinds of things you searched on, sites you’ve been can come in and influence it So we have these personalized results And now, you’ve got people like Eli Pariser and his Filter Bubble saying, well this is terrible, because now you’re just showing us things we all like And we’re not getting the diversity So what’s your reaction to that? Do we have this kind of a bubble? Should there just be normal results for everybody? Is this is a harmful type of thing? AMIT SINGAL: So you can just imagine, a normal result for searching for a restaurant in Newport Beach would be all New York City restaurants OK? That would be a normal result, because by population and masses and consumption and usage, that’s probably what wins out So we do personalize results And there are cases, like restaurants, where we personalize them tremendously Or weather, we personalize weather results to where you are And there are other cases, like the query “banks.” Guess what? If I started returning Lloyds Bank in the US, US wouldn’t be very happy Or if I returned Bank of America in the UK, they’ll say something snarky about us So there is that level of personalization which makes results tremendously relevant And then the type of personalization that Eli’s Fliter Bubble talks about is the view personalization, that

I’m just getting my view in first. And the truth in the search is that we are returning relevant results But our algorithms are so finely balanced to return some relevant results for you, and some results that have the opposing point of view, as you have seen time and time again in the controversies that we have just talked about, that our algorithms are tremendously balanced to give you a mix of what you want and what the world voice says you should at least know I can give a personal example, the query “Lord’s.” You guys are thinking Dungeons & Dragons and swords and whatnot But I’m an Indian who loves cricket, OK? And there’s this cricket ground in London, Lord’s For me, that’s the first result However, all the swords and Dungeons & Dragons are right there, number two and three And that’s, I would say it’s an ideal page for me DANNY SULLIVAN: Cool I’ll go with you on that I speak a little cricket I’m going through a Sophie’s Choice of which question to throw away You You don’t You all go No, I’ve got two, and then we’ll go into the Q and A. And we should make it fit So we all not only have personalization of results, we have socialization of results, which is sort of a subset of that personal, who you know, who you’re friends with, and so on You’ve been using social search for about two years now and having that influence things But now you have your own social network So how is that going to have an impact? In particular, Google Plus and the +1 system, on the search results that people are seeing? AMIT SINGAL: That’s a very good question So Google was a pioneer in launching social search We were the first one to launch it, where based upon who you know, we can bring their recommendations to right top of your search results sometimes, or we can just bring them to your attention And this is great Because if I’m searching for a plumber or a locksmith, I would much rather have these businesses reviewed by one of you, who I trust, as opposed to the New York Times Square problem of everyone yelling, I’m the best locksmith Please hire me So social search is a great advance in search technology And social search is based upon who knows who, and who knows about what And when you put these two things together, a very powerful system emerges You basically marry the real world, how the real world functions, into search And that search is more relevant, because we can bring to you experts, and sometimes experts and friends you may know, who may know something about what you want to know And that’s a much better system than it is today With Google Plus, we would indeed– Google Plus has been a very successful system We are very excited about it And with Google Plus, we have much more data on who you know and what they know, so that we can improve your search experience going forward DANNY SULLIVAN: Have you notched up the social dial a bit, since it’s launched? AMIT SINGAL: We experiment all the time with numerous things MATT CUTS: It’s early days, but it’s pretty exciting– DANNY SULLIVAN: That’s Google for you guys MATT CUTS: –to look at the potential of how well this might work over time, so DANNY SULLIVAN: OK, I’ll squeeze in this last thing, and then we’ll go to the Q and A. And also, because we also want to talk about some of the team building stuff, I want to catch you after the end of the Q and A. Come back to you about some of the things that have helped you You started as a team of 10 people around a Ping Pong table And now you’ve got hundreds of people all over the place, and yet you still make it together So we want some ideas on what people can take from that But Ben had– you had mentioned a statistic when you were talking, something like two humans only agree on relevancy 80% of the time And that in fact– I think you had said this, maybe [UNINTELLIGIBLE] said it, but– and that if it was a year later, and I showed you the same set of results that a year before, you had rated, you’d only agree with yourself 80% of the time So how do you know if you’re relevant? I know you mentioned the humans and things that are out there, but can anybody know? And can we independently assess whether or not you’re getting it right, when you make these kinds of assumptions of, this algo treat will make everything better? AMIT SINGAL: Yeah, no, let me go to a sports analogy In soccer, you have tens of games in a season And if you lose two, you’re out In baseball, you have 160-plus games in a season And winning or losing a single game isn’t really going to determine whether you’ll make it to the World Series Just look at our Giants And what we do is, by giving thousands of queries, and each query to multiple human beings, we can statistically measure the inter-human being agreement on relevance that you mentioned, which is about 80% in general

I say this is relevant, with 80% probability he will say this is relevant So we can measure that and incorporate that inter-human agreement into our deep scientific measurement that I’ve talk about And once you know that, then you can rig up statistical tests to say, have I done enough samples through to definitively say with 95% confidence, things like t-tests and so on, that the new system is indeed better DANNY SULLIVAN: But does it skew a bit? Because you ask these people, who are not experts on the things that they’re searching for, to tell you whether or not they think things are relevant, so? AMIT SINGAL: But time and time again, this 80% statistic holds up So for your query, I’m 80% accurate Thus, for my query, you’re 80% accurate And just knowing that and incorporating it into our evaluation formula allows us to do the right thing BEN GOMES: And the live data actually has– we have huge volumes of live data, even when we do tiny experiments, right? And so there, you can really make use of strong statistical techniques to make sure you’re getting the answer right And there, the person who’s looking has a real information need And so you get a good determination of whether they are being satisfied MATT CUTS: But I think the part that might frustrate you is Google has spent 10 years trying to figure out how to evaluate potential changes But we haven’t talked as much about that outside of Google And so to have a third party to say the relevancy is better of this search engine or that search engine is really difficult And unfortunately, a lot of people will do three queries, and that’ll say oh, OK, I found my homepage Therefore, this is the best search engine And so there hasn’t been as much rigor in the third-party analysis A lot of it is reduced down to metrics about who happens to have more growth and queries this month, or something like that DANNY SULLIVAN: OK, we’re going to go to the open Q and A. And I’m also going to ask you about where you think things will be in five years from now when we get to the end of that, too So I’ll save the last five minutes there If you have a problem with ranking on Google, if you could move to this side of the room And we’ll get those questions to you So we have our first question over here AUDIENCE: OK In the last year or two, I’ve seen more and more of what I’ll call type-ahead, where Google attempts to finish my typed query And I think I’ve seen search-ahead, where I actually get results before I hit return Can you comment on how that’s worked from your perspective and where it’s going? BEN GOMES: Yeah, so we started doing Google autocomplete about four or five years ago And the idea there is that we still think of the search process in terms of the time from you have an information need to the time you’re seeing the information you want And so autocomplete actually helps you formulate the query Because if you get the query formulation wrong, if you’ve got a spelling mistake, if you’ve got the wrong words, it’s very hard for you to get into the right information So autocomplete was a first step in helping you formulate your query and essentially greasing the rails to your answer Now the second step is, the best way that you can evaluate whether this is the right query, is if you can see the results So the idea behind Google Instant was to begin to show you the results as you type So you can begin to evaluate whether your partial query is what you wanted And we find that people get relevant results well before they are finished, several seconds before they finish typing their query And so, we are seeing a lot of benefit to users from actually showing them results early And we’ve been very, very pleased with some of the impact it has on the time it takes the users to get the information they want AMIT SINGAL: Ben is being modest. He led Google Instant And Google Instant has been a huge success for us We are very proud of what we have done there MATT CUTTS: And in fact, Google obsesses about speed to the point where we just launched something where, if we really think it’s likely that you’ll click on the number one result, then we might pre-fetch that, pre-render it, so by the time you click on that result, it shows up instantly You don’t even have to wait for that to download So there’s hundreds of people at Google thinking about every aspect of the search session And how can we give you bionic arms to search faster, and bionic brains, and all that sort of stuff And the goal is to get you to what you want as quickly as possible AMIT SINGAL: They didn’t buy my Google helmet idea with electrodes DANNY SULLIVAN: Next question AUDIENCE: [INAUDIBLE] what you’ve done to earn your success over the years, day-to-day in your development [INAUDIBLE], how does it affect you knowing that competition authorities around the world, including the FTC, may well be [INAUDIBLE] looking over you shoulders, oh and [INAUDIBLE] expensive and time-consuming to have to keep up with other companies, [INAUDIBLE] reputation [INAUDIBLE] yet it’s always a roll of the dice [INAUDIBLE]

What is it you hestitate to do? What is it you don’t, that doesn’t get done? Because of this [INAUDIBLE] MATT CUTTS: Yes, that’s a great question I think, having worked at the company for 10, 11 plus years, the core essence, the principals of the company have remained the same Because it’s almost like Spiderman, right? With great power comes great responsibility And I like to think that we’ve operated with the idea that someone could be watching over our shoulder, and tried to make the right decisions all along Now, you’re right It can be a roll of the dice And so, some of the things that we’ve been doing are trying to educate people How does search work? What are the different policies? So for example, we’ve made over 450 different videos that talk about common questions and answers And if we can do a good job of explaining, yes, there is some manual stuff, whenever there’s spam But the vast majority of it is algorithmic And explain how these are best practices And how every search engine does them to some degree Then hopefully, that decreases the odds that there will be some miscommunication, and some fluke will result in a problem We have a very good legal team, and they think a lot about all of these issues But I like to think that the core values at Google are such that, we try to behave as if we were the underdog We try to figure out, how can you step into the other person’s shoes? So how can you have a good appeal process? How can you scale up on communicating? How can you scale up on transparency and communication? But you have to be mindful that, as an engineering company, we always have to have the idea that we can be wrong We are a very disruptive company in a very disruptive space, which is technology And so you have to be mindful of that responsibility And if there are people who have suggestions for ways to improve, you absolutely want to be open to that and listen to that Whether that’s someone you meet at a conference, someone who’s Tweeting to you, or whether it’s someone who’s a regulator DANNY SULLIVAN: Next question AUDIENCE: Hi I had a question about what you do to deal with it the quote unquote bad guys For instance, I understand searches are generated as a result of an algorithm But, for instance, J. C. Penney, a couple of months ago, that New York Times article about how it gamed the system What did you do to affect their rankings? And what can a company like J C. Penney do to gain in the rankings afterwards? MATT CUTTS: Yes, absolutely So some of the Googlers who worked on that exact thing are a couple tables away from you, as a matter of fact, right now So for people who didn’t read the New York Times story, this was– J. C. Penney had engaged an SEO company, a search engine optimization company And there are certain things that Google prefers not to see in our search results So if you are buying links that pass page rank, it’s almost like a form of payola It’s like you’re paying someone to say nice things about you, but you’re just paying them to link to you And so, that’s something that’s against our guidelines Just like you’re not allowed to do payola on the radio And so, whenever we see a violation of our guidelines like that, we are willing to take action We absolutely try to write algorithms to spot that, counteract those links, to defuse and detect them so that they aren’t an issue in the first place And strangely enough, we’d already detected those links several times I think, if there was a failure in that particular case, it was that we didn’t escalate, and take stronger action earlier But then the flip side is, once the company tries to clean things up– and J. C Penney did a very good job of trying to make sure that they were doing ethical search engine optimization after that incident happened– then you have to make allowances and say, OK, how do we let you back into the search results? And at all times, the goal is to try to make sure that you’re ranking appropriately in the search results So it is a tricky problem in that, in some cases, judgment does come into play whenever you’re looking at violations of our guidelines But the vast, vast majority of the time, you want those to be handled with the computer programs, with the algorithms So that you don’t have to bring that judgement into play BEN GOMES: Our goal, in general, is to reflect the real authority of a web site The goal of search in general It’s not to be ahead or behind it And when somebody’s trying to game the system to be ahead of their real authority in the world, that’s when we have to take action DANNY SULLIVAN: Next question AUDIENCE: Thank you all for a great panel so far Ben Parr of Mashable So you had this cool product called Google Realtime Search, where you could be searching, getting real-time updates and feeds through Twitter, Facebook, et cetera Then the Twitter deal expired, and it disappeared And this is a question for my Google Plus audience, by the way So the question is, when, or if, is Google Realtime Search coming back? And do you need Twitter for that, or is that going to come primarily from Google Plus data? AMIT SINGAL: So what Ben is referring to is, we launched Realtime Search based upon data from various real-time data generation, information generation systems like Twitter, Facebook fan pages, and so on And our deal with Twitter expired

We didn’t come to an agreement And after that, we decided that the value that product was providing was not enough for our users, and we took it offline And we are actively working on– as we speak– figuring out, using our current G+ data and other sources that are out there, to revive the same functionality into Google search results So I could say, stay tuned We are working hard on it DANNY SULLIVAN: Why didn’t we get– and why don’t we have now– just, the ability to search Google Plus? You’re a search company, and we don’t have the ability to search your own social network AMIT SINGAL: Your feedback is very well taken, Danny Believe me And again, we are on it DANNY SULLIVAN: We are always thinking MATT CUTTS: It’s fair to assume there’s always lots of things we want to do, and a finite amount of people So– DANNY SULLIVAN: Next question AUDIENCE: With the addition of Plus One search results, does this offer a new way to game the system, where people can plus one, plus one, and get their searches results to the top, or how does that work? DANNY SULLIVAN: You can buy those now, I’ve read somewhere MATT CUTTS: Not to be cynical, but every change in search involves a potential way to game the system I’ve gotten a little jaded over the years But, the thing that’s actually nice is, if you think about ranking, as it existed a few years ago, it was primarily based on links, anchor text, what’s on the page So we have over 200 different signals The idea that the web might move from anonymous pages in some dark corner, where you don’t really know who wrote what, to a web where you actually do have some sort of annotation that says, this person wrote this, or this person vouched for this page, and you can have the reputation of individual people or authors– so if someone who’s a New York Times columnist shows up on a forum and leaves a two line reply, that can be really important, even if nobody links to it So it’s absolutely the case that people will try to game social signals, Google Plus, all of these things You already see people trying to sell plus ones But there’s different ways, where you have new signals, and different ways to intersect that, and hopefully prevent that And the idea that you can get a big win from all of these potential new signals is absolutely worth giving it a try So– AMIT SINGAL: And no signal is used in its absolute form Every signal in its absolute form has its shortcomings, like you mentioned for plus one We crossed hundreds of signals to build what’s Google today MATT CUTTS: You can certainly imagine having fun like, oh we’ll by some plus ones, and then see what shows up You can play games like that DANNY SULLIVAN: The plus one data right now– it is used as a ranking signal for when you’re logged in, correct? AMIT SINGAL: When you’re logged in, it is a social signal if someone you know has vouched for something, indeed It’s part of [INTERPOSING VOICES] DANNY SULLIVAN: And if you’re logged out, is it being used as one of many signals? AMIT SINGAL: We are experimenting with numerous things, always DANNY SULLIVAN: Which is Google speak for yes AUDIENCE: So my question comes from the standpoint of image and video searches So if I was to search for a moonlit beach, and if the images or the video wasn’t tagged with those specific words, are you guys doing some– advance the art of video and image search, which can get me moonlit beaches, without those words being tagged? AMIT SINGAL: So our image search algorithm is far more sophisticated than just looking for those words, either in the caption or nearby on the web page There’s a lot of computer vision technology built into our image search algorithm Some of the team members are here today Which, basically, use those images to say, hey, this one looks like a beach, and so does this one And now, you have a positive loop You can find some great images of beaches And then you can find some other images of beaches that didn’t really say beach, or didn’t see beach with the same density as our algorithms would have liked So it’s a good question We use a lot of that technology It works very well, and we are constantly improving it, as you saw with our most recent launch of Search by Image An image comes to you, you say oh, what’s that? You just drop it into Google Image Search and, with very high likelihood, we’ll find it for you, what you’re looking at Using the same vision algorithms DANNY SULLIVAN: Next question AUDIENCE: I come from the mobile industry And I’m sure that mobile phones have different kinds of search queries I’d be interested in a comment on that But the next thing that we think is coming is machine-to-machine mobile communication And I’m wondering if you’re preparing for machines to be generating queries in that way And how different do you think machine queries will be then people-generated queries?

BEN GOMES: Well, I think the first point is not actually that accurate The mobile query stream is getting to be more and more like the web query stream A while back, it was different When phones were really slow, when interfaces were not full web browsers, the query stream was very different But today it’s gradually approaching exactly the same distribution as the desktop query stream And so, to answer that part of the question, I think, actually, that things are much more similar that you think AMIT SINGAL: And let me also add, to that part of the question, with the innovations that we have made at Google, with things like voice search, where– it’s hard to type on a mobile phone, so we gave you voice search Amazingly accurate voice search And we start seeing human beings behaving the same way as they behave on desktop I would say that on mobile devices, we still get somewhat more local oriented queries, what’s near me And for that, we have launched numerous innovations Like on Google’s homepage itself, there’s a list of hot things you can do There’s restaurants, coffee shops and so on and so forth So mobile query stream is actually reflecting what users need, which means it’s coming closer to the natural distribution of queries And mobile has been a great success for Google with all the innovations that the search team has made in mobile Not only by voice search, but things like, our buttons are more touchable on a mobile interface Our maps are far more designed for this tiny interface We have done immense amount of work on mobile, and that reflects in our success in mobile Now, the question about machine-to-machine searching Right now, we haven’t observed that much of it happening And I think, once that system picks up, we would have to analyze how that looks But in an ideal world of search for me, search would be so accurate that you can just type a query, and the machine should assume that the search engine, the other machine, would answer it correctly, and then build a whole equal system on top of that platform And that’s our dream platform that we are building together DANNY SULLIVAN: Next question AUDIENCE: This may be a follow-up on the bionic arm question, or a comment there Which is, I’ve heard people in Google talk about search list search, which I interpret to mean that, maybe you can even take the personalization to the extent of anticipating what the user might want, without even typing a query So I’m curious about your points of view on that, and what are the challenges that poses to you? AMIT SINGAL: So it’s a great question Because this kind of technology is what we, as kids, dream of, right? Computer would tell you what to do And– the truth is– MATT CUTTS: Maybe not tell you what to do But help you understand what might be possible AMIT SINGAL: And the truth is, first of all, you can see that in a future that’s possible, based on pieces that we already have, you can build systems where a computer can help you tremendously in making you a far more efficient human being So my phone already has my calendar It knows when I’m free It has my to-do list, which says I have to buy a baseball mitt And it has a map, based on Google Local, of all the places that sell And it knows where I am So it’s not too far out there that you can imagine that computer can gently prompt me, hey, please do pick up baseball mitt You are three minutes away from Sports Authority right there and you have 30 minutes free on your calendar MATT CUTTS: And by the way, it’s kind of annoying I have to have– yes, that would be nice It’s kind of annoying that I have to have a to-do list at all, right? Because Google has announced something called Google Wallet And wouldn’t it be great if you could go to the grocery store and you could buy things with your Google Wallet? All you do is, you tap to pay And then, over time, if you wanted to, and gave permission, Google could say, oh, you haven’t bought cat food for six weeks Normally you bought it every four weeks Do you want to just add cat food to the list? And then, finally, I don’t have to think about, oh, I need A1, or I’m out of salad dressing, or whatever All these little traces we leave, if you’re willing to opt-in for those kinds of things, Google could be the little tap on your shoulder that’s like, hey, don’t forget to get wet cat food AMIT SINGAL: And the key there is that users have to opt-in to these things It’s a critical aspect I am a human being who deeply cares about my privacy And that’s the key part Everything we do at Google, we think about that Sometimes we get it wrong We stand up, apologize, and move on But that’s how these systems and technology will evolve If someone told me, 20 years back, that you would type into some machine, what’s the height of Mt Everest, and it’ll spit out the answer, I’ll go, you’re smoking crack, buddy Go on Right? But, see, what’s possible with technology– we all have to dream it together, and then build it in a privacy-preserving way

DANNY SULLIVAN: If only you guys could answer our email, that would be a real solution Next question AUDIENCE: Brian Fox, Western Union First of all, Google has to know that my cat died last week, because I blogged it And you shouldn’t send me that That’ll be your job I’m curious about your vision– your image recognition technology Do you use the power of the people doing the queries, when they confirm that the beach really was what they were looking for? Does that loop back into your algorithm and improve its intelligence? AMIT SINGAL: Numerous factors go into deciding, in image search, what images are most relevant, and most liked by users And there is that positive loop that, based on users’ choices of what we return, we can improve the system going forward So when you combine the powerful research algorithm based on words with the power of vision algorithm and this wonderful loop, that’s when you get what’s Google create Image Search today AUDIENCE: Lauren [? Chaude, ?] also with Western Union We’re a 160 year old company, and I’m intrigued with the dichotomy of culture, which I care about So you’re obsessed with statistics and algorithms, and yet you talk about glass walls, and the power of being one minute away from each other, and being physically co-located Can you talk about that in this global world, and web, and blah blah blah, all that stuff? Thank you MATT CUTTS: Yes, absolutely It’s been enormously helpful to be very close to the relevant people And Google has offices around the world And so, for example, numerous people in our office will have a video conference unit right there, where the glass wall, they can look behind them, or they can look through the glass screen and talk to someone in New York And so, with very little work, it’s easy to bring up somebody and collaborate, from Tel Aviv, from New York, from Zurich And that makes a huge difference You always have time zones, but just being able to have that face-to-face connection makes a huge difference Short-circuit an email conversation Hop into some sort of face-to-face communication It really saves a lot of time and a lot of misunderstandings AMIT SINGAL: And we have found this proximity of teams– it makes us tremendously efficient We can churn out things, as Danny pointed out, in one case, in five days, we could launch an algorithmic change at this scale And this proximity doesn’t always have to be physical And by video conferencing– and in our office, there are two or three video conferencing units that are open to the world, all the time– and people can just say, hey, can you unmute? I need to talk to you And there we go We have a conversation BEN GOMES: And I think, in the history of Google, we went through periods where we were very densely packed But we also found– and that was not by design, it was just that we hadn’t gotten more space– but we also found those were extremely efficient periods in the company Where people were packed into an office and they communicated a lot more than they otherwise would have It might not have been their first choice, based on their background Many people came from backgrounds where you had your own office, and so on But it created a kind of energy that I don’t think arises otherwise, without that kind of density MATT CUTTS: There are some companies where every developer has their own office Whereas Ben– there was one office as that had three Bens in it And so they called it the Ben Pen But it really does make a difference to be able to just turn around And we have all these cues, right? You can do heavy-duty video conferencing You can do a hang out in Google Plus now You can do Google Chat And there’s these cues that are subtle, like, well, I’m red, but you can interrupt me if it’s really, really important All the way down to email, and meetings, and those sorts of things So having that spectrum to be able to choose what’s best to get in touch with someone, whether it’s something that’s a quick hit, 20 seconds, or a half hour meeting, that really makes a big difference in terms of being able to collaborate AMIT SINGAL: In the early days, we used to say, we pack them tight and give them deodorant And that’s how it works DANNY SULLIVAN: And we’ve got time for one more question AUDIENCE: So Shailesh from Citrix So thanks for the panel for wonderful insights into the search So my question is about, when I do my search, most of the times, I get results back saying that, 20,000 results in two seconds or so But thanks to your excellent algorithms and principles, first two or three links I get my results, many times So why bother spending time for finding and searching for the 19,000 plus results, and giving them to us Can’t you save time there? AMIT SINGAL: So you have seen that number sometime appear on Google’s result page that says we have 20,000 or 200,000 results But the truth, indeed, is what Shailesh said, that if you haven’t found what you’re looking for in the top two or three results– which shouldn’t happen that often, or you can send us mail– then really, going down further is not that useful It’s just that our algorithms do compute numbers for all those results And so we give the user an indication of how much there is

But that doesn’t mean you have to read 20,000 and feel pained about that BEN GOMES: I think, in some ways, it’s a historic artifact There was a time when, when you did a misspelling, you would get much fewer results than the real query, right? And so people used to use it as an indicator It’s no longer true today We correct your spelling, and so on But it’s a historic artifact that has, I think, a little bit of nostalgic value for us, too MATT CUTTS: And by the way, I’ll give you one tip, which is that, it is an estimate It’s not an exact count So if you ever notice, we only give three significant digits when we guess how many results there are That’s a little cue to let you know, it’s not really 982,000, it’s roughly 982,000 DANNY SULLIVAN: And I’m grimacing, because I know I could still do a search where I should be getting a smaller number, but I actually get a bigger number, so– MATT CUTTS: It’s a rough estimate DANNY SULLIVAN: The last two things I want to come back to the team aspect So you’ve talked a little bit about it already, you can network, but– MATT CUTTS: Yes, there is one metaphor that Amit mentioned, which is the baseball metaphor Which is, when you’ve worked together for so many years, it’s almost like having that many games in a season You don’t get that frustrated if you lose one time, or if somebody tells you to go back to the drawing board Because I remember, whenever I launched Safe Search, this porn filter, the first time, I was all ready to go I was already to flip the switch And two engineers tested it out and said, you have too much stuff labeled as port that’s not really porn And I had to go back to the drawing board and figure out how to make it better And at the time, I was [GROWLING] I was kind of frustrated about that But over the course of doing many, many, many, many, many launches, you build up that trust to where you say, this person’s looking out for the best interests of the user This person’s looking out for the best interests of Google So take that as constructive feedback Don’t get so caught up in one particular battle, one particular controversy, because it’s guaranteed, tomorrow, there will be a new controversy A new point of discussion And that’s helped a lot in making things more collegial AMIT SINGAL: Yes, and now I look back at my 10 years of being part of this group– we started around a Ping Pong table with 10 people Now we have hundreds of people in our group And unwittingly, somewhere, we developed the principle that has made this team out-innovate every other team out there And that principle was, I would put leaders in place who I respect technically So the entire management hierarchy of the group is built of people like ourselves, who were engineers, wrote code, can understand what an engineer is going through Their happiness, their pain The entire group hierarchy is built from people who have worked in the group for many, many years Hundreds of people report to us now, but everyone who reports to us has managers who have been there many years The whole group leadership is built off purely technical people with deep technical knowledge about search, and deep understanding of what goes into this innovation machine that is Google And that, we just put in place early on because I couldn’t find enough people to manage people as the group was growing So I said, why don’t– Matt, you manage 10 people OK So Matt got 10 people And then, Ben, you manage 20 people And that’s how we grew the group And it has served us very well This is a lesson in leadership that we have all learned in retrospect We weren’t designing for this But it so happened, that if you are in the innovation space, you need to make sure that your leaders are so technical, that everyone that works for those leaders respects them as technical people And that has worked well BEN GOMES: And I think, all our leaders all stay with technical titles And they think of themselves as, first and foremost, engineers, as we do too Even though they do a lot of management and so on, their self-perception is as technical engineers DANNY SULLIVAN: Can I just say, by the way– your title is– [INTERPOSING VOICES] are you senior, assistant principle? AMIT SINGAL: Principal engineer DANNY SULLIVAN: And your title is? BEN GOMES: Google fellow DANNY SULLIVAN: And you’re title is? AMIT SINGAL: My title is also Google fellow DANNY SULLIVAN: Which pretty much has nothing to do with what you actually do MATT CUTTS: But at Google, you can get anything you want printed on your business card Literally, they do not care There have been some pretty crazy business cards printed Because, titles are– at least, at Google, they don’t make as much sense to obsess about And so, if you can get whatever you want on your business card, then you don’t fixate You don’t obsess about it You worry more about the job at hand, and that tends to work well DANNY SULLIVAN: And then, in a Tweetable, or Google Plusable, or Facebookable short statement, where are we at five years from now, other than bionic arms doing all of our searches? AMIT SINGAL: So all of us have our views of where we want to head five years from now I was raised on a healthy dose of Star Trek And I want my Star Trek computer OK? That’s what I want in five years from now

I should be able to talk to it, ask it whatever, and it should be able to have a conversation with me And search is, of course, fundamental to that MATT CUTTS: I want Star Trek in five years, but in one year, I want the ability to get reminded that I need to get cat food while I’m on my way home And also the ability– Google Voice Recognition has gotten very good And it can’t be that much harder to make a well-punctuated email So I want to be able to do my email while I’m driving home, I’m talking, maybe– at a stoplight, or whatever– and have it look as if I’ve very carefully crafted it by typing That can’t be that hard of a problem That shouldn’t even be a three year problem So I’ll go back and file a bug when I get back BEN GOMES: Yes, I think I share Amit’s vision of the Star Trek computer But I think, in the shorter time frame, I want the search on my phone, which is with me all the time, to work really, really smoothly and effortlessly And it’s not quite there yet It’s gotten a whole lot better Voice recognition is getting there Transmission is getting there You can now do these amazing things Like, you can take a photograph– I was in South American recently I was in a restaurant with Spanish menus I took a photograph of the menu And with image recognition and translation, I could then translate the menu into English I was like, wow That’s amazing And it’s all of– you can see where you’re going to go But you’re not there yet And I want that reality to become completely fluid, completely reliable, so that people all over the world can actually communicate, and get information really easily And I think this matters, not just here, but particularly in other parts of the world, where people don’t have as easy access to information Where people in India, and people in Africa, and people in the Third World, really, have access to this information the way that we take for granted in many ways today And we have access And I think that will empower their lives And I think that that will change the world in many ways AMIT SINGAL: So let me communicate our enthusiasm for search by saying, I feel like a kid going to candy store every morning And you haven’t seen nothing yet DANNY SULLIVAN: Well it sounds like all search in the future will be mobile And we won’t even be searching It will be happening for us Thank you all very much for being here It’s really been a delight I wish I had another three hours to keep going at it So–