Why? Attach seasoned product teams from Google to the research org and have experienced PMs paired with experienced EMs launch products.
Trying to transition a research org into a product org is going to be needlessly painful, especially since the research org needs to be firing on all cylinders in this hyper-competitive space.
Yep. I joined Google X Robotics (which became Everyday Robots, which got canceled) just as the org was winding down a big R&D push and moving to product development. Engineers were palpably hurt by their various projects being canceled, and in my opinion they never had a viable product strategy beyond “let’s see if we can find a consumer use for this robot”. This strategy ultimately failed and they canceled the project, let go a bunch of the people, and now the robots are being used for AI research. So the whole shift from R&D to product development was a failure. They could have saved a lot of grief and money if they just continued as an R&D org, and in my opinion they would have left open some important doors which would have really helped with AI research.
I'm afraid this is going to be another Everyday Robots or Boston Dynamics. Google is ruled by managers, not visionaries. Strategy is 'try and see if it works in 4 years. if not cancel'. They followed it in many cases. So, DeepMind's cancellation is long overdue. A couple of years back one of Google's top managers talking about AGI said it's most likely to happen in DM. But current LLM boom happened elsewhere. Likely managers are disappointed in DM.
Just because a company is 99% one thing, doesn’t mean that the 1% remaining don’t have moments of real genius. What it does mean however is that when those real genius moments happen, the company isn’t in a good position to capitalize on it (ala Xerox or Kodak mentioned previously, but there are so many more).
Because that's the hallmark of business-minded people being at the helm and not engineers, which has been an issue for Google for a long time: the greatest inventions ever seen will be squandered in terms of the org itself because management can't see beyond the next few quarters and won't invest properly in it.
??? Xerox and Kodak are immortal household names that dominated their niches for decades, what level of "success" would satisfy you? what are you lamenting, that they didn't completely enshittify their products while they were on top and torch their brands to the ground for a little more revenue?
Well, strictly speaking modern LLMs are transformers (not counting state space models like Mamba, for the moment). Not sure what hidden Markov chains have to do with modern LLMs.
Well, LLMs are not AGI. They have serious limitations [1] and honestly my fear is that it’s too soon. I agree Google is ruled by managers (that’s why I hated it), and my fear is that the managers have FOMO and want to push to productize asap even tho the tech isn’t ready yet.
Look what happened when Google tried to throw an LLM in to search. Absolute shitshow. That’s not ready to become any kind of product!
If they kill R&D now to focus on productizing something that is half baked, they will fail to develop those new inventions which might get us to AGI. When I worked at Google X Robotics I was hired on to the remnants of the last research team, which was dissolved six months after I started (I was moved to hardware test engineer). Our subteam really wanted to research multi-finger grippers but we got overruled, so the robot had to do everything with a two finger pinch gripper. Which is fine for research but absolutely unsuitable for real world tasks. It couldn’t even operate a spray bottle without special attachments and they thought it was going to clean people’s homes!
[1] I am sharing this one a lot lately but I’m very moved by Yann LeCun’s arguments about the limits of autoregressive approaches here. As a robotics engineer I have been dismayed at all the attention LLMs are getting despite serious limitations that make them generally unsuitable to solve some of the most important problems in robotics. https://youtu.be/1lHFUR-yD6I
> my fear is that the managers have FOMO and want to push to productize asap even tho the tech isn’t ready yet
This is exactly it. With the limitations ChatGPT is encountering around safety and hallucination, Google probably should've just said "we're working on something awesome - hold on" and kept plugging away before releasing, instead of ex-Product CEO making them release something now, even if half of the demo video is fake.
I'm using chatGPT to Google something for around a year (or whenever bing browsing became available), and I'm yet to set a single hallucination based on web search results. May be Google is just not very good at this.
Neither is Yann (who has since proposed a different architecture / vision which is yet to take off), but my comment was meant to highlight a recent claim from another accomplished researcher in the field.
Right but I’m not talking about claims, I’m talking about arguments. As a robotics engineer running a farming robot project, I am all too aware of the serious differences in computational challenges and dataset availability between text and image data on the web and the kinds of problems that once faces in robotics. LeCun doesn’t just claim LLMs aren’t up to the task, he provides a detailed list of provable shortcomings which I feel overall make for a compelling argument. I’m hopeful that his JEPA may shed some light on possible solutions, but it’s also a fact that one can find issues with proposed engineering solutions even if they don’t have their own better solution. It you say you have a faster than light space engine designed, one wouldn’t need to have their own functional design to show why yours didn’t work (although such expertise is always helpful). And, well, he did invent convolutional neural networks, tho as I say such expertise is not required to raise valid arguments against some proposed solution.
fwiw, I agree with Yann & you (and many others!) on the shortcomings of LLMs (not from a position of authority, but as a matter of opinion).
I meant to counter-balance OP's point in that there are other equally accomplished individuals who aren't swayed by Yann's (and others accelerationists like Andrew Ng) arguments or claims.
Same experience at EDR. It was a great research platform, but nowhere near ready for the real world. Compute, power, functional safety... The "real" working robots of today are completely different. A successful product from EDR would have looked completely different.
God, yea. I've flipped between R&D and product development a few times in my career, sometimes at the same company, and it's a really rough transition even for an individual experienced in both. I can't imagine trying to flip a whole research team to make products is going to go well, especially when the products are getting a ton of well-deserved bad press and a lot of those researchers were coming from academia rather than elsewhere in industry in the first place
I tend to agree. I'm curious whether this is Deepmind saying "I think we could do things better, let's do this ourselves" or the leadership of Alphabet saying "Get these ivy league intellectuals to prioritize productionizing products!"
Would seem far more sensible to allow Deepmind to continue to release hit after hit in the ML research world, and simply embed "fly on the wall" PM's into their org that can independently productionize any golden nuggets they happen to create.
I feel it's more of leadership of Alphabet saying, "Our people literally invented transformers, so how come we're at the bottom of AI race instead of at the top? A random non-profit took out research and run with it, and they're the hottest company in the world now. This is unacceptable![0]".
Bad idea. People good or lucky enough to land in R&D like doing R&D. Force them to be product people, I expect most of them will leave.
I also think people and society also give themselves way too much credit for their successes. There's plenty of smart hardworking people out there who continue to contribute but never stumble upon a unicorn.
To a large extent its luck, a much larger contributor than people realize.
All you can do is to play the game, consistently contribute and work hard on R&D and products and you improve your odds of stumbling upon success. But it's never guaranteed.
Yes, but that doesn't explain how OpenAI able to walk and chew bubble gum at the same time, unless you think there is some extra spark that Google is lacking.
I think OpenAI was chilled out, and mostly lucky. Chilled in the sense that Google seems to be too full of itself, and their practice of publishing groundbreaking research without actually publishing anything other people can use is, frankly, annoying. OpenAI managed to leapfrog them by slapping a chat interface on a tuned GPT-3 and putting it on the Internet.
The chat service bit is them being chill, but the real spark was the model. I say they were lucky, because AFAIK back then no one expected LLMs to show so many and so advanced general capabilities. This took everyone by surprise, and since people could already play with it, ChatGPT took off on its own - it had so much real, transformative value, that it spread out with zero marketing. That's a rare, bona fide case of "word of mouth", it was just that useful. But that wasn't a strategy, that was luck.
To their credit though, OpenAI turned this early win into an opportunity and is excellent at exploiting it. Being small helps.
> it had so much real, transformative value, that it spread out with zero marketing
IMHO, in retrospect, the failure gradient of early LLMs is underappreciated in driving adoption.
Windows 95 failure: blue screen with inscrutable error code. Everyone noticed that.
LLM failure: run-around non-answer (user shrugs and tries again) or confident and plausible incorrect answer (user doesn't recognize this without research).
Essentially, the ways in which LLMs didn't work were the most hidden and hardest to discover failure mode.
Which was perfectly tuned for the "I'm going to try this thing for 5 minutes and be amazed" first impression.
Which allowed subsequent generations to backfill the capability gaps.
Tl;dr - We shouldn't underappreciate quiet-failing as a product adoption driver.
>We shouldn't underappreciate quiet-failing as a product adoption driver.
I'm certain it drives a lot of early user retention in the short term, but I feel strongly that this is ultimately a very myopic view which will prove catastrophic in the long term in much the same way that swallowing exceptions at runtime builds compounding technical debt you'll have to reckon with sooner or later
more broadly, there is just so much handwaving away all the black box parts of deep neural networks that are completely opaque and there seems to be very little interest in building the tooling to properly visualize, explore, and DEBUG latent space; until those priorities change this whole thing is a huge time bomb.
imagine if instead of coming with full memory dumps and diagnostic codes, BSODs just said "sorry, your computer had an oopsie!", and not a single engineer at Microsoft had a complete understanding of why the BSOD happened in the first place; sometimes it just does that! whoops!
> imagine if instead of coming with full memory dumps and diagnostic codes, BSODs just said "sorry, your computer had an oopsie!"
So, MacOS? ;)
In all seriousness, I wasn't opining on the usefulness of opaque/hidden errors, but rather the effectiveness of them.
In an alternate reality where the first LLMs instead spit back an error reference instead of English, I don't think we would have seen nearly as rapid mass market adoption.
And, not to put too fine a point on it, early conversational LLMs and image diffusion models were literally trained so their junk output is as plausible as possible.
GPT-3 was out for years already for Googlw to see, but with what OpenAI saw of it they began training an expensive GPT-4, before chatgpt success, maybe started before RLHF.
Agreed. As an example of that, Meta's kept its research org, FAIR, still doing fundamental research. Research orgs are great at demos, but actual productionalization takes a different mindset.
Referrals are vetted by global teams who are just checking boxes - it's very common to have to reach out to the recruiter for the role directly to resurrect a rejected referral.
Definitely the people working on the model. It ultimately doesn’t matter what the users want because you can’t arbitrarily deliver an experience. You can only deliver what it’s possible to extract from the model, so growing the possible things the model can do well is most important.
In my mind you could not be proving that we need product people more. They'd never say "It ultimately doesn’t matter what the users want" - they'd say "let's find a way to build what users want" not "let's grow the possible things a model can do well".
I have experience in both R&D and product. Both need different approaches to work. The goals of people working on the model will be different from product people. As mentioned by the other user, a product team can look at things produced from research and see how it can bring it to users.
If engineering/research is all mattered we would have maybe two order of magnitude more successful products or companies. Because product-market fit is a thing we don't have any successful research turning to successful product.
Must transfer value, and the guy in charge of the company is not good at allocating the company's resources to do that with an eye on long-term results.
One thing that classic Google did right was embed researchers into product groups. It's true there were always some teams that were pure research but for the most part researchers were working within a product group.
Then some acquisitions and internal musical chairs and it became less like that. Now I'm not all doom and gloom like this article (although with DeepMind why not leave well enough alone? They do excellent research). But, it does seem suboptimal to pivot all the way to AI Product Factory... were there no other existing product factories they could have turned instead?
The "Hybrid Approach to Research" paper describes how "Google Research" first started when Google was mostly, if not entirely, about Search and "Research" was part of the Search organization.
In these days, there were no "pure research" roles, nor were there formal designations for "research scientists" as a career ladder at Google. There were "SWEs" and in some cases "Members of Technical Staff."
Since then, "Research" became its own organization or "Product Area" at Google (i.e. the equivalent of a company division). "Google Brain" was also created. Deepmind was acquired. All of these existed simultaneously, however Deepmind remained as an organizationally separate entity. In this era, the "Research Scientist" role was created, which generally existed exclusively within "Google Research." A large span of this era had John Giannandrea ("JG") at the helm of the Google Research org; (note: Giannandrea left to head and build Apple's "AI/ML" organization, which includes Siri, a few years ago.
After JG's departure, Google Brain and Google Research were brought together under the common leadership of Jeff Dean, as an organization called still called "Google Research" with a branch still called "Google Brain." For perspective, it may be useful to consider too that the size of "Google Research" in staff headcount here measured in the several-thousands. This configuration existed for the last few years, with the latest changes being the merging of "Google Research" and Deepmind into "Google Deepmind."
-
I am a Xoogler, formerly from this product area. One of the things I and at least a few others observed was that "Research" was becoming defacto synonymous with "Machine Learning / AI," yet not all of Google's storied research accomplishments, or problem areas, are limited to Machine Learning and AI.
In the last few years, Google made its public statements of being an "AI-first," previously "mobile-first," company in recognition that it would be incorporating and leveraging ML and AI technology across all of its products and services.
This raised a significant question: What should "Google Research" or Research at Google be if product areas across Google began full incorporation of AI/ML technology and methods in their products? What if they incorporated their own AI/ML teams? If Google was truly successful at becoming AI-first, how should "Google Research" define and focus its organizational purpose, research portfolio, and show its value when Moonshots/X also exists within Alphabet? Over time, there were many parts of "Google Research" and research at large across Alphabet that felt that their purpose, or at least their individual reason for joining, was to do "pure research," yet this is not how the organizations started at all in the beginning. Many researchers and teams also knew that for practical reasons (e.g. promotion) that they generally needed to present and align their work with things like product launches with partner organizations.
I suppose we are seeing some of the answer to this with Google DeepMind stating that they will be aligning more strongly with creating AI products, but in addition to the question of what happens to foundational research (for AI), what happens to foundational research in non-AI areas for Google and Alphabet?
I agree: hybrid teams with a diversity of product/research skills at the team level is the way to go. It is thinkers and doers that need to come together.
It is way easier said than done, though. You need true buy in from a ton of stakeholders — employees being the primary ones. And people get set in their ways.
I do like a product bias though. Not because it is more valuable somehow but because it provides the applied scientists deeper exposure to the problem space, early and often.
> Hassabis says that he’s learning more about introducing products and that Google’s product teams, in turn, are dealing with the novel challenges of generative AI, which has the potential to behave unusually when placed in the hands of the general public.
This part isn't rocket science.
Step 1) Post on 4chan and SomethingAwful "What is the worst thing you could do with genAI? Go."
Step 2) Test your beta product against all the answers you get.
the tricky part with systems like these isn't to find and fix the worst things that people can come up with on 4chan because they're by definition obvious. The much trickier part is finding the little problems that way more people run into and that most people might not even immediately recognize or report.
And that is a very complicated science in particular with something that can be as fuzzy and intransparent as generative AI.
This has been going on since long before[1] the recent AI craze.
IMNSHO it seems Google just cannot miss an opportunity to mess up the basics in the quest for amazing and then fail at amazing or cancel it just as they are about to achieve it. And, ironically this has transformed their search engine from unbeatable leader in its field to something much closer to what it replaced.
> While no one is getting as much computing power as they want, the supply is tighter for teams engaged in pure research, say the former employee and others familiar with the lab.
Good! Maybe they will focus on researching how to make these things more compute efficient.
true. I wonder how much energy in food/farming to develop a 6 yr old human is required vs. the amount required to run a hojillion GPUs running the latest generative algorithms
Most people can probably answer about some subset of questions better than GPT4 can, but I don't think there's a human alive who could answer nearly as competently on >50% of the questions if gets asked. So I don't know why you'd benchmark it against a 6 year old. If you compare the carbon impact of training one of these to the carbon footprint of the average American family, it's an incredible deal in terms of utility.
Hasn't everything in ChatGPT - been posted by 6 year olds on reddit, and ChatGPT is simply a very impressive indexer and query interface? :-)
The real question is ChatGPT a better information retrieval tool that the old Google search interface before they dumbed it down?
For me the main differences are that for ChatGPT it summarises across multiple sources - sometimes good, sometimes not, and the refinement of queries feels much more natural with it's use of context.
Though I often find myself fighting both the new Google search interface and ChatGPT to try and get the right answers to the specific area I want.
Looking back, I feel like the AlphaFold 3 launch a month ago was a precursor to this move. The public-facing side of AlphaFold 3 ('AlphaFold Server') is severely constrained; if you want the novel parts around drug binding prediction you need to pay Isomorphic Labs instead.
Let's not forget that Demis Hassabis (DeepMind CEO) created Theme Park
so he knows how to create products. I have full confidence in his leadership.
Buy more Alphabet (GOOG) shares!
Saw him briefly at my first job at Lionhead Studios, and also worked with Alex Evans (Media Molecule cofounder, coauthor of InstantNGP, legendary demoscener, ...) there. Pretty amazing how much talent was buzzing around there.
Demis understands the “customer” and can use everything he has learned in the last 20 years to build something incredible. If he can build a great/successful game with low resources, he will smash a consumer product with unlimited resources and excellent people.
Hadn’t these two AI orgs within Google been fighting over resources for a long time? At the end of the day that’s just counterproductive. A merger was all but guaranteed and it’s clear given current stock market sentiment why the product team was chosen. Doesn’t mean I don’t feel bad for the Deepmind researchers impacted. The genAI hype is sparing no one, not even the foremost AI labs in the country.
> In May, the lab released a new version of AlphaFold, a landmark tool for predicting protein structures. Hassabis says it could develop into a $100 billion business, but some people at Google have questioned whether he should be dedicating so much time to it.
Wonder what the timeframe on that speculative return is?
Good on them for cracking the whip on those intellectuals and extracting value out of them. Even smart people don't get a pass when it comes to driving shareholder value and fast ROI.
> Researchers inside the AI unit have told colleagues they’re proud of their advances on Gemini, such as its “context window,” the amount of information the system can analyze at once. This is particularly useful to a company whose enormous amount of data is one of its key competitive advantages.
what does a large context window have anything to do with google's data moat?
I suppose superficially, for retrieval augmented generation use cases, the more data you have (and the better you are at retrieval), the more useful extra space in the context window is because you put things from retrieval into the context of the model to do the generation. That said, literally everyone has more than enough data to completely dwarf the context window of any of the existing models so it seems true but irrelevant.
One thing about gemini that may be a benefit here is not just the size of the window, but the fact that the context window seems to be better utilised by the model. GPT-4 seems to have a characteristic where the start and end of the context window are much more important to the model than anything in the middle, meaning that if you stuff the window with retrieved data, things in the middle of the context get ignored by the model. Istr that is not the case with gemini, which takes more notice of things in the middle of the window. Maybe attention is all you need.[1]
Meta needs to emulate this because they're sinking tens of billions without being focused enough on delivering profits with all that hardware they're spending treasure on.
Meta did $12.4B profit in Q1 2024 which was 116.7% increase year-over-year. They have a lot of treasure to spend and the treasure chest keeps getting bigger.
Pretty untethered take. Meta has seen one of the largest growths in profit in the past few years. Their improvements in AI have made far more profitable their AI ads targeting business
Essentially Google has no idea how to leverage their AI as Gemini for search just does not work.
Apple went with deep AI integration into OS in hope to sell more devices and Microsoft went full blown corporate + windows devices. Google can try to integrate it maybe with Chrome OS to sell more devices? They are also trying with various providers (like Samsung) but nature of Android is that people might just go with basic apps and stuff.
I didn't think the AI hype was overdone until I saw AAPL jump so much after announcing AI features. Looks nice, but it's not going to make anyone switch to iPhone, and there's no lock-in. iMessage must be way more valuable, and that advantage is maybe going away.
That's the thing - apple stocks raised due to AI features but it is basically a speculation - if it does not move sales of iPhones, the stock will go down.
I think MSFT is the best positioned long term - especially when they start producing their own chips - as they have the right moat to vendor lock in. Add to that the fact that AWS missed the AI boat completely and their could gain market share from AWS too.
The problem is that none of Google's products are unified into a fully used ecosystem.
For example with Apple they integrated everything together seamless into the OS - granted I am not sure if people are using their email app or calendar that often. But with AI they integrated it all together at least.
With Microsoft they benefit from their tight integration between Office suite, Outlook, teams (and calendar integration between outlook and teams is quite convenient) etc. They only have issues with consumer products as they are unable to achieve the same level of integrations as they achieve within the corporate - corporate Windows instances with laptops and stuff are corporate to Apple products for consumers. Microsoft does not have user facing products, but their enterprise solutions are nicely connected to each. And new services like Loop or Copilot are just naturally expanded on that.
But Google? I literally use Gmail but only for emails. Chrome for browsing but I have no integrations between Chrome and Gmail aside the account overall in my flow. They have their streaming service with Youtube Premium but it is not really that connected to overall other infra or services - unlike for example Apple, that is offering their Apple One subscription. App Stores? Google Play exists in its own universe that has no relation to other google services either. And that's without AI stuff.
There is something missing between google services.
I dunno. Android, Chrome, ChromeOS, Workspace. There’s a lot there that’s knit together. Compared to what, Safari, Pages and Messages? Teams, Word, and Outlook?
They all have their issues. Maybe the crux of it is chat. If your work is integrated into your chat — the day to day online social space — it’ll feel “integrated”
The difference is that people who use Android don't use ChromeOS for most of the part. Or ignore Workspaces altogether.
With Apple their are not going after Safari and whatever integration - they are embedding stuff into OS across the devices. It helps that they have their Apple One subscription and cloud drive integration. Also MacOS, iOS connectivity etc. Search across all your devices and files, analytics.
With Microsoft the whole Office 365 integration is extremely tight - share files, use analytics dashboard, integration with sharepoint, outlook and calendars and so on.
Just like with GCP and other Google's offerings - they are good separately and people are fine using them independently. But they don't work nicely together.
Overall probably a good move for Google. They were were spending 100s of millions of dollars a year to train RL agents to get good at playing Qwop for close to a decade.
I strongly disagree. Research is how you stay ahead of your competitors, yes even silly sounding research like training RL agents to play Qwop!
Research is how Google invented the transformer that underlies so many current Gen AI models.
Pivoting research to products is exactly the kind of short term thinking consultants or mercenaries would propose, get promoted off of, and leave just before the consequences start rearing their head
They did stay ahead of their competitors. In a lot of ways they still are. In a "making profit from AI" way, they're not. But that's what you hire product people for. They had the SOTA translation system of the entire world for a literal decade and didn't monetize it enough to blow their competition to dust. That's not the fault of researchers, and research shouldn't halt or be redirected to profit as a result. Rather, Google should get some better managers / product people to turn the magic their researchers create into profits.
They are ahead in terms of AI infra and capacity. Research it's not as easy to tell. They are behind in terms of shipping products. Yes a lot of researchers got hired away, but they still up there in the race
In a PR perspective, at least AlphaGo did a prominent job for Google. It failed to keep its prestigious status but at least it's a leadership/product side issue, not DeepMind's.
That stuff gave Google the appearance of "nerds playing with their toys." Sure it's worthwhile research, but it's not great how many things Google announced that close to 0 people could use. They're trying to be more user-oriented now, and because of Cloud, their users are businesses.
No. AlphaGo et al gave the appearance of "world class researchers achieving state of the art on problem classes that humans have been devoting lifetimes of study for thousands of years". the current desperate LLM scramble, including stuffing shitty, hallucinating, half-baked paraphrase spam to the top of every Google search result page, gives the appearance of "panicked flailing and tacit admission that high level decisionmaking has been captured by terrified MBA types"
There's a good long middle ground. Search results do seem desperate now, but they got into this position in the first place by failing to execute when they had the advantage. None of Google's customers have ever cared how well an AI can play Go. If there's one place the bragging could've translated to profits, it'd be Tensorflow dominance, which didn't happen.
RNN gameplay agents and reinforcement learning are a thousand times more interesting than stochastic parrots that rephrase Google for you, you'll see once this hype wave finishes dying out in a couple years
It seems to be difficult to turn the pure research back into new products. Apple famously got lots of ideas for free from Xerox PARC. Google researchers wrote the Attention Is All You Need paper and they're now desperately playing catchup because they couldn't convert it to any kind of product. There's nothing wrong with companies investing in pure research, but these large companies sometimes are unable to take advantage of the research. The people running the business want to keep doing what got them successful, not some new experimental thing that might not work.
> Google researchers wrote the Attention Is All You Need paper and they're now desperately playing catchup because they couldn't convert it to any kind of product.
This isn't true. The transformer underlied Google Translate for a long time. They just didn't monetize Google Translate heavily enough. It's still one of the best translation services out there. And its ability to translate real-time conversations has been around for years now.
Yeah being first doesn’t mean you win automatically. There’s a story about the Ramones playing a show at a famous club in NYC and everyone in the crowd went home and started bands that became way more successful and famous than who they were trying to be like. …I think blondie was one of the bands that came out of that crowd.
