Looks like he aligned himself with the wrong folks here. He is a system builder at heart but not an expert in chip design or EDA. And also not really an ML researcher. Some would say he got taken for a ride by a young charismatic grifter and is now in too deep to back out. His focus on this project didn’t help with his case at Google. They moved all the important stuff away from him and gave it to Demis last year and left him with an honorary title. Quite sad really for someone of his accomplishments.
Not a ML researcher, too? He was working on neural networks in 1990. Last year he was under Research and now reports directly to Sundar. What do you know that we don't?
I don't think he got taken for a ride. Rather, he also wanted to believe that AlphaChip would be as revolutionary as it claimed to be and chose to ignore Chaterjee's reservations. Understandable, given all the AlphaX models coming out around that timeframe.
I mean Jeff Dean is probably more ML researcher than probably 90% of the ML researchers out there. Sure, he may not be working on state of the art stuff himself; but he's too up the chain to do that.
What are you even talking about? Jeff had a hand in TPU, which is so successful that all other AI companies are trying to clone this project and spin up their own efforts to make custom AI chips.
> Some would say he got taken for a ride by a young charismatic grifter and is now in too deep to back out.
Was the TPU physical design team also taken in? And also MediaTek? And also TF-Agents, which publicly said they re-produced the AlphaChip method and results exactly?
What did the TPU physical design team say about this publicly? Can you also point to a statement from MediaTek? (I've seen a quote in Google blog, but was unable to confirm it). Who in the TF-agents team has serious physical design background?
Are you really suggesting that the TPU team does not stand behind the graphs in Google's own blog post? And that MediaTek does not stand behind their quoted statement?
The court case provides more details. Looks like the junior researchers and Jeff Dean teamed up and bullied Chatterjee and his team to prevent the fraud from being exposed. IIRC the NYT reported at the time that Chatterjee was fired within an hour of disclosing that he was going to report Jeff Dean to the Alphabet Board for misconduct.
Markov’s paper also has links to Google papers from two different sets of authors that shows minimal advantage of pretraining. And given the small number of benchmarks using a pretrained model from Google whose provenance is not known would be counterproductive. Google likely trained it on all available benchmarks to regurgitate the best solutions of commercial tools.
As Markov claims Nature did not follow their own policy. Since Google’s results are only on their designs, no one can replicate them. Nature is single blind, so they probably didn’t want to turn down Jeff Dean so that they wouldn’t lose future business from Google.
"Settled" does not mean "Dean did nothing wrong". It means "Google paid the plaintiffs a lot of money so they'd stop saying publicly that Dean did something wrong", which is very different.
Good question. I thought the tpus were a way for Google to apply pricing pressure to nvidia by having an alternative. They are not particularly better (it’s hard to get utilization), and I believe Google continues to be a big buyer of nvidia chips.
Doesn’t look like it. In fact the original paper claimed that their RL method could be used for all sorts of combinatorial optimization problems. Yet they chose an obscure problem in chip design and showed their results on proprietary data instead of standard public benchmarks.
Instead they could have demonstrated their amazing method on any number of standard NP hard optimization problems e.g. traveling salesman, bin packing, ILP, etc. where we can generate tons of examples and verify easily whether it produces better results than other solvers or not.
This is why many in the chip design and optimization community felt that the paper was suspicious. Even with this addendum they adamantly refuse to share any results that can be independently verified.
> Yet they chose an obscure problem in chip design
It is not obscure (in chip design). If anything it is one of the most easily reachable problems. Almost every other PhD student in the field has implemented a macro placer, even if just for fun, and there are frequent academic competitions. A lot of design houses also roll their own macro placers since it's not a difficult problem and generally adding a bit of knowledge of your design style can help you gain an extra % over the generic commercial tools.
It does not surprise me at all that they decided to start with this for their foray into chip EDA. It's the minimum effort route.
Sorry. I meant obscure relative to the large space of combinatorial optimization problems not just chip design.
Most design houses don’t write their own macro placers but customize commercial flows for their designs.
The problem with macro placement as an RL technology demonstrator is that to evaluate quality you need to go through large parts of the design flow which involves using other commercial tools. This makes it incredibly hard to evaluate superiority since all those steps and tools add noise.
Easier problems would have been to use RL to minimize the number of gates in a logic circuit or just focus on placement with half perimeter wirelength (I think this is what you mean with your grad student example). Essentially solving point problems in the design flow and evaluating quality improvements locally.
They evaluated quality globally and only globally and that destroys credibility in this business due to the noise involved unless you have lots of examples, can show statistical significance, and (unfortunately for the authors) also local improvements.
That’s what the follow on studies did and that’s why the community has lost faith in this particular algorithm.
> Most design houses don’t write their own macro placers but customize commercial flows for their designs.
Most I don't know, but all the mid-to-large ones have automated macro placers. Obviously, the output is introduced into the commercial flow, generally by setting placement constraints. The larger houses go much further and may even override specific parts of the flow, but not basing it on an commercial flow is out of the question right now.
> The problem with macro placement as an RL technology demonstrator is that to evaluate quality you need to go through large parts of the design flow which involves using other commercial tools.
Not really, not any more than any other optimization such as e.g. frontend which I'm more familiar with. If you don't want to go through the full design flow (which I agree introduces noise more than anything else), then benchmark your floorplans in some easily calculable metric (e.g., HPWL). Likewise, if you want to test the quality of some logic simplification _in theory_ you'd have to also go through the entire flow (backend included), but no one does that and you just evaluate some easily calculable metric e.g. number of gates. These distinctions are traditional more than anything else.
Academic macro placers generally have limited access to commercial flows (either due to licensing issues or computing resource availability) so it is rather common to benchmark them in other metrics. Google paper tried to be too smart for its own good and therefore incomparable to anything academic.
