I feel this so much. I woke up at 1am last night stressing about AI and my potential lack of productivity. 2am rolls around, and I could not get back to sleep so I worked till 4:30am. Slept fine till 7:00am. AI has been causing a lot of stress for me and many others lately. My biggest source of stress is what will AI transform the human work world into by the time my children need a job? Most of us live in a capitalist society so AI utopia is right off the table.
The more difficult it is for humans to consistently and accurately compare model outputs the more opportunity there is to spread FUD (Fear, Uncertainty, Doubt). Considering valuations of these companies and the astronomical investments being made, a sabotage campaign with bots or paid users on reddit, twitter, YouTube, or whatever socials could go a long way towards knocking market cap off the competition. Not saying that's happening, just saying its an obvious target. Even if the goal is not nefarious, people with a perceived bad experience are 2-3x more likely to complain. So even without bad actors involved, a new model may need to be significantly better in order to break even on the old net promoter score.
If data center water use is such a concern, why not require that data centers invest in closed-loop cooling systems? By closed-loop, I'm talking about re-condensing evaporated water and allowing the water to cool. Cooling the water would be more expensive in hotter environments, but still achievable. These data centers seem to have wild amounts of money for investment, why not just mandate conservation requirements?
We should just charge a fair price for water. Something that covers capital, operating, and decommissioning costs. No need to pass specific regulations or add legal complexity. It would solve itself. Imagine any other service saying "Oh no, we have too much demand, we need to make it illegal." Just put out bonds, and build up capacity.
Condensing/cooling the water takes even more electricity though. So you're trading water savings for increased energy use. Maybe OK if it's all renewable, but in most areas it's not.
Anthropic is loosing the good will they built with devs faster than they built it. Its the anti-competitive and anti-opensource behviors that will erode their dev customer base. No clue how much of Anthropic's revenue is based on devs paying for claude subscriptions, but they are going to lose that quickly.
I would have jumped ship, but OpenAI saying "hold my beer" when Anthropic declined the Pentagon's safeguard removal demands is the only thing that has prevented me from jumping ship. I've considered Chinese AI services but I'm too concerned with data (proprietary code) exfiltration.
Then you should consider alternative LLM API providers, who are not based in China but host the same (or roughly the same, depending on the quantization and other deployment specifics) models as your "Chinese AI services".
I only use copilot for the occasional auto-complete suggestion. I'm betting I could run a lightweight local LLM with llama.cpp to get similar functionality. Maybe this would be a decent replacement https://github.com/TabbyML/tabby
I'm too concerned with data exfiltration to use many AI services unless their terms of service state they will not use your data for training or anything else. Zero retention is what I'm looking for. I care because I frequently work on proprietary code that I do not personally own (as most employed software devs do). So if I am using an AI service with proprietary code, I want assurances that there is no retention and no training happening. From my American perspective Chinese companies don't have the best track record of not training on proprietary information. I guess LLMs in general are trained on a lot of proprietary information. I just don't want to be responsible for unintentionally exfiltrating my employer's proprietary code.
My recent frustration with Claude has been it feels like I'm waiting on responses more. I don't have historical latency to compare this with, but I feel like it has been getting slower. I may be wrong, and maybe its just spending more time thinking than it used to. My guess is Anthropic is having capacity issues. I hope I'm wrong because I don't want to switch.
There was a really good point in this podcast episode about the speed of LLMs. They are so slow that all of the progress messages and token streaming are necessary. But the core problem is that the technology is so darn slow.
As someone who both uses and builds this technology I think this is a core UX issue we’re going to be improving for a while. At times it really feels like a choose 2+ of: slow, bad, and expensive.
I feel like there have been enough hyperbolic claims by Anthropic, that I'm starting to get some real Boy Who Cried Wolf energy. I'm starting to tune out, and assume it is a marketing ploy. Trust me, I'm an Antropic fan, and I pay my $200/month for max, but the claims are wearing thin.
Thank you. I have the same card, and I noticed the same ~100 TPS when I ran Q3.5-35B-A3B. G4 26B A4B running at 150TPS is a 50% performance gain. That's pretty huge.
reply