Yeah, Gemini is a pretty junk product. You can't use it for anything useful. Both Claude 3 Opus and ChatGPT 4o are much more useful. I personally use Claude first and then ChatGPT. Claude will do whole-program ports. It's quite nice. I used it to transform a crappy old node.js script directly 1-1 to Python and then I got it to refactor stuff and it was great. I could have written the code easily but tediously, and it was trivial to read so this was fantastic.
Gemini Advanced just gave up. I'm told the API is better than the interactive app and I hope so because the thing is total garbage. I guess when you spend all your time making sure that stakeholders are aligned you don't have much time to write good code.
I tested out 7 different LLMs today with a seemingly simple task: Write a Python script that takes a protobuf binary and produce a template proto file with placeholder names. Gemini did the worst, with even the import being straight out wrong. GPT4 got slightly closer but still had a bunch of just straight up wrong usage. Surprisingly, Yi (Chinese or Japanese?) was the only one other than Claude Opus that gave code which actually ran on the first try (but still slightly wrong).
I ended up writing the code myself in Golang (which I’m more familiar with) in about half the time it took me to debug the scripts pumped out by the LLMs. There is still a long way to go until we can be certain they save more time than waste.
Interesting. I would have assumed they're good at that. I usually do the whole thing collaboratively with the LLM. So far, the tasks I use Claude 3 Opus for it seems to get right. Thank you for sharing your experience. I shall try Yi.
Gemini Advanced just gave up. I'm told the API is better than the interactive app and I hope so because the thing is total garbage. I guess when you spend all your time making sure that stakeholders are aligned you don't have much time to write good code.