I see claims that it makes you 7-8x more effective. That hasn't been my experience. Maybe 5-10% improvement at best. Instead of Googling myself, LLMs can sometimes give me the answer more quickly than I could find it.

Once you ask it anything not easily scraped on the internet it hallucinates like crazy and sounds so confident about it. Maybe I'm just horrible at prompting, but I can't help but feel we're still N breakthroughs away from having it really impact dev jobs.

People that are 5-10x more effective at hard skills like programming, well I'm just convinced they weren't a good programmer to begin with and are doing easy problems. Basically, anyone that says that is telling on themselves.

There are plenty of tasks I'd normally procrastinate on, or be reluctant to do, because they're tiring, boring, or emotionally difficult for reasons specific to myself. That is, cases when I have the knowledge and the skills, but lack the willpower or composture (or glucose / caffeine in my bloodstream). Using GPT-4 for with that kind of work isn't saving much time vs. what I could do, but it is compared to what would actually happen, which is either me procrastinating on it, delaying it for a better time (next morning, day with less meetings, etc.), or suffering a 2-10x performance penalty from having to fight through my own emotional blocks.

On the net, this isn't making me 5-10x more effective at work. It's probably not even 2x, short-term. Mid-term, 2-5x would be possible, because all the things I did earlier than later add up. Time will tell.

Am I telling on myself here? Maybe. Sorry not sorry. I am a human being, with a human brain, which means some things that should be easy for me, become hard for unrelated reasons. GPT-4 is one of many tools I have to overcome such challenges, but it's a particularly powerful and versatile one, so I'm happy that I can use it.

(Also, INB4, I have access to company-approved deployment on Azure, so I'm in the clear with using it at work.)

But if you're outsourcing a significant portion of your overall work (day-to-day, not just off-days) then that's more what I'm getting at. The people that are like "I couldn't imagine programming without copilot" or such. Every one of those that I've met is missing important base knowledge about programming in general that ends up making a lot of technical debt for themselves. The "build fast and break things" style realistically only works if you have a certain level of expertise OR the project doesn't need to be robust. Former because to actually be fast you need to have a good picture of the whole or else you end up chasing one thing to the next and your overall path is far slower but might seem faster because you're sprinting the whole time.

It can make devs 5-10x more effective at certain self-contained programming tasks, even if they aren't 5-10x more effective at their job overall. If you have to write a script in a language you haven't used before and it needs to interface with an API whose documentation you haven't read, and the script isn't mission critical but a "nice to have", GPT can massively shorten the learning curve required to just get this work done. Being a good dev doesn't excuse you from having to pay the cost of learning the syntax of this language and basics of the documentation more thoroughly than if you didn't have GPT.

Similar to anyone who says, “When you'd have a hard time googling something, you ask the LLM.”

Just my opinion.

I'm not saying it will never happen, just that it will be very obvious if it does. 5-10% sounds about right to me.

This means that I went from 0 opensource commits to 4000 since chatgpt came out.

Not just that, but I've gotten not only more adventurous, but have the time to consider doing drastic refactors and spend much more time thinking about my software.

I won't call it 10x or 100x, because that wouldn't mean anything, but surely it is a paradigm shift for me, completely world changing.

I never heard of type hints before but ironically I use them on everything now, since it's easier to lint.

  - generate comments (hit or miss, but at least it can rewrite my random notes into consistent notes)
  - generate type annotations
  - refactor "broadly" (say, "rename all variables to match the following style" or "turn this class into a dataclass like XXX" or "transform the SQL queries into builder queries using XYZ"). Often requires some manual work but it gets a lot of tedious stuff out of the way
  - reverse-engineer clean API specs by just pasting in recorded HTTP logs
  - clean up logs into proper enums by generating the regexps
  - write CLI tools to probe the system (say, CLI tool to exercise the APIs mentioned above)
  - generate synthetic test data
  - transform HTML garbage into using a modern component system / react
  - transform legacy react/js into consistent redux actions
  - generate SQL queries at the speed of mouth

Can you go on more about this, please? This sounds, frankly, heavenly, but the second sentence gives me pause. I guess it's not necessarily a question of how reliably it can "broadly" refactor but rather how broadly "broadly" is meant to be taken...

>- generate SQL queries at the speed of mouth

...and this? I'm not really a database guy, but I do keep hearing from them about how (eg, a database guy's) stateful knowledge of a database can result much, much more efficient queries than eg a sales guy with a query builder. Are the robut's queries more like the former or the latter?

I would highly recommend taking a piece of code (any code) and then just start experimenting. Here's a few prompt ideas:

  - make this a singleton
  - use more classes
  - use less classes
  - create more functions
  - use lambdas
  - rewrite in a functional pipeline style
  - extract higher order types
  - use fluent APIs
  - use a query builder
  - transform to a state machine
  - make it async
  - add cancellation
  - use a work queue
  - turn it into a microservice pipeline
  - turn it into a text adventure
  - create a declarative DSL to simplify the core logic
  - list the edge cases
  - write unit tests for each edge case
  - transform the unit tests into table-driven tests
  - create a fuzzing harness
  - transform into a REST API
  - write a CLI tool
  - write a websocket server to stream updates into a graph
  - generate a HTML frontend
  - add structured logging
  - create a CPU architecture to execute this in hardware
  - create a config file
  - generate test data
  - generate a bayesian model to generate test data
  - generate a HTML frontend to generate a bayesian model to generate test data and download as a csv
  - etc...

There really is nothing that can go wrong, in the worst case the result is gibberish. The code doesn't even need to build or be correct for it to be useful. These models are trained to be plausible, and even more importantly, self-consistent.

When prompted with code in-context, these things are amazing at figuring out consistent, plausible, elegant, mainstream APIs. Implementing them correctly is something I usually tend to do manually instead of bludgeoning the LLM.

Because of the points this is the nearest to the work that some colleagues do, I anakyze this point (but you could ask similar questions about many of the other points):

In my experience, writing correct SQL queries (which often tend to be quite non-trivial because of the internal complexity of the projects) typically involves a lot of knowledge about the whole system that my colleagues and I work on. Even if I could copy-paste this information, written down once, into the AI chat window:

- I seriously doubt that any of these AI chat bots would be able to generate a remotely decent SQL query based on this information, if only because these SQL queries look really different from what you would see in typical CRUD web applications (for a very instructive example think into the direction of ETL for unifying historically separated lines of business where you often have lots of discussions with the respective colleagues to clear up very subtle details what the code is actually supposed to do in some strange boundary cases that exist because of some historical reasons (which one wants to get rid of))

- even explaining what the SQL query is supposed to do would in my opinion take more time than simply writing it down. Even ignoring the previous point: it is very typical that explaining in sufficient detail what the code is supposed to do would take far more time than simply writing it. A lot of programming work is not writing some scaffolding of some CRUD app or implementing a textbook algorithm.

I find that many of the generative AI models (GPT-4, 3.5, even MPT-30B running on my laptop) are really shockingly good at SQL.

Paste in a query and ask it for a detailed explanation. I've genuinely not seen it NOT provide a good result for that yet.

Generating new SQL queries is a bit harder, because of the context you need to provide - but I've had very strong results from that as well.

I've had the best results from providing both the schema and a couple of example rows from each table - which helps it identify things like "the country column contains abbreviations like US and GB".

If you've found differently I'd love to hear about it.

I have not directly tried it (the employer does not allow AI chatbots for any application intended for production (i.e. more sensitive stuff), but only for doing experiments), but working on the code I very rarely had the problem that I could not understand what some single (SQL) line of code does in the "programming sense".

The central problem that rather occurs often is understanding why this line does exist and why things are implemented the way they are.

Just to give an example: to accelerate some queries, I thought some index would make sense (colleagues principally agreed; it would likely accelerate a particular query that I had in mind). But there exists a good reason why there exists no index at this table (as the respective colleague explained to me). This again implies that for ETL stuff involving particular tables, one should make use of temporary tables where possible instead of JOINs; this is the reason why the code is organized as it is. This is the kind of explanation that I need, which surely no AI can deliver.

Or another example: why does some particular function (1) have a rights check for a "more powerful" role and a related one (2) does not need one? The reason is very interesting: principally having this check (for a "more powerful" role) does not make a lot of sense, but for some very red-tape reasons auditors requested that only a particular group of roles shall be allowed to execute (1), but they were perfectly fine with a much larger group of users being allowed to execute (2). Again something that no AI will be able to answer.

Part of the trick of making good use of LLMs is having a good instinct as to what kind of questions they will be able to ask and what kind of questions they are likely to mess up.

As a quick demo, I threw quite a gnarly real-world SQL query from this notebook - https://observablehq.com/@simonw/blog-to-newsletter#sql - into GPT-4 just now and gave it the following prompt:

> Explain this SQL query four times: first, as a high level explanation of what it does. Secondly, as some pithy highlights as to clever tricks it uses. Thirdly, as a step-by-step guide to exactly what each piece of the query does. Finally, provide some advice on how the query could be improved.

Here's what it produced in response: https://chat.openai.com/share/47b2f1c5-6b88-4ff2-b6d1-8f58da...

The high level explanation is, I think, excellent. I wrote that query and I'd forgotten the detail about how it sorts with entries first.

The tips for improvements are the weakest part, since they make assumptions that don't hold for this particular query (for one thing, SQLite doesn't have the ability to run stored procedures).

So far you've said that things like "This is the kind of explanation that I need, which surely no AI can deliver." but have not actually tried the system?

As GP asked: have you tried ChatGPT or similar LLMs? If not, go do it .. you may be surprised.

But have you tried? GPT-4 is great for coming up with hypotheses of why thing are the way they are.

In this case I might try a prompt along the lines of:

"Here is the schema for a table: (schema here). Here is a query that runs against it: (query here). Provide several suggestions for potential indexes that might speed up the query, and for each of those suggestions provide several hypothetical reasons that the index might be a bad idea."

I tried a version of that here, the results look pretty good to me: https://chat.openai.com/share/dc5d7007-cf05-4e5c-89c4-fc669b...

So, at each step, I do almost trivial transformations.

One key ingredient is that the DSL should include many "description" fields that incorporate english language, because that helps the model "understand" what the terser DSL fields are for.

Straight SQL is a crapshoot, and as you said, more often than not, either obviously or subtly broken or for another database. Which makes sense, considering how much different flavors of SQL it has in its training corpus and how much crappy SQL is out there anyway.

Another thing that helps is use extremely specific "jargon" for the domain you want to write queries for. Asking for "accrual revenue" and "yoy avg customer value" (yes, yoy, not year over year) often tends to bring back much higher quality than just asking for "revenue" or "customer value".

I use that to write code all the time, because ChatGPT can write the code, run it, get an error, then re-write the code to address the error.

Here are two recent transcripts whereI used it in this way:

- https://chat.openai.com/share/b062955d-3601-4051-b6d9-80cef9...

- https://chat.openai.com/share/b9873d04-5978-489f-8c6b-4b948d...

If you replace "lives of ordinary people" by "productivity improvement", you actually have a good (and rarely discussed) point:

> https://en.wikipedia.org/wiki/Productivity_paradox

The core hypothesis behind my belief is that introduction of computers to replace a class of tasks - up to and including a whole job type - is just shifting the workload, diffusing it across many people, where previously it was concentrated in smaller number of specialists. Think e.g. the things you use Word, Excel, Powerpoint, Outlook, etc. (or their equivalents from other vendors) for - before software ate it, a lot of that used to be someone's job. Now, it's just tacked onto everyone's workload, distracting people from doing the actual job they were paid to.

That would seem like obviously stupid way to do, so why would businesses all fall for it? I argue it's because even as shifting the workload makes everyone in the company less productive on the net, it looks like an improvement to accounting. Jobs with salaries are legible, clearly visible on the balance sheets. So is money saved by eliminating them. However, the overall productivity drop caused by smearing that same work across rest of the company? That's incremental, not obviously quantifiable. People and their salaries stay the same. So it all looks like introducing software and obsoleting some jobs saves everyone money - but then somehow, everyone is experiencing a "productivity paradox". But it's not a paradox if you ignore the financial metrics with their low resolution - focusing on what happens to work, it seems that IT improvements are mostly a lie.

I would also add that it may be of a net benefit that fewer roles are needed. But that net benefit overwhelming goes to the owners of the company. And that's what we've been seeing the last 30+ years the very wealthy have become much more wealthy while everyone else is worse off. ()

() growing wealth inequality is very complex and I'm sure would be happening anyway. I'm not saying computers cause wealthy inequality but they don't seem to be doing much good in fixing it either

That, but also:

- Secretaries were better at this work because that was their specialization, and they enjoyed efficiencies coming from focusing on doing a single specific kind of work.

- Those increments of extra work add up.

- Moving that work to everyone else means you now have highly paid specialists doing less and less of the specialized work they're paid for. In many cases (programming among them), context switching is costly, so the extra work disproportionately reduces their capacity at doing the thing they're good at.

This all adds up to rather significant loss of productivity.

Read up on improved crop yields for subsidence farmers who got access to weather reports through smart phones.

A lot of people are not starving now because of that one change alone.

I think that's a poor example. I was getting weather reports on pre-smartphone Nokias.

Farmers could have gotten weather reports on feature phones.

Computers have done amazing things for humanity. I would not want to go back to a computerless world...

Quick example. A doctor in the 90s would expect to look forward to less time wasted on paperwork thanks to computers but administration work is increasing. Likewise most professors at most universities will tell you they spend more and more time on administration. Shouldn't this be one of the primary things a computer could address.

My point is great gains brought by a technology such as computers may not translate to great end user benefit. Largely due to compensating inefficiencies elsewhere in the system.

So for example. It may be that chatgpt makes people 10x more productive. But if management gives bad direction it doesn't matter. If most software being developed is redundant then it doesn't matter. If most software is just making ads run faster then it doesn't matter. The technology needs to be appropriately directed to be of noticeable social benefit.

I think the NIMBYs have done pretty well in creating their own Utopia in terms of Canadian real estate

I feel like Google result quality has declined though, so chatGPT is just bridging the gap; not sure if I'm more productive than I was when Google worked better. ChatGPT often gives me concrete things to google which give me the results I want in the end, and save me a bit of time, but that's only because Googling the thing in the abstract doesn't give me those results any more.

...and I'm not enough of a maniac to trust what ChatGPT tells me without reference checking.

Even that 5-10% boost is mostly just from how horrible Google has become. I used to be able to find the proper sources much faster, but now I might as well ask LLMs since even that wait to generate the answer is technically "faster".

It seems to be better at helping me with stuff I don't know anything about and that has more flexibility in the results. I got it to make a website for me (something I don't know how to do) and it did ok, though required a lot of back and forth, and it picked a lot of defaults, meaning if I'd had more rigid specifications it would have been harder.

I tried to get help in writing a pretty simple python function (some conditional formatting of text) and quickly realized would have been much better looking up what I wanted on SO and writing something myself than get caught in an awkward cycle of taking it's versions which had missed various requirements and trying to get it to correct then.

Anyway it seems to be best at giving minimal examples of things I don't know about, which is helpful. I'm still trying to find the right balance between when I should use it and when I'm better doing things the "old fashioned" way.

This is what working with an expert looks like. They can’t read your mind.

However unlike an LLM, an expert knows what questions to ask to help you express what you want. This is currently lacking in AI, it doesn’t know how to guide you towards a solution you can’t verbalize because you are not the expert and don’t know what to specify.

GitHub Copilot is also very good because it has the context of your open files or selected text and will pretty regularly give you exactly what you want for the next line of code.

I will say it's much weaker for less popular libraries and programming languages, but for most full stack engineering tasks it gives a huge productivity boost

I don't think the grand OP is downplaying anything at all.

And generally speaking, 10% increase in productivity is extremely (repeat 3 times) significant. Remember most developed countries' GDP only grow <3% annually.

``` Given these X files, help me assess how best to implement feature Y. Ask any clarifying questions that may assist with writing a best practice solution. *File 1* *File 2* *File 3* ... ```

If the context length is a problem I'll use claude-100k.

Pretty much always results in good quality code.

I'll then typically open a new window context and get it to review according to best practices highlighting missing e.g. error handling or possible edge cases.

We will always be N breakthroughs away. Why? Because as our uses of computers improve, our work will become ever more complex. And our efforts will become even longer shots than they were before.

I needed to setup a websocket server, chatGPT saved me tons of hours because Google results are now all useless blogspam so even finding simple resources has become hard.

I needed a custom ui component built, chatGPT gave me a good base and I was then able to also it to go add accessibility keyboard handlers, something I do not know e how to do myself.

I was having some CSS issues and chatGPT was able to tell me what was going on.

IMHO it is a huge productivity booster, but only because I already know how to architect good software.

I’ve held off on some projects (but not others) completely because I needed help but I didn't trust the help to not put in backdoors or compete directly. I also needed to learn some concepts in advance.

With ChatGPT it has definitely been a 10x boost because I can trust it not to put in backdoors, at least for its own benefit. Its a great pair programmer, I dont take its solutions out of the box, I walk through them with it and iterate on improvements together. Instead of waiting for a response on StackOverflow and getting into a moderation war in the process, I just ask it and get instant responses that are far better than someone chasing digital karma points.

I’m coding in ways I wouldn't have, lots of bitshifting, assembly. I can just ask “how would this same procedure be done in assembly”, “why is the syntax like that”.

things that are both hard to ask and hard to google.

it predicts what it needs to explain, unprompted, instead of someone online that wants you to prove that you’ll take initiative in scouring documentation and trial and error yourself.

these are just some of the ways that its a productivity boost for me, let alone acknowledging the subcontractors and developers I simply didn’t seek out to hire.

Do you want a dystopia? Because that's how you get a dystopia :(.

I use ChatGPT when form matters but content doesn't, i.e. emails, letters, etc. I tell it the topic or the information that should be present and most of the time it generates a good email or letter.

If I need to find a specific piece of information I use FastGPT by Kagi, I'm consistently surprised by how good it is at finding exactly what I ask it, and it provides real references too.

"Given this json blob what is the typescript type?" "Given this type, make a function that loops over X and makes a map of Y?"

Or I use it to write code in languages I'm not familiar with. Heck I even used it to make some github actions cause I severely dislike yaml.

It did hallucinate a bit on the github actions (especially since it didn't know the latest api), but it put me on the right track. So I don't know, I think it enables me to do work in areas I normally wouldn't, like Java. It isn't always right, but it helps me immediately test ideas / iterate.

I think of it like reference I can ask questions without getting sass or having to worry about some toxic coworker wondering why I don't just know "x". It's not always right, but neither are my coworkers / humans. So I guess, treat it like a human aka fallible?

Lately, I've revised my view a bit. AI can get novices started on getting a project done and it can be a source of information for the project. Googling works but I think it's much faster querying an LLM. It's good for certain types of projects but I wouldn't trust it for important projects unless I can double check but the double checking will make it a drag to use. I think LLMs do have a place for novices but you have to be careful.

For simple things LLMs can excel but as you add complexity, you really need to double check

the produced playbooks passed the syntax check, that much i can admit. it went downhill from there, producing files with completely invalid keywords.

I'm assuming you are talking more about coding, in which case I agree.

10% is disruptive:)

GPT-4 is not immune to these issues, but it does much, much better.

Whenever I exceed GPT-4 quota (OpenAI has a hardware shortage) and switch to GPT-3 I remember how useless it is and I'm better off waiting a few hours for more GPT-4 quota.

Give it basic directions. I'm in my car turning right on spring garden onto Broad street. I Travel north 10 miles on the same road. Where am I? It has no clue

GPT4 nails your question about going east from New York in a boat, even though I think that’s a very poor test of ‘intelligence.’

I have no idea what you’re asking with your second question (what city are you talking about? There is no context), so I guess I, too, am not intelligent.

> My prompt:

What happens if you go due east exactly from NYC in a boat. Eventually you'll hit land where does it hit?

Start your response with "Let's work this out in a step by step way to be sure we have the right answer:" and work this out in a step by step way to be sure you have the right answer

> GPT-4 answer:

Let's work this out in a step by step way to be sure we have the right answer:

1. Firstly, the direction "due east" means you would be following a line of latitude, rather than a straight line on a globe. This is because the Earth is a sphere (more accurately, an oblate spheroid) rather than a flat surface.

2. New York City is located approximately at 40.71 degrees North latitude. If you were to go due east from NYC, you would be following this line of latitude across the Atlantic Ocean.

3. The first landmass you would likely encounter on this journey would be the westernmost points of mainland Europe, specifically the Iberian Peninsula which comprises Spain and Portugal. However, due to the curvature of the Earth and the specific latitude of New York, it is more likely that you'd hit the northern coast of Spain, around the region of Galicia.

4. So, based on this understanding, if you started in New York City and traveled due east, you would likely end up hitting the northern coast of Spain.

However, it's important to note that this answer assumes idealized conditions and that you're strictly adhering to the 40.71-degree North line of latitude. In reality, ocean currents, winds, and other factors might cause a vessel to drift from this line. Plus, the exact landfall can vary depending on the precise starting point in New York City, which is spread out over a fairly large geographical area.

---

https://imgur.com/ExRr1YN

> My prompt:

I'm in my car turning right on spring garden onto Broad street. I Travel north 10 miles on the same road. Where am I?

Start your response with "Let's work this out in a step by step way to be sure we have the right answer:" and work this out in a step by step way to be sure you have the right answer

> GPT-4 answer:

Let's work this out in a step by step way to be sure we have the right answer:

1. First, we need to identify where Spring Garden Street and Broad Street intersect. These are streets located in Philadelphia, Pennsylvania.

2. From the intersection, you said you would be turning right onto Broad Street, indicating that you are now heading north on Broad Street.

3. You then proceed to travel 10 miles north on Broad Street.

Determining the exact location after these steps is a bit more challenging because street maps generally don't provide mileage measurements. The scale can vary, and the actual distance can depend on the specific path taken.

However, to give a rough idea, Broad Street in Philadelphia is approximately 13 miles long from its southern end to its northern end. Given that Spring Garden Street is relatively central, traveling 10 miles north could possibly place you near or past the city limits of Philadelphia, but without more detailed map information, it's difficult to be more precise.

You'd likely end up in the vicinity of Cheltenham or Elkins Park, which are northern suburbs of Philadelphia, if the distance traveled is relatively straight, but again this could vary based on the specific path taken and the precise start location. This is a rough estimate and the exact location may vary. For precise navigation, it's recommended to use GPS or a reliable map application.

Getting it to code, it can save me, a fairly experienced coder, quite a lot of time because it can do some boring things - boilerplate stuff, or things I can’t be bothered to look up. And I can tell if it’s done it right or not.

I’ve seen people with less coding experience use it and just take its suggestions at face value with painful results and definite detriment to their learning.

The best test I’ve seen for what it’s good at is things that are hard to do, but easy to check.

But in any knowledge economy type job, there is no ceiling on skill. Creative, curious, intelligent people who are already high skill are also going to be the most effective at using a new and weird and tricky tool in interesting ways to develop their skills.

Generative AI will make some old skill obsolete, but people who are generically good at picking up new skills will be much more effective at wielding it. Those are the same people who were good at picking up the old skills, too.

Other commenters have pointed out the difficulties users with limited domaine expertise having in parsing the output and distinguishing what is useful, novel, interesting from what is trivial or incorrect.

This slows down the iterative process of learning how to use the tool as well as refining queries because they don’t have a big enough database of ground truth knowledge to calibrate against.

That being said these can work well as educational tools to rapidly get non-experts to a point where they can participate in conversations with experts. In that regard one might say that the bulk of the value is in the non-expert users because they are far more prevalent than experts.

The biggest advantage I saw for the expert researchers was that they knew (1) which big questions to ask, and (2) how to break down those big questions into smaller, more precise questions.

I don't think the paper matches the assertion made in the title of this article - it's talking about a very specific use-case for generative AI, which won't necessarily generalize to "in all cases, workers with less experience gain the most from these tools".

I grew up without a dishwasher (when we asked our parents why we didn't have one my dad responded, "what do you mean? We have six!" referring to us kids). As an adult we have one in the house but it is rarely used. Washing dishes is a menial task but I enjoy it. It's hard to articulate exactly why. Needless to say I'm quite an experienced dish washer, there's an art to it. My process is to collect any dishes that aren't already stacked, fill the sink with just enough water, wash everything well (cutlery in first, out second last, washed individually), then wipe down all the benches and sink. I leave the dishes to drip dry. They get put away later.

When we use the washing machine I still have to do most of the steps above. Except the washing which is 40-120mins of free time for me. But then I have an extra step of checking each dish or piece of cutlery while I'm putting it away. Most things get washed well, but about 10-15% of items have food baked onto them that I then need to soak or rewash (with a more abrasive scrubber than I'd usually use). Maybe this is down to my lack of experience stacking? The whole process takes longer, but it's (arguably) lower touch and (I hear) uses less water.

Which is better? In a commercial kitchen a dishwasher for sure, efficiency at all costs.

But do we need to remove all menial tasks from our workflows? I'm not so sure.

Also, just like getting a manual dishwashing routine figured out, dishwashers are all different and need to be learned like any tool. Understanding where pieces are, which cycles run when, which compartments open and close and even how they do it, are all important. It really is best to think of it as a tool.

Some useful videos:

https://youtu.be/_rBO8neWw04 https://youtu.be/Ll6-eGDpimU

I'm thinking patience and, I guess, the other meta skills and behaviours that aren't directly related to the washing itself.

That said, your pro-dishwasher explanation talks to some of these too.

When people claim that technology disadvantages workers it usually just reflects their prejudices about working people. So called knowledge workers like to think only they can leverage tech while the call center workers are fixed automatons. But in reality people at a lower skill ceiling have a much easier time leveling up.

ChatGPT when it works well is basically a Young Lady's Illustrated Primer (https://en.wikipedia.org/wiki/The_Diamond_Age). Giving underskilled people free access to information will always benefit them the most.

For example, for places where English isn't the first language, LLMs are widely used for content-based tasks (posts, emails, support), because waiting 10s for Bard or GPT to output something higher quality is worth their time. Perhaps these folks end up being more attuned than usual, more in harmony and sync with the LLMs?

Speaking from experience, it took a while for me to move from vim to vscode, from Javascript to Typescript. We probably prefer our silos and feel uber comfortable in existing setups.

  Start your response with "Let's work this out in a step by step way to be sure we have the right answer:" and work this out in a step by step way to be sure you have the right answer

Source: https://github.com/openai/openai-cookbook/blob/main/techniqu...

If your team can't autocomplete APIs, and if you can't run everything locally so you can step through your code with a debugger, and if you don't have good tests so you need an AI to tell when you broke something (see also profiling, telemetry, release quickly and confidently, etc) - you're going to get a million percent more mileage just doing these sort of basics correctly (and fast) vs investing in AI because the open secret is that most of engineering time is spent reading and debugging code rather than writing it. This should be table stakes but so many engineering teams (even at bigtech companies) can't do the above. So they're all signing up for copilot but they still iterate with logging statements and catch regressions in production.

1. Google replacement for simple queries. Like "how do I HTTP POST with go?". Saves like 30 seconds of browsing.

2. Google replacement for vague questions. Like "Which ingress controllers are popular and what's their strengths".

3. Procrastination eliminators. Like "I need an identifier for ..., can you suggest me some proper names?".

4. Boring tasks which could be solved with enough bash scrpipts or regexes, but with ChatGPT it's often faster.

5. Most impressive to me: "I want to format the following YAML according to the following rules: <vague imprecise rules>. Example: ....". This sometimes saves me really lots of time, because it acts like a human and able to follow imprecise instructions. I could either do it myself spending lots of time or write some formatter spending, again, lots of time.

I was never able to utilize it for "smart" tasks, like if I couldn't solve something and it solved it for me. It's not smart yet. But it's an useful tool in my toolbelt, definitely worth $20 and saves me time and sanity for some boring tasks.

However, in all cases I have seen a very substantial decline of the capabilities of chatgpt-4 with the last few releases. E.g. it used to get code snippets most often right before. Whereas now it tends to be wrong most of the time. Usually it conflates capabilities of several distict libraries, just hallucinating (extrapolating) some non existing functions or attributes.

I personally suspect they are 'cleansing' the training data, and/or driving severe 'schizophrenia' into the model through conflicting RLHF.

feels like there is a viewpoint floating around that chatgpt is valuable in part because of the RLHF work

but does this makes it generally capable? or just prove that you can hire essentially the people you would hire to staff a call center (language skills + good generalists) to create a language model that behaves essentially like a call center (refuses to take any actions that are not allowed)

I’ve been in tech 18~ years and get a lot out of copilot and chatGPT every day.

Take for example this automated help that I found only minutes ago - and seems to prove that generators can be racist (...I am joking):

https://ibb.co/v3VRTQQ

For example, we all have different weights for how much trust information we get from, e.g, Wikipedia, stackoverflow, high profile bloggers in your domain, conspiracy websites, listacles, blog-spam sites etc, marketing materal etc. If the info is on a social site, is highly upvoted? Is the info from an academic journal, arxiv.org, an academic blog, or a wordoc you downloaded from scribd [1]?

Yes, this is judging a book by it's cover, but it's a heuristic that tends to work well.

By contrast, LLMs present all information to you with the same confidence in the same homogeneous interface. There is no external context. So all your normal heuristics for judging reliability are broken. -

[1] https://xkcd.com/1301/

when i asked are you sure about that it owned up and and said sorry. wtf?