> Have there been any good studies on programmer productivity?
Yes, but mostly from the companies developing these products:
* The CoPilot productivity study by Peng et al. that is summarized in a Blog post [1] with some additional details in the Arxiv preprint [2]. They find "developers who used GitHub Copilot completed the task significantly faster–55% faster than the developers who didn’t use GitHub Copilot". (Grep for "but speed is important, too" in [1].)
* Amazon's Q1 earnings report [3] includes a paragraph stating that "participants [using] CodeWhisperer completed tasks 57% faster (on average) and were 27% more likely to complete them successfully than those who didn’t use CodeWhisperer.
* I seem to remember seeing something on Replit's blog a while back with a similar number, but can't find it anymore, so maybe I'm mistaken.
These speed numbers are on specific and well-specified programming tasks, which is of course only one part of a developer's job, so there's a lot of opportunity for impactful research in this space. I suspect that the Empirical Software Engineering community will be awash in third-party empirical studies asking all sorts of more specific productivity questions by this time next year.
Yes, but mostly from the companies developing these products:
* The CoPilot productivity study by Peng et al. that is summarized in a Blog post [1] with some additional details in the Arxiv preprint [2]. They find "developers who used GitHub Copilot completed the task significantly faster–55% faster than the developers who didn’t use GitHub Copilot". (Grep for "but speed is important, too" in [1].)
* Amazon's Q1 earnings report [3] includes a paragraph stating that "participants [using] CodeWhisperer completed tasks 57% faster (on average) and were 27% more likely to complete them successfully than those who didn’t use CodeWhisperer.
* I seem to remember seeing something on Replit's blog a while back with a similar number, but can't find it anymore, so maybe I'm mistaken.
These speed numbers are on specific and well-specified programming tasks, which is of course only one part of a developer's job, so there's a lot of opportunity for impactful research in this space. I suspect that the Empirical Software Engineering community will be awash in third-party empirical studies asking all sorts of more specific productivity questions by this time next year.
[1] https://github.blog/2022-09-07-research-quantifying-github-c...
[2] https://arxiv.org/pdf/2302.06590.pdf
[3] https://s2.q4cdn.com/299287126/files/doc_financials/2023/q1/...