flow

📝 Some notes on programming and coding with AI assistants

Most IDEs come pre-packaged with their own AI assistant for coding. But who should use them?

Recently, I have been wondering about AI coding assistants (AICA). A lot of what I do day-to-day is coding, so having a convenient assistant is good.

I have tried a few and my general impression is simple: it works well, but it frequently introduces bugs which I have to fix. Ultimately, I have a hard time understanding whether I see an improvement in my coding ability or not - on one hand I can code faster; on the other I have to fix more bugs.

So I took a quick dive into the literature (some early works trying to predict productivity gains are available, but, alas!, behind these keystrokes lies an empiricist):

• GitClear conducted their own study into this, showing that post-AICA code has a higher churn rate (i.e. code requiring maintenance further down the line) ¹
• Some works focus on understanding whether AICA solutions are good without verification - they are not^{2 3}, with a recent work showing that only 30% of code suggestions are accepted ⁴
• In terms of productivity, the same work claims a 30% increase in productivity (!) ⁴. However, productivity is measured as the acceptance rate of AICA suggestions, which, despite being a somewhat dumb metric, is associated with perceived productivity ⁵

From other works on the impact of coding assistants on productivity, we get a heterogenous picture:

1. a study asking participants to implement an HTTP server in JS as quickly as possible showed that people using AICA were able to get the job done 50% faster ⁶
2. a study from Uplevel (on 800 of its customers) saw a 40% increase (!) in bugs with very little impact on efficiency (PR cycles reduced by 1.7 minutes) and no impact on burnout rate ⁷
3. a study shows an increase in PRs but also a decrease in build success rates ⁸
4. a study surveying developers determined that they are generally positive about AICA, but are somewhat fearful that more junior developers won’t have the opportunity to own up to their skills if they rely too heavily on these tools; additionally they were more keen to boast its efficiency boosts when tasks are easy or repetitive (i.e. the usual boilerplate code) which leaves more time for learning and creative thinking ⁹
5. a study on developers in the public sector highlights a similar trend, with developers feeling that they have more time to focus on more meaningful and rewarding tasks if they use AICA ¹⁰
6. showed that GH CoPilot has a more positive impact when used by experts and that it can be harmful for novices as it suggests buggy/complicated code which novices cannot fully comprehend ¹¹

My take on this: it can be useful, but a lot of the benefits are overstated and novice developers benefit greatly from steering clear from it while learning. Benefits appear to lie in a speed-quality tradeoff curve, so each position will have a specific position on this.

🔗 Para onde foi o microbioma do cancro?

Um artigo científico publicado em 2020 prometia revolucionar o diagnóstico de cancro. Quando outras equipas não obtiveram os mesmos resultados, foram lançadas dúvidas sobre a investigação e sobre a empresa que se fundou inspirada nas conclusões.

Published in Shifter

🔗 As ferramentas mudam mas as lutas continuam – a Inteligência Artificial e o que quer dizer para o teu trabalho

Os novos modelos de Inteligência Artificial captaram a atenção de tudo e todos. Com a sua capacidade para gerar texto como nunca antes visto, surgiram medos de que possam destruir empregos em massa. Será essa ameaça real? Ou os pressupostos desse medo são mais sociais e políticos do que tecnológicos? Escrito com João Gabriel Ribeiro

Published in Shifter

🔗 Where is my mind: notas sobre leitura mental e Inteligência Artificial

Nos últimos seis meses, por duas vezes tivemos momentos em que os cabeçalhos das notícias anunciavam algo sobremaneira fascinante: a Inteligência Artificial tornou-se capaz de ler a mente humana. Contudo, há uma distância entre os cabeçalhos e a realidade.

Published in Shifter

🔗 De Sokal a Schrag e o que (não) aprendemos no caminho

O caso Sokal não só é injusto para os estudos culturais como beneficia tremendamente as ciências e previne que um olhar mais sério e cético seja colocado sobre inumeráveis estudos.

Published in Shifter