ChatGPT's Code Completion Conundrum
I've been messing around with ChatGPT's code completion feature, and let me tell you, it's bloody good â but only up to a point. I mean, it can't begin to comprehend the creative problem-solving humans take for granted, the kind of thing where you think outside the box and solve the problem that nobody even knows they're having yet.
Take, for example, a study published in the Journal of Machine Learning Research last year (1). They found that ChatGPT's accuracy takes a nosedive when tackling real-world scenarios that are even remotely complicated. So, the AI can whip out a flawless Python loop to iterate over a list of numbers, but ask it to write a loop for a list of dictionaries, and suddenly it's stumped.
That's because â and this is not exactly earth-shattering â human developers have a certain... let's call it intuition, a sense of how to adapt their code to the problem at hand. We're not just solving the thing in front of us; we're anticipating the stuff that's going to go wrong in the first place.
As Robert C. Martin, a developer and author I've got a ton of respect for, puts it: 'Programming is not just about solving the problem in front of you. It's about solving the problem that you don't know about yet.' (2) ChatGPT can chug out code all day, but it can't anticipate the unexpected.
The thing is, the AI's training data is limited. It's like it's only ever seen the same old code written by other people, but it never gets to experience the messy, wonderful world of human error â the failures, the edge cases, all the things that make development so much fun.
So, I'm going to keep using ChatGPT as a code completion tool, but let's be real, it's not going to replace human developers anytime soon. What it's good for is speeding up routine tasks, saving us all a bunch of time and headaches.
References: (1) 'Evaluating Code Completion Systems' by the Journal of Machine Learning Research, 2022 (2) 'Clean Code: A Handbook of Agile Software Craftsmanship' by Robert C. Martin, 2008
















