AI Assistants Offering Help to Programmers
But Copilot and related tools not capable of doing the whole job on their own; programmers who “know how to code” still needed
By John P. Desmond, Editor, AI in Business
When ChatGPT was released in November of 2022, programmers feared it would take their jobs away for its ability to write snippets of code that seemed very convincing. Maybe a few of them would even work as intended.
While wholesale replacement of programmers has not come about yet, if it ever will, serious software engineers have developed ways to use ChatGPT and other AI tools as their assistants, landing a hand with vocabulary and syntax rules, and exploiting patterns and recognition that plays to strengths of AI.
For programmers, figuring out how to use AI to help get the work done, is the name of the game.
“AI could be the tool that fast-tracks your career, or it could be the obstacle that derails it. What matters isn’t just whether you use it, but how,” stated Isaac Lyman, senior software engineer and author of a recent account posted on his blog headlined, “Three Types of AI-Assisted Programmers.” An independent developer, he is the author of the book, “Your First Year in Code.”
The first type is the “no-code engineer,” who may begin with a ChatGPT conversation, then use that to write a small program in a major language, followed by tweaks and bug fixes until it’s ready to be loaded into a development environment, and it actually works. “It feels like you’ve cracked the industry wide open,” he stated.
However, “The problems are hidden under the surface,” he stated. Namely, AI tools are trained on code from thousands of real projects which vary in quality and completeness. “It is only occasionally reliable, maintainable, secure or bug-free,” he stated.
He reports that studies have found that code produced from tools in the class of ChatGPT, GitHub Copilot and Amazon CodeWhisperer will pass from 30 to 65 percent of unit tests, and is “secure” 60 percent of the time. These tests are run on small pieces of “straightforward” code.
“There is no data on AI’s ability to write applications as complex as the average legacy app,” he stated. “It’s probably unable to do so at all.”
So where does AI fit for those not trained in coding? “Some would say it doesn’t. A programmer is someone who knows how to code.”
Next is the junior engineer. “Practically every junior developer feels overwhelmed at their first job. It’s like moving to Spain after a year of Duolingo lessons,” Lyman stated. It may take a few months or longer “before you have a clue what’s going on.”
The depth, complexity and higher standards of real-world applications are a different world from the showcase projects worked on at programming bootcamp, he noted. It might be tempting to lean on ChatGPT to fulfill some coding tasks and “get away with it … “But eventually, it will be time to pay the piper.”
And this leads to the greatest risk in using AI coding tools, “Developers who rely on them may never become good,” Lyman stated.
The senior engineer has the fundamental programming skills in place. What role, if any, should AI coding tools play for the experienced developer? “It may help to clarify the boundary between yourself and AI,” Lyman suggested, such as by using AI as a check against the developer’s own code, to get a different point of view, or using AI to churn out code quickly, then check it to see if it does the job.
The author participated in the beta of GitHub Copilot, using it for contract work. “I found that it noticeably increased my development speed,” he stated, noting, “It did nothing for the correctness or maintainability of my code.” If getting paid by the hour, saving time might not be so strategic. But if salaried, saving time writing code frees up time to focus on code quality, documentation or other tasks “that make your software more sustainable.”
College Prof Used AI to Switch Out of Python
The use of AI as an assistant in coding tools was similarly explored by Noah Gift of Duke University when he was teaching his coding class last year. He had planned to teach the popular entry-level programming language Python, but switched to Rust at the beginning of the class, a language that is newer, more powerful and harder to learn.
A software developer with 25 years of experience, Gift had only just learned Rust himself. But he had the assistance of Copilot, which was built on top of OpenAI’s GPT-3.5 and GPT-4.
“There’s no way I could have learned Rust as quickly as I did without Copilot,” stated Gift in an account in MIT Technology Review. “I basically had a super smart assistant next to me that could answer my questions while I tried to level up. It was pretty obvious to me that we should start using it in class.”
As of last October, Microsoft had one million paid Copilot users in 37,000 organizations, CEO Satya Nadella stated in a report from ZDNet. The product was first released in June 2022.
GitHub, acquired by Microsoft in June 2018, co-developed the Copilot model, called Codex, with OpenAI. It is a large language model optimized for writing code. Working as an assistant to a programmer, it “predicts what the programmer is trying to do and suggests code to do it,” according to the Tech Review account.
The prompt updates as the programmer types. The programmer can accept the Copilot suggestion by hitting the tab key, or ignore it. A study of nearly a million Copilot users published by GitHub and Keystone Strategy consultants in June, a year after the tool’s release, found that programmers on average accepted 30 percent of its suggestions, according to the Tech Review account.
“In the last year Copilot has suggested—and had okayed by developers—more than a billion lines of code,” stated Thomas Dohmke, CEO of GitHub. “Out there, running inside computers, is code generated by a stochastic parrot.”
Results of a test run by GitHub and Microsoft in 2023 showed that programmers using Copilot completed their tasks 55 percent faster than those who did not.
“The way that I would think about it is that you have an experienced developer sitting next to you whispering recommendations,” stated Marco Iansiti, a Keystone Strategy cofounder and a professor at Harvard Business School, where he studies digital transformation. “You used to have to look things up on your own, and now—whammo—here comes the suggestion automatically.”
At Duke University, Gift found comparable results. “Rust has got a reputation for being a very difficult language,” he stated. “But I was pleasantly shocked at how well the students did and at the projects they built—how complex and useful they were.” Gift said the students were able to build complete web apps with chatbots in them.
Read the source accounts and related information at the blog of software engineer Isaac Lyman, in MIT Technology Review and from ZDNet.