ChatGPT's Coding Skills: Surprising Strengths and Major Flaws

ChatGPT's Coding Skills: Surprising Strengths and Major Flaws

Introduction

Artificial intelligence is now being used to write code, a task traditionally performed by programmers. But how does an AI code generator, like OpenAI's ChatGPT, compare to human programmers?

Evaluating ChatGPT's Coding Capabilities

A study in the June issue of IEEE Transactions on Software Engineering assessed ChatGPT's code in terms of functionality, complexity, and security. Results showed a wide success rate range from 0.66 percent to 89 percent, depending on task difficulty, programming language, and other factors. While sometimes surpassing human efforts, AI-generated code raised significant security concerns.

Advantages and Limitations

Yutian Tang from the University of Glasgow noted AI's potential to enhance productivity and automate tasks but emphasized the need to understand its strengths and limitations. Comprehensive analysis helps identify issues and improve techniques.

Testing Across Languages and Timeframes

Tang's team evaluated GPT-3.5's performance on 728 LeetCode problems in C, C++, Java, JavaScript, and Python. ChatGPT performed well on problems before 2021, with success rates of 89 percent for easy, 71 percent for medium, and 40 percent for hard problems. However, its success rate dropped for post-2021 problems, falling to 52 percent for easy and 0.66 percent for hard problems. This decline might be due to the lack of exposure to newer problems in the training data.

Critical Thinking and Feedback

ChatGPT lacks human-like critical thinking, limiting its effectiveness on new problems. Despite this, it produced code with smaller runtime and memory overheads than at least 50 percent of human solutions. While ChatGPT handled compiling errors well, it struggled to correct its own mistakes due to a lack of problem understanding.

Security and Complexity

The study found vulnerabilities in ChatGPT-generated code, such as missing null tests, but many were easily fixable. Code complexity varied, with C being the most complex, followed by C++ and Python, which were similar to human-written code.

Recommendations for Developers

To improve ChatGPT's performance and reduce vulnerabilities, developers should provide detailed information and specify potential issues in their prompts. This helps the AI better understand complex problems and avoid common pitfalls.

Conclusion

ChatGPT shows promise as a coding tool, but its effectiveness varies based on task and training data. Understanding its strengths and limitations is crucial for integrating AI into software development, allowing developers to enhance productivity and streamline tasks.

Source:  IEEE Spectrum - How Good Is ChatGPT at Coding, Really?

Image: Lukas from Pexels