AI as a Debugging Assistant

By Christian Schneider

Recently I thought to myself, ‘Wouldn’t it be great if there was an AI tool which can just debug my code for me, so I can focus on my research instead of wasting so much time on debugging?’ and thus I started looking into the available tools. To begin with I tried out the ones I already knew: ChatGPT and GitHub Copilot. I used both to debug my most recent code and while they were often helpful, they also had notable limitations. In this article, I will compare their capabilities, highlight shortcomings, and discuss what’s needed to make AI assistants better at accelerating the debugging process.

Advantages of ChatGPT and GitHub Copilot

ChatGPT

ChatGPT allows users to explain specifically what they want to do in a given code segment and get back suggestions on what can be done to realize the desired behavior. ChatGPT is also providing detailed explanations of what the suggested code does, making it easy to understand whether it behaves as desired or requires further modification. This ability to reason makes it possible to even discuss broader conceptual questions or debug complex logic errors.

GitHub Copilot

With GitHub Copilot it is possible to get context-aware code suggestions in real-time as it integrates with IDEs. This can accelerate the debugging process quite a bit, since simple test cases can be written in seconds. It also corrects most compilation errors, saving time in the process.

Comparison and Limitations

For a quick overview I’ve summarized the key features, strengths and limitations of both tools in the following table:

As seen in the table, ChatGPT often lacks sufficient context. This is understandable, as it can’t know the full context from small code snippets alone. Since it doesn’t integrate with IDEs, it can’t give any real-time suggestions and a lot of copy and pasting of relevant code is necessary.

Copilot is slightly better when it comes to context. However, since it doesn’t know the full code base, it sometimes makes non-sensical suggestions. For example, it might struggle to make suitable recommendations if a function is defined in a different file. Copilot also isn’t of much help when it comes to debugging complex logic errors since it is not really able to reason or understand user intent all that well. I certainly tried to explain what I want, but often times it would just ignore part of the requirements I set and give me back a code that wasn’t doing anything different than the previous code.

The Future of AI Debugging

To unlock the full potential of AI as a debugging assistant a hybrid between both tools is needed:

• IDE Integration: Combine Copilot’s real-time, context-aware suggestions with the reasoning capabilities of ChatGPT in one IDE integrated tool.
• Full Code Base Awareness: Provide the AI with access to the entire code base, enabling it to understand dependencies across files and thus make better suggestions.
• Advanced Reasoning: Enhance the AI’s ability to engage in meaningful conversations about bugs, understand user intent, and offer solutions tailored to complex issues.

Conclusion

Overall, I can say that both tools certainly can make the debugging process easier, however they are far from perfect. In the future even more powerful AI tools that combine the strengths of the present ones could be made. These could significantly boost productivity in research by freeing up a lot of time spent on debugging and thus let researchers focus more on their actual research topics.