Maximizing AI in Software Development: Leveraging ChatGPT With Parasoft

ChatGPT, one of the hottest topics of 2023, is a language model trained by OpenAI. It’s capable of understanding and generating natural language text and has been trained on an enormous collection of data, including the source code of many open-source projects written in various programming languages.

Software developers can leverage this vast amount of knowledge to assist in their work because it has a semantic understanding of source code. Developers can use ChatGPT to generate useful code with the correct prompts.

Writing code is one application of ChatGPT in software development and there are many others. The focus of this post is on ChatGPT generating code for use in a professional software development environment.

There are already coding plugins available, such as Github Copilot, that use the same technology as ChatGPT to assist software developers. These plugins can analyze the code that developers are writing and generate suggestions for consideration.

With its access to an extensive corpus of code, ChatGPT can write code that’s accurate and provide helpful code suggestions, making the development process faster and more efficient.

ChatGPT & Code Generation

ChatGPT and the future of AI code generation are bright, but there are some flaws. Let’s walk through the pros and cons.

Pros of Chat GPT Writing Code

Although it’s in early adoption, professional programmers are already realizing the advantages of leveraging ChatGPT.

• Supports multiple programming languages. Although ChatGPT is elusive in answering exactly how many languages it was trained on, the most popular languages in use today are the ones best represented in its training set.
• Improves productivity from generated code to more capabilities like suggestions for unit testing, automation scripts, architecture planning, and functional implementation.
• Deeper semantic understanding of source code compared to traditional artificial intelligence and machine learning (ML) tools. ChatGPT has shown that with the right prompts it can understand the function and behavior of code. Its analysis examines the code’s logic and identifies potential edge cases, boundary conditions, and dependencies that might not be obvious at a superficial level. Traditional AI tools are limited to a much smaller training set and constrained output.
• Assists developers unfamiliar with a language or target environment. ChatGPT can help developers who are new to a project get up to speed on the environment and language being used. Although there are caveats, as discussed below, there’s a good opportunity to learn from this code.

Cons of Chat GPT Writing Code

Using ChatGPT to write code has some drawbacks because the technology is new, and the provenance of the generated code is unclear. Here are some of the cons to using ChatGPT in software development.

• IP leakage when using proprietary code and information in prompts. Code used as prompts for ChatGPT becomes part of the training set and is potentially available for other users. Samsung employees found out about these risks recently when they leaked proprietary code to ChatGPT. Commercial solutions like Copilot allow users to turn off the option of using their proprietary code for training.
• Provenance of the generated code. Does ChatGPT’s training data include proprietary code or open source code with nonpermissive licensing? Will there be a need for attribution or licensing for generated code? It’s unclear where the code generated by ChatGPT is coming from since it’s based on examples drawn from the Internet. Although the training data is primarily open source code, there’s still a need to understand licensing requirements.
• Lack of consistency in answers to coding prompts. By design, each answer can and likely will be different each time the same prompt is used.
• Quality and security are concerns since bugs and poorly written code in the open source data used in training might leak into the output from ChatGPT, or the generative AI model may simply make mistakes. ChatGPT does not vet its answers for security or performance, and the code it generates might not even be compilable.
• Developers may accept code with nonobvious problems in it. Since the code comes back neatly formatted and documented, it’s easy to take it for granted. It’s more important than ever to apply good code quality processes, such as code review, static analysis, and unit testing, to generated code to ensure it works as expected.
• ChatGPT is only as good as its training data. When standard development practices change in the industry, like a library API, it’s possible that some ChatGPT responses will include out-of-date information.

Overcoming ChatGPT’s Code-Writing Limitations With Parasoft’s Technology

Parasoft is currently doing active research into the synergy that could benefit customers by combining the advanced capabilities of generative AI models like ChatGPT with the deep software test automation capabilities currently offered by Parasoft solutions. The research focuses on two areas.

1. How to leverage ChatGPT in the automated software testing platform.
2. How to use the platform to validate ChatGPT-generated code.

Leveraging ChatGPT in Parasoft’s Automated Software Testing Platform

There’s great potential for synergy between ChatGPT and Parasoft’s existing software test automation capabilities. We can leverage the deep semantic understanding of code contained in GPT-based models to enhance the testing offerings already contained in the Parasoft platform, including test generation, static analysis, and reporting capabilities. Specifically, we’re focused to improve the following.

1. Generation of code fixes for static analysis violations. Parasoft already possesses a wealth of information about coding standards and static analysis violations. We should be able to use that information to prompt ChatGPT to automatically generate fixes for discovered code violations.
2. Generation of intelligent, semantically aware unit tests. Parasoft’s current unit test generation capabilities are optimized to cover as many code paths as possible without creating redundant tests. We expect that we can combine that with ChatGPT’s semantic understanding of code to generate tests with better names and a set of input values that result in good code coverage and include real-world, boundary condition values.
3. Generation of positive, negative, and security API test scenarios. Parasoft’s API Test Generator in Parasoft SOAtest does a great job of creating functional API test scenarios based on recorded traffic. But it’s limited by its ability to create scenarios only for traffic it has seen. By leveraging ChatGPT, we want to generate API test scenarios that exercise endpoints contained in an OpenAPI definition in a variety of different ways, both expected and unexpected.

Using Parasoft’s Platform to Validate ChatGPT-Generated Code

Parasoft can increase the usefulness of new GPT-based code generation by overcoming issues in the quality, security, and performance of generated code. Our platform is tested and proven in many different application areas, including safety-critical software. We already use AI and ML to assist in static analysis, test generation, and self-healing.

While ChatGPT code generation is bound to become popular, it’s still critical that the code it creates be subject to vulnerability scanning, quality assessment, and testing, such as that provided by the Parasoft platform.

Benefits of ChatGPT & Software Test Automation

Verify and validate generated code. Teams must use software test automation tools to analyze the code generated by ChatGPT with static analysis and test it with testing tools. Generated code should be treated like any other developed code, if not more carefully, to ensure it meets the quality and security goals of the project.

Comply with coding standards. Solutions like Parasoft’s static analysis can ensure that the ChatGPT-generated code complies with coding standards and best practices. They can check the code against predefined rules and guidelines and flag any violations.

Improve security. The security of the generated code by ChatGPT is unknown, and it’s possible that vulnerabilities and associated software weaknesses are hidden within the nicely formatted output. Treat all generated code like you would third-party or open source code. It should be vetted before being integrated into your application.

Increase productivity. Code generation by AI is likely to become an important innovation with the possibility of huge productivity gains. For example, Parasoft’s unit test generation technology is highly scalable for the bulk creation of test cases. In addition, the improvements in capabilities for software testing tools that leverage generative AI bring even more benefits. A true win-win.

Summary

Software organizations can enhance the quality of their testing and minimize the manual work involved in designing and running tests by using software test automation solutions augmented by generative AI technology. In the near future, Parasoft expects to release enhancements to its platform that use ChatGPT to aid in improving the precision and effectiveness of static analysis, unit testing, and API testing, which will ultimately result in better validation for both ChatGPT-generated and human-written code.

To fully benefit from this new AI technology, it’s important to validate ChatGPT-generated code and treat it like any newly developed code. Software test automation tools like those from Parasoft can help make ChatGPT a useful tool for development while lowering risk and improving productivity.