Thinking of “hiring” an AI tool for your development needs? Ask these questions at the interview

Ever since ChatGPT was released in late 2022, the internet has been abuzz with equal parts doom and optimism. Love it or hate it, AI is coming to your development organization. Even if you don’t plan on developing an AI product or hiring an AI development bot to write code for you, it may still be integrated into the tooling and platforms used to build, test, and run your artisanal, handmade source code.

And AI tools will bring unique risks that potentially offset the huge gains in productivity offered by automating tasks that once required human brain cycles. These risks stem from how the AI is trained, built, hosted, and used—all of which are different than other software tooling currently used by developers. Understanding risk is the first step in managing it, and to help you understand potential risks associated with your incoming AI tooling, we’ve written some interview questions that should be part of the onboarding process.

These questions should be asked regardless of the AI’s type or purpose.

• Where will it be hosted? Modern AIs currently require dedicated and expensive hardware to do the astounding tasks we’re seeing make headlines today. Unless you’re going to acquire a brand-new data center, your AI bot will work remotely and require the same security considerations as remote human workers using remote access and offsite data storage.
• What kind of safeguards are in place to prevent IP loss as code leaves the boundary? Everything from smart TVs to cars are reporting usage data to their manufacturers. Some are using that data to improve their software, but others are selling it to advertisers. Understand exactly how your AI tool will use or dispose of source code or other private data it uses for its primary task.
• Will your inputs be used in future training for the AI? Ongoing training of the AI models will be an increasing area of concern both for owners and those whose data is used to train the model. Owners, for example, may want to keep advertisers from biasing the AI bot in a direction that benefits their clients. Artists who share works online have had AI image-generation bots replicate their styles and are very concerned about the loss or theft of creative identity.
• What is the fidelity of its results? ChatGPT’s most well-known drawback is the inaccuracy of its results—it will confidently assert falsehoods as truths. This has been referred to as the AI “hallucinating.” Understanding how and where an AI may hallucinate can help manage it when it does.

On top of that, AI owners and developers will have their own host of security concerns. These new concerns include threats to the AI’s training model that can corrupt its results or disclose proprietary information about how the model operates. Additionally, the AI model will have to interface with APIs, the web, mobile and other applications that need to be built securely.

Developers will have to ask specific questions when using AI tooling such as an AI security scanner to manage risks introduced during software development.

• Is an AI tool the best fit for this use case? Understanding what AI is and isn’t good at is key. The further a task can be broken down into “make a decision according to learned rules” or “write content that passes learned rules,” the better AI will be at it. The further a problem drifts from that, the worse AI will be at it.
• What safeguards are in place if the tool doesn’t catch something or hallucinates something that’s not there? Never introduce a single point of failure into your processes, especially one that can hallucinate. Rely on traditional defense-in-depth practices or the “Swiss cheese” method of managing risk, in which even if one layer misses a problem, the next will catch it.
• What oversight is required to vet the tool results? This is an old problem made new: traditional logging guidance breaks down into two parts. The first is capturing data about important events, and the second is auditing the logs. Until AI matures further, and its drawbacks are understood or mitigated, humans will still need to be kept in the loop.

More and more developers are “hiring” ChatGPT to write source code. Initial reports are that ChatGPT is capable of writing source code in many programming languages and is fluent in all the common and publicly discussed languages. But due to limitations in this beta version’s training and model, the code it produces isn’t always perfect. Often, it contains business logic flaws that can change how the software operates, syntax errors that might blend different versions of software, and other problems that appear human in origin. Put another way, ChatGPT is a junior developer. When using code written by this junior developer, thought has to be given to how it will be managed.

• Who will be its manager and ensure the code is functional, optimized, high-quality, and up to security standards? Junior developers need senior developers. Every line of code will have to be tested, and some will have to be fixed. However, initial reports are that this proofreading process is faster and easier than writing code from scratch.
• Is it injecting or remixing training code into your codebase? A more insidious threat is that sometimes AI bots like GitHub Copilot produce source code that perfectly replicates blocks of code from its training data. Antiplagiarism tools will be needed to ensure license risk is managed.
• Where does the bot get its training data? An AI model will only be as good as its training data. If the bot is trained on old or incorrect code, it will produce old and incorrect results.
• Where is the engine hosted? Again, an AI bot that analyzes source code will need to bring the source code to its processing facility. Give extra thought to how the data is secured, used, and disposed of after it leaves your company’s boundary.

December’s release of ChatGPT heralded a new age in software development. It’s important to lean into the changes rather than get knocked out by them. When adopting these new tools, understand that the more things change, the more they stay the same: It’s always better to prevent a security incident than to be caught unawares by one.

To learn more, visit us here.