Teaching Machines to Test Software

A single tool that points fuzzing, symbolic execution and large language models at your code, then explains what it found in plain language.

Modern software runs everything from cars to medical devices, and a single missed bug can be costly or dangerous. Yet testing is still one of the slowest and most manual parts of building software. Researchers are taking this on by building a Test Automation Tool, or TAT, that puts several powerful testing engines and a helpful AI layer behind one simple web interface.

A high-level view of the TAT. The user supplies code or a binary and picks the engines, the orchestration layer runs them inside containers, and a language model turns the results into a grounded report.

One tool, many testing engines

The idea is straightforward. A tester brings a piece of source code or a compiled binary, chooses how they want it tested, and lets the tool do the heavy lifting. Behind the scenes the TAT drives a set of well-known open-source engines, each running safely inside its own container.

Fuzzing with AFL++ throws millions of unexpected inputs at a program to see what makes it break. Symbolic execution with KLEE reasons about many possible paths through the code at once, reaching corner cases a person would rarely think of. Binary analysis with angr works directly on compiled programs, which helps when source code is not available. A Python path with pytest and coverage rounds things out for higher level checks. The tester watches live output as a run happens, then receives a structured report of crashes, findings and coverage.

Where the AI comes in

Running the engines is only half the story. The TAT uses large language models in two grounded ways.

First, it helps generate test scenarios. Language models can produce many realistic patterns of intrusions and non-compliant transactions, which lets teams check whether their detection logic really catches bad behaviour rather than only the obvious cases.

Second, it turns raw results into something a person can read. After a run, the tool distils the machine output into a compact summary and asks a language model to write a short analysis and practical suggestions for improving the test harness. One rule keeps this honest. The model only ever sees the summary of measured facts, never the source code, so it cannot invent a number that was not really observed. When coverage stops growing, the report also flags when a deeper symbolic execution hand-off with angr becomes worthwhile, taking the binary and its format into account.

A real example, and what comes next

To show the value in practice, the team studied a small piece of automotive style code that copies incoming data into a buffer. The insecure version has no length check, so a crafted message can overflow the buffer. Fuzzing quickly triggers the crash, and symbolic execution confirms that the fixed version, which adds proper bounds checks, closes the hole. This before and after story is exactly what the tool is built to surface.

Efficiency matters too. Every run of the tool can be measured with an energy tracker that reports how much electricity it used and roughly how much carbon that represents. This concern runs deeper than any single run. In a published case study on automotive software security, the team profiled a full AI-assisted pipeline and found that most of its energy went not to the AI model but to validating the data used to train it, more than 90 percent of the total, a result with real consequences for how such systems should be designed [1].

The tool is moving toward integration into everyday development pipelines, together with documentation and training so teams can adopt it with confidence. The goal is simple to state and hard to achieve. Make thorough, intelligent testing something that happens automatically, so the software we all rely on is a little safer by default.

References

[1] Polat, I. Arif, A. Kanak, S. Ergün, and A. S. Atalay, “AI-Assisted Pipeline for Code Security Quality Measurement in Software-Defined Vehicles: Dataset Generation, Fine-Tuning and Energy-Aware Evaluation,” in 2026 IEEE International Workshop on Metrology for Automotive (MetroAutomotive), Jun. 2026, pp. 203-208.

Related articles