UnitTestBot Java/Kotlin
UnitTestBot C/C++

Introduction

What is UnitTestBot?

UnitTestBot (aka UTBot) tool provides engineers with a simple 1-mouse-click way to automatically generate unit tests and visualize coverage for projects written in C and C++ Programming Languages. It is an IDE extension (the client) paired with server application.

The tool uses symbolic execution for test cases generation. UTBot uses KLEE Symbolic Virtual Machine enchanced with a new advanced bidirectional symbolic execution algorithm developed by Research Group from Saint Petersburg State University. Tests are generated in form of Google Test Framework.

Tests Generation: Regression Suite

UnitTestBot generates test cases by code, trying to cover maximum statements and execution paths. We treat source code as source of truth assuming that behavior is correct (corresponds to initial user demand). Generated tests are placed in the so-called regression suite. Thus, we fixate current behavior by test cases.

What is the purpose? You obtain full control of future code changes. Any code change now can't break behavior silently. If developer changes the code, then some tests will fall on CI. Then, ther are two possible scenarios:

  • Initial behavior was correct but developer broke it. And, as a result, generated test helped to find a bug.
  • Initial behavior was incorrect and developer fixed it. Then, test must be manually updated to fixate new behavior.

UTBot provides dramatic code quality improvement in the long term: after tests are generated no change can break behavior without an alarm.

Test Generation: Errors Detection

UnitTestBot is not only able to fixate the behavior in an existing code, but it also helps to find real bugs in your project. Some code fragments might be invalid, while developers don't know about it. For example, buffer overflows, assertion failures, segmentation faults, and so on. If UTBot finds an execution path leading to such a situation, it creates a special test case that is added to the so-called error suite.

While all tests from regression suite are green right after generation, tests from error suite are red (they fail if you run them). Such tests generated by UnitTestBot are a convenient way to discover critical problems as soon as possible. After developer fixes a bug, tests can be rerun to verify the problem is gone.

Code Coverage

UTBot tries to generate test cases that:

  • maximize code coverage (by statements and branches);
  • minimize number of tests, i.e. if you drop any test case from generated suite, your code coverage decreases.

Of course, in an ideal world we would like to test every execution path of the program (so-called path coverage), but the number of such paths is often infinite due to loops and recursion. UnitTestBot aims to maximize other types of coverages: statement coverage and branch coverage, most commonly used as metrics for coverage quality assessment tools.