Tool Competitions

This year as well we are pleased to announce the thirteenth edition of the testing tool competition. The competition has the goal to experiment with testing tools for a diversified set of traditional and emerging systems and domains. This year we have three competitions, namely the Self-Driving Car Testing Tool Competition, the CPS-UAV Tool Competition and the REST API Testing Tool Competition.

Self-Driving Car Testing Tool Competition

Self-driving cars (SDCs) are equipped with onboard cameras and various sensors, which have already demonstrated the possibility of autonomous driving in real environments, leading to great interest in various application domains. However, despite the necessity of systematically testing such complex and automated systems to ensure their safe operation in real-world environments, there has been relatively limited investment in this direction so far.

The SDC Testing Competition organized jointly by the International Conference on Software Testing, Verification and Validation (ICST) and Search-Based and Fuzz Testing (SBFT) workshop @ ICSE is an initiative designed to inspire and encourage the Software Testing Community to direct their attention toward SDCs as a rapidly emerging and crucial domain. The joint call is meant to help interested authors/participants reduce travel costs by selecting the most convenient and close venue.

The competition focuses on regression testing for test suites executing system tests of SDCs in simulation. Regression testing involves test selection, prioritization, and minimization. For the competition, we focus on the first aspect - the test selection, as detailed in the competition platform repository

.

CPS-UAV Tool Competition

We invite researchers to participate in the competition with their test generation tool for Unmanned Aerial Vehicles (UAVs). The UAV Testing Competition organized by the Search-Based and Fuzz Testing (SBFT) workshop is an initiative designed to inspire and encourage the Software Testing Community to direct their attention toward UAVs as a rapidly emerging and crucial domain. Tools will be evaluated against a benchmark with respect to relevant criteria, as done for other, previous competitions.

The UAV Testing Competition organized jointly by the International Conference on Software Testing, Verification and Validation (ICST) and Search-Based and Fuzz Testing (SBFT) workshop @ ICSE is an initiative designed to inspire and encourage the Software Testing Community to direct their attention toward SDCs as a rapidly emerging and crucial domain. The joint call is meant to help interested authors/participants reduce travel costs by selecting the most convenient and close venue.

The competition is detailed on Github.

The REST League

We invite researchers and practitioners to participate in REST League with their test case generation tools for RESTful APIs. The competition targets black-box testing tools, which have access to the OpenAPI documentation of the API under test only. Tools will be evaluated on REST-specific performance metrics (e.g., number of endpoints successfully tested, server-side failures spotted) and more standard test metrics (e.g., code coverage).