The infrastructure concerning the Java tool competition is available on GitHub and can also be tried using Docker: https://github.com/JUnitContest/JUGE. Please refer to the GitHub repository for instructions.
Related publication:
@article{Devroey2022,
author = {Devroey, Xavier and Gambi, Alessio and Galeotti, Juan Pablo and Just, René and Kifetew, Fitsum and Panichella, Annibale and Panichella, Sebastiano},
title = {JUGE: An infrastructure for benchmarking Java unit test generators},
journal = {Software Testing, Verification and Reliability},
pages = {e1838},
doi = {https://doi.org/10.1002/stvr.1838},
}
It has been reported in the literature that automatically generated test cases are often affected by test-specific bad programming practices, i.e., test smells, that may hinder the quality of the test’s source code and, ultimately, the quality of the code under test.
In this edition of the Java Tool Competition we will evaluate the smelliness of the generated test cases. Intuitively, on one hand, a test case would be smell-free if no test smell (i.e., Mystery Guest, Eager Test, Assertion Roulette, Indirect Testing, Sensitive Equality, or Resource Optimism) is present in the test case. On the other hand, a test case would be smelly, on a scale from 1 up to 6. 1 if it only contains one test smell, 2 if it contains any two smells, etc.
The assessment on whether a test case contains 0, 1, 2, 3, 4, 5, or 6 smells will be done by conducting a study with human evaluators who will annotate whether each of the six smells is or is not present in a random sample of generated test cases. In a nutshell, we will follow the guidelines and the annotation protocol defined in the paper Test smells 20 years later: detectability, validity, and reliability (Section 3.5).
The average smelliness score across this sample will count for 20% of the final score (which is used to rank the tools).
Fill the Google form https://forms.gle/1dLeo1JCgYqim8AP9 and send a zip file with your tool to the organisers (please find their contacts at the end of this page).
We will run our tool with the benchmark and you will receive by email the results of the code coverage and mutation score. You will need these data to write the report (see point 4).
We will conduct an evaluation on the smelliness of your test cases. We will send you a report on that, which you should also discuss in your report.
You will need to submit a short report, which will be included in the workshop proceedings. The report must follow the IEEE conference format. The page limit is 2 pages including references. Please submit the PDF to the organisers by the deadline (please find their contacts at the end of this page).
Finally, the winners will be announced live during the workshop in April.
Will be announced later. Stay tuned!
The Java tool competition is organized jointly by: