Python Test Case Generation Tool Competition

Timeline

Important SBFT’24 Dates:

  • Tool Competition Submission: December 1 2023
  • Competition report: 16. January 2024
  • Notification to authors: 21. December 2023
  • Camera-ready due: 25. January 2024
  • Author’s registration due: TBA
  • Workshop: TBA April 2024

Benchmarking Platform

The infrastructure concerning the Python tool competition is available on GitHub: https://github.com/ThunderKey/python-tool-competition-2024. Please refer to the GitHub repository for instructions.

  • Automated Testing for Python in Industry and Open Source (Nicolas Erni)

Competition Process

To participate effectively in the Python Tool Competition, it is essential to follow a structured approach. Firstly, dedicate time to developing your Python test generators, ensuring they are robust and efficient. Once your generators are ready, test them rigorously using the provided target files to validate their functionality and effectiveness. Following successful testing, adhere to the submission instructions to submit your tool for evaluation. It’s important to note that the evaluation of submitted tools will be based on coverage and mutation score metrics, utilizing target files that are not disclosed to the participants. Therefore, the thoroughness of your test generators will play a critical role in your tool’s performance and ranking in the competition.

Tool submission (by 1 December’23 AoE)

To submit your test generator for the competition, please send an email to all the organizers listed below:

  • Nicolas Erni (Zurich University of Applied Sciences) - gann@zhaw.ch
  • Al-Ameen Mohammed Ali Mohammed (Zurich University of Applied Sciences) - alam@zhaw.ch
  • Christian Birchler (Zurich University of Applied Sciences) - birc@zhaw.ch
  • Pouria Derakhshanfar (JetBrains) - pouria.derakhshanfar@jetbrains.com
  • Stephan Lukasczyk (University of Passau) - stephan.lukasczyk@uni-passau.de
  • Sebastiano Panichella (Zurich University of Applied Sciences) - panc@zhaw.ch

In your email, make sure to include the following information:

  1. A link to the repository where your test generator code is hosted. If the repository is private, please ensure you have invited the competition organizers to access the repository.
  2. The name of the version tag associated with your submitted code. Alternatively or complementary, the authors can the commit hash that corresponds to the version tag mentioned above.

This information is crucial for the evaluation and tracking of your submission. Thank you for participating in the competition, and best of luck with your submission!

Notification of the results for structural metrics (code coverage and mutation score) (by Early Jan’24 AoE)

We will run your tool with the competition platform using dedicated validation target files and you will receive by email the results of the code coverage and mutation score. You will need this data to write the report. To successful tool papers a template on how to write the report will be shared upon acceptance notification.

Tool report deadline (by 16 January AoE)

You will need to submit a report, which will be included in the workshop proceedings. The report must follow the IEEE conference format. The page limit is 2 pages including references. Please submit the PDF to the organizers by the deadline.

Results

Finally, the winners will be announced live during the workshop in April.

Organizers

The Python tool competition is organized jointly by:

  • Nicolas Erni (Zurich University of Applied Sciences) - gann@zhaw.ch
  • Al-Ameen Mohammed Ali Mohammed (Zurich University of Applied Sciences) - alam@zhaw.ch
  • Christian Birchler (Zurich University of Applied Sciences) - birc@zhaw.ch
  • Pouria Derakhshanfar (JetBrains) - pouria.derakhshanfar@jetbrains.com
  • Stephan Lukasczyk (University of Passau) - stephan.lukasczyk@uni-passau.de
  • Sebastiano Panichella (Zurich University of Applied Sciences) - panc@zhaw.ch