News !
As you’ll see below, the Stamina competition is now closed. Congratulations to the winners once again!
This website has been officially closed on 21 of March 2018, meaning that all features related to the competition
itself have been disabled (login, compete, submissions, etc). You can still download the training and test sets, though. Please send an email to blambeau@gmail.com with any question.
However, we are currently in the process of updating this website to become an online benchmark for grammar induction (applied to software model synthesis). This means that you can still download training and test sets, register and participate, use the oracle (once registered), and … try to outperform the winning algorithm!.
Our new benchmarking oracle is more verbose as it reports your exact score after submitting a solution rather than a simple success/failure answer. We hope that you’ll use and enjoy it. Additional features will be added as soon as we can! Feel free to contact us with any question and/or idea.
The Stamina Organizers.
About
The STAMINA (STAte Machine INference Approaches) competition aims to identify the best technique for learning deterministic finite state machines (FSMs). It extends former competitions, especially Abbadingo, and focusses on the complexity of the learning with respect to the alphabet size. Stamina relies on an adapted generation protocol for target machines and strings. It aims to encourage the development of novel learning algorithms and to encourage cross-fertilization between the machine learning and software engineering communities. A prize of £700 (about $1146) will be awarded to the winner.
The most significant contributions will be selected by the competition organizers and their authors will also be invited to submit their work in a special issue of the Journal of Empirical Software Engineering.
Additional details are given on the protocol page; see also instructions to participate. Implementation of baseline learning algorithms is provided. You can also download the training and test samples, and create a competitor account (required to submit results).
The Competition is closed !
The competition ended on the 31st December 2010. Submissions are no longer taken into account for computing the grid below, even if the oracle stays available in your participant section! The offical winners are:
Marijn Heule (TU Delft) and Sicco Verwer (TU Eindhoven, photo)
with the DFASAT algorithm
Congratulations to them !!
Competition grid
The grids below provide feedback about the competition results. The first one presents cell winners and best challengers so far.
| Sparsity of the training sample | |||||
| 100% | 50% | 25% | 12.5% | ||
| Alphabet size | 2 | MVdPyA/Equipo | MVdPyA/Menor | ||
| 5 | DFASAT/DFASAT | DFASAT/DFASAT | |||
| 10 | DFASAT/DFASAT | DFASAT/DFASAT | |||
| 20 | DFASAT/DFASAT | DFASAT/DFASAT | |||
| 50 | DFASAT/DFASAT | DFASAT/DFASAT | |||
This second grid presents statistics about challengers and submissions: # challengers with the cell broken / # challengers with at least one problem broken / total # of submissions for the cell.
| Sparsity of the training sample | |||||
| 100% | 50% | 25% | 12.5% | ||
| Alphabet size | 2 | 2 / 5 / 138 | 0 / 4 / 411 | 0 / 0 / 82 | 0 / 0 / 44 |
| 5 | 1 / 1 / 153 | 0 / 1 / 188 | 0 / 0 / 22 | 0 / 0 / 18 | |
| 10 | 1 / 1 / 128 | 0 / 1 / 100 | 0 / 0 / 16 | 0 / 0 / 16 | |
| 20 | 1 / 1 / 80 | 0 / 1 / 94 | 0 / 0 / 11 | 0 / 0 / 10 | |
| 50 | 1 / 1 / 164 | 1 / 1 / 122 | 0 / 0 / 59 | 0 / 0 / 17 | |
Organizers & Contact
Please send an email to stamina@listes.uclouvain.be with any questions.
- Kirill Bogdanov, The University of Sheffield, UK
- Christophe Damas, Universite catholique de Louvain, Louvain-la-Neuve, Belgium
- Pierre Dupont, Universite catholique de Louvain, Louvain-la-Neuve, Belgium
- Bernard Lambeau, Universite catholique de Louvain, Louvain-la-Neuve, Belgium
- Neil Walkinshaw, The University of Leicester, UK
