Google Summer of Code™ 2007 Project: Support for Mutation Testing
Mutation Testing, or Automated Error Seeding, is an approach where the testing tool makes some change to the tested code, runs the tests, and if the tests pass displays a message saying what it changed. This approach is different than code coverage analysis, because it can find code that is executed by the running of tests but not actually tested.
Mentored by Sebastian Bergmann, Mike Lewis participates in the Google Summer of Code™ 2007 to implement Mutation Testing for PHPUnit. These wiki pages document Mike's efforts throughout the summer.
- Original idea as posted on the PHPUnit Ideas page
- Literature Review
Project News: August 11, 2007
There are a couple of problems with the project that need to be addressed before release. Some users may have been getting segmentation faults when trying to perform mutation analysis on some files. This has to do with an update to the parse_tree extensions dealing with doc comments and at least one other parsing problem. I have talked to William about these problems and have been assured that he is working very hard this week to put out fixes. I will post here as soon as those issues are cleared up. In the meantime, here is an example of a working mutation analysis session: http://phpfi.com/255738.
Project News: July 25, 2007
The most recent version committed to the mutation_testing branch is in a testable state. I would encourage anyone interested to try out the new mutation feature in this branch. The user must have a few extensions in order to use this new feature: parse_tree, runkit, libxml, and xsl. If you have these then you can execute the following command to begin mutation analysis: phpunit --mutation <file> <testFile>, where <testFile> is the file containing unit tests and <file> is the class file that <testFile> tests. It is best if a high level of code coverage is achieved before mutation testing. Check out Mutation Testing Online for a good introduction to using mutation analysis.
Certainly there are still some kinks to be worked out. For one, the method for discovering the line number of a mutation is less than desirable, but users should not see much difference. Improvements in this area may have to wait for changes in parse_tree. Also, there will likely need to be some more development on the handling of unary operators. The standard for mutating on unary operators is to delete them, replace them with other unary operators of the same type, and insert them where appropriate. Currently, PHPUnit deletes and replaces unary operators but does not insert. Insertion would be desirable; however, there may be a proliferation of trivial mutants since PHP is untyped.
I would appreciate any input. I will be out of touch for the next week and a half but please send any feedback to me at lewismic@…. Thanks.
Project News: July 1, 2007
This is a brief update on where I am in the project; what I am currently working on; and, what I intend to work on in the near future.
Currently, a command such as phpunit --mutation=<file> <testFile> will perform a series of mutations on <file> and run the unit tests in <testFile> on each mutant until a mutant is not killed. The mutants are saved to files of the form "<class>_Mutant_<randNo>.php" in a mutants directory. Mutations are performed based on a set of relational operators (e.g., ==, !=) and shortcut operators (e.g., +=, %=).
I am now working to handle non-killed mutants. Currently, the program simply exits and prints statistics if a mutant is not killed. Ideally, the program will describe to the user where in the original source code the particular mutation took place and what operator was used. For example:
In BankAccount.php, line: 99 >= might be ==.
While this seems fairly basic it is complicated by the fact that the parse tree discards whitespace. So the problem must be approached from a different angle.
In the future, I will implement a method to differentiate trivial mutants from non-trivial mutants. A trivial mutant is computationally equivalent to the original source, and is no more than an annoyance to the user. I will expand the mutant operators from relational and shortcut operators to include arithmetical, conditional, and logical operators. These operators will be inserted into source code in addition to substituted for other operators, as is the current operation. I will finish with optimization based on performance and correctness tests with different sets of operators, clean-up of the source code, and documentation.
Let me know if you have any suggestions. lewismic@…
