Can surefire and failsafe plugin deliberately randomize execution order of test classes?

It appears that surefire and failsafe plugins execute test classes in order while tests defined within a class execute in undetermined order.

To discover tests which rely on order (what we consider bad tests) we want to force the order to be different for each run. Ideally, we'd have a mechanism to disable randomization or a seed number that would repeat the order (must like the old palm OS emulator had a seed number that drove a sequence of random tests).

Let me know if you know a way to do this? If not, I guess I can work one into a local fork and then submit it.

Thanks

Peter

Answers


Some of the other answers link to the surefire maven documentation page, but like most maven documentation it provides no examples of how to actually specify the settings in the maven XML morass. Here is how to do it with the surefire plugin:

<properties>
  <surefire.plugin.version>2.16</surefire.plugin.version>
</properties>

<build>
 <plugins>
   <plugin>
     <groupId>org.apache.maven.plugins</groupId>
     <artifactId>maven-surefire-plugin</artifactId>
     <version>${surefire.plugin.version}</version>
     <configuration>
       <runOrder>random</runOrder>
     </configuration>
   </plugin>
 </plugins>
</build>

Specify the runOrder of Surefire to "random" http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#runOrder


I think this is more the responsibility of your unit test framework, not the Surefire/Failsafe plugins, which are just responsible for bootstrapping the test framework.

There is already a Stackoverflow question about how to make Junit tests run in random order (the answer is to use a custom ClassRunner):

How can I make my JUnit tests run in random order?

This library supplies an implementation if you don't want to write your own: http://randomjunit.sourceforge.net/


First it seemed to me that you are mixing things. Maven-Surefire-PLugin is responsible to run unit tests where it is the case to be independant of the order of execution. Maven-Failsafe-plugin is responsible for exectution of integration tests which is different, cause integration tests could be dependant on the order which os no problem. Apart from that maven-surefire-plugin has some possibilities to influcence of the order of execution order:

http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#runOrder

This will be of course influenced by the testing framework which you use. In JUnit you can influence the order only in limited way. In TestNG it is a complete different story, cause TestNG has the ability to define dependencies etc.

Maven-Failsafe-Plugin has the same capabilities to influence the order of executions.

http://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html#runOrder


Need Your Help

XPages: Dynamically include a custom control based on a value from a repeat?

custom-controls xpages repeat xpages-ssjs

I have a custom control where I loop thru an configuration object to build tabs for my interface.

In Core Graphics, why is Alpha channel preferred to be most significant? (kCGImageAlphaPremultipliedFirst vs kCGImageAlphaPremultipliedLast)

iphone ios core-graphics cgbitmapcontextcreate

Most code that creates a bitmap context uses kCGImageAlphaPremultipliedFirst, to create ARGB, instead of kCGImageAlphaPremultipliedLast to create RGBA. Is there a reason ARGB is preferred?