Random build failures on Jenkins
So this question is going to be somewhat vague, and I apologize, but I don't really even know where to start. We use Jenkins to automate some builds during the night. We execute our builds with maven. For a while, everything was working out nice. We had findbugs, and code coverage results and a steady history of successful builds. All of the sudden, seemingly out of now where we are getting random NoClassDefFound exceptions while the unit tests are being run. Its very strange, because before hand, the log messages clearly show it building the sources and succeeding. Then the unit tests start to run and they instantly die with the aforementioned exception. We did recently perform an update of Jenkins, but I'm not entirely convinced that has much to do with it. The really strange part is that the builds don't always fail with this exception. Sometimes they run and pass fine, other times they fail immediately. The failures are not consistent in terms of the specific unit tests that fail, or the specific classes which cannot be found. While they seem to revolve around the same small set of classes and unit tests, there doesn't appear to be any real pattern to it.
Has anyone else out there seen this with maven builds on jenkins? I really have no idea what to do. The log messages aren't exactly useful, and I don't know what I could do to the Jenkins configuration to get better debugging information out of the failed builds. I can gather whatever information anyone may need if anyone has any ideas. Here is an example stack trace from one of the recently failed builds:
com.ipti.ptl.common.problems.detectors.CMDRProblemDetectorTest.afterPropertiesSetTest Failing for the past 3 builds (Since Unstable#411 ) Took 18 ms. Error Message com/ipti/hardware/bcicinterface/CPacketReceivedArgs Stacktrace java.lang.NoClassDefFoundError: com/ipti/hardware/bcicinterface/CPacketReceivedArgs at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2521) at java.lang.Class.getDeclaredMethods(Class.java:1845) at com.ipti.common.eventbus.EventBus.subscribe(EventBus.java:107) at com.ipti.ptl.common.problems.detectors.CMDRProblemDetector.afterPropertiesSet(CMDRProblemDetector.java:23) at com.ipti.ptl.common.problems.detectors.CMDRProblemDetectorTest.afterPropertiesSetTest(CMDRProblemDetectorTest.java:37
I have experienced similar things. Two important tips we have learned to maintain consistency over time:
- Configure the build to "Checkout fresh sources" each time you build. Sometimes things can otherwise go stale
- Check if you are using clean package or clean install. install will store the artefact in the .m2 repository local to the user running Jenkins. If you are not, any given project may have to download it's dependencies from your configured repository during it's build time. I would recommend using clean install just to get the local artefact in position ready for dependant builds.
It is important to double-check each of your Jenkins builds against those two rules. Inconsistency between projects that inter-depend is also the source of random confusion.
In addition to what @jmkgreen says, or if you try those steps and they don't solve the problem, I'd check for a problem in the artifact's classpath. Surefire and JUnit don't guarantee that tests will run in a particular order. Running tests in a different order may cause classes to be loaded in a different order and result in these weird issues. This blog post has a really nice explanation.
If that's what seems to be happening, here are some steps you can take to clean up the artifact's dependencies.
Remove any unneeded dependencies
Ensure there is only one version of each dependency included (e.g. only one Spring framework).
Pay particular attention to artifacts where the group ID or artifact ID has changed. For these cases, the Maven dependency resolution process cannot detect that one artifact should override the other and includes both. This can cause interesting and sometimes non-deterministic problems. One example is spring.jar vs. spring-core.jar.
If the application includes more than one dependency supporting the same functionality, you'll need to manually exclude the unwanted dependencies as described in the Maven documentation.
Ensure that related dependencies are all using the same version
For example, if the project includes several Spring artifacts (spring-core, spring-jdbc, spring-aop etc.) make sure the version is the same for all of them. Use dependencyManagement or add direct dependencies as needed to make versions consistent.
Replace any -all-type dependencies with their component parts
... and only the ones needed by the project. -all jars may contain every class needed to run the library - repackaged into the jar file where Maven's dependency resolution process can't get at them - instead of referencing them as dependencies.
For example, mockito-all contains a repackaged version of Hamcrest, which could conflict with any other version of Hamcrest a project wants to use (and likely won't cause the hard-to-fix problems until runtime!).