Automated spider test

I'm looking to add a very simple layer of automated integration testing to our current Continuous Integration setup. (CI currently only checks for build breaks).

Is there a product that will:

  • From a base URL, spider a site & report back any 404/500 error codes?
  • Allow me to add a step to logon, to be able to spider the authorized pages?

Bonuses / would-be-nice:

  • Report JS errors
  • Report 404s linked from CSS

I've had a quick look at SilkTest & Selenium, and they don't seem to feature quite such a site-agnostic approach. (The logon step is obviously something they can do...)

We're simply wanting to cull out the simplest/dumbest of regression errors, and we have an absolute minimum of time to implement such an automated check - hence the spidering. Ideally the solution can be run on the command line, and output its results in something I can parse into TeamCity (continuous integration package).

Much appreciated.


Here is a list of utilities to look at.

SilkTest should be able to handle your use case, you'll need to write a script that navigates through your page, depending on the complexity of your page, a simple recursive descent might be sufficient. If it gets more complex, you might need some sort of already visited URLs to avoid infinite loops.

As for the results, if you use either Silk4J or Silk4Net which both use xUnit runners to drive the tests, I assume you should be able to get the results into TeamCity.

Need Your Help

Why is sysctl producing E_INVAL on Mac OS X?

c objective-c macos sysctl

Below is a pared-down (error/null checks omitted) snippet of C/Obj-C code that uses sysctl to get the argv of a particular process with PID 50.

Error when setting icon with CX Freeze

python build-process cx-freeze

I keep getting this error when I add the icon arugment to my script: