( I guess I'll find out in my first attempts )
I would take a look at some of these engines, I'm guessing Rhino may be the best fit for you as its written in Java. Their tutorials outline embedding Rhino in a Java environment.
- A Rhino embedding tutorial: http://www.mozilla.org/rhino/tutorial.html
- V8 is Google's engine. V8 is not currently threadsafe, so it probably won't suit your needs in a server environment. http://code.google.com/p/v8/
Edit in response to your second question.
Can't help on the Java servlet front, no experience with it.
My gut feeling is that you'll be happier if you choose something that doesn't involve COM. I just wanted to make sure you had all the options in front of you.
Check out http://www.commonjs.org/
If all you need to do is bless your HTML as Excel, you might be better served sending your HTML to the server and have it served back with a content-disposition header and the appropriate MIME type. Server-side JS doesn't normally implement a browser-like DOM so Ext isn't going to work server-side.
Alternatively you could generate your HTML serverside and skip the roundtrip. For example the POI Java library can generate real binary Excel files with multiple sheets and cell functions.