Calling matlab callback/function handle from Java
How do I pass a matlab function handle to a Java object and invoke it from within Java (that is, I want Java to tell matlab when it is ready with a calculation). I am trying to use the com.mathworks.jmi.Matlab class for evaluating Matlab expressions in the Java object, but I can't see how to 1) transfer the callback funcktion handle to Java, and 2) invoke it from Java possibly using the com.mathworks.jmi.Matlab class.
Hmm. Looks like JMI is one of those matlab internals things which may be subject to change in future versions. I found these online articles, not sure if they will help in your case.
I'm afraid that I do not know of a way to do exactly what you requested, but there are two ways you can do something similar:
1) If you set a Java callback from within Matlab, you can send it a Matlab function handle. When the Java callback event gets raised, your Matlab function will be invoked. An example for this can be found here: http://UndocumentedMatlab.com/blog/uicontrol-callbacks/. Note that this does not need JMI to run.
2) JMI needs a string (function name or command line) for Matlab to evaluate. If you know the function name in advance you can use it. Otherwise, you can prepare a Matlab switch-yard function (whose name is known in advance) to dynamically direct your callback to the appropriate Matlab action.
I plan to run an article about JMI in the http://UndocumentedMatlab.com website in mid-April, so stay tuned...
matlabcontrol is a Java API which will allow you to do that. It can invoke a MATLAB function using feval. It cannot operate directly on a function handle, but as mentioned by KitsuneYMG, you can use func2str to convert a function handle to a string. To get started, you can take a look at the walkthrough.
To pass a callback into matlab, you pass the name of the matlab function and it's arguments into com.mathworks.jmi.Matlab.feval("matlabControlcb", ... , 0); The ... is an Object where  = command and [1..end]=arguments.
See Also: FEVAL FUNC2STR
try this tool: http://jamal.sourceforge.net/
It does exactly what you need and well-documented.
The main principle it is based on is to make RMI call to the server part that is run inside Matlab. The output is returned to java program. Again, it depends on jmi...