Friday, October 06, 2006

Process Management in ASControl

In OracleAS 10.1.3.0, there wasn't any gui for editing the process management settings of Oracle Application like there was in 10.1.2.x. In OracleAS 10.1.3.1 - a patch to 10.1.3.0 - this part of the release gives you that editability back and more. The underlying OPMN.xml is surfaced via JMX so it is fully programmable as well as surfaced in ASControl.

This screen shows how you can view and click down to edit port allocation:



This screen shows the OC4J port management and JDK options:


This screen shows the network and process management port configuration:


This tends to be a little less humbling for those who aren't familiar with the underlying XML files that I faithfully subject you to in my various posts :-)

Like before, download and try it here:

http://www.oracle.com/technology/software/products/ias/index.html

Thursday, October 05, 2006

Creating OC4J Instances in ASControl

In OracleAS 10.1.3.0 the OC4J instance creation was available as a command line option as documented here:

http://download-west.oracle.com/docs/cd/B25221_04/web.1013/b14432/topology.htm#sthref325

In OracleAS 10.1.3.1 the instance creation, like with OracleAS 10.1.2.x, is available in ASControl again. Below is a screen shot sequence of what it looks like:

1. Within Application Server Control instance click on the instance link:



2. Click on create OC4J instance:



3. Name it and determine which group of OC4J instances it will join:



And you are done. Nice and easy :-)

Try it yourself at:

http://www.oracle.com/technology/software/products/ias/index.html

Wednesday, October 04, 2006

Thread Dump with OracleAS on Windows

A colleague, Troy Hedges, asked me today about getting a stack trace of OC4J on Windows for an issue he was having. My first question was is it OracleAS managed with OPMN or stand-alone?
'
Normally when running stand-alone OC4J on Windows-- because you normally just run it with:

set JAVA_HOME=
set PATH=/bin;%PATH%;
java -jar oc4j.jar

-- the quick and dirty way to get a thread dump is to hit CTRL-BREAK and voila, thread dump. This is a nice feature of Java for those intent in seeing what really is hung up in their application.

However, when you go to the normal Oracle Application Server deployment environment on Windows which uses the Oracle Process Manager OPMN to manage OracleAS, getting a thread dump is a little trickier because OPMN is a second process which is spawning the JDKs which run each OC4J instance.

And because there is no kill -3 on Windows likeUnix/Linux you are left trying to figure out how to get a thread dump for a process (OPMN) containing the process you want the thread dump from (an OC4J instance) . While OPMN gives you all sorts of great stuff - death detection, process management, clustering etc, this layer adds one more step to debugging that needs to be understood.

Troy was about 98% there when he asked me - just about when he discovered the sendsignal.exe utility popular for many Windows users - and with it was pretty much done. I add here a bit of color commentary so others can reproduce what he did.

Looking under the covers of OPMN -- which we have done in several other posts here -- you can see that the configuration for the JDK that runs each OC4J instance managed by OPPMNis done inside of the configuration file <ORACLE_HOME>\opmn\conf\opmn.xml.

For this example here is the relevant bits from my OPMN.xml:

<ias-component id="cluster_group" status="enabled"> <process-type id="j2ee1" module-id="OC4J" status="enabled">
<module-data>
<category id="start-parameters">
<data id="java-options" value="-server -Djava.security.policy=$ORACLE_HOME/j2ee/j2ee1/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false"/>

</category>

<category id="stop-parameters">

<data id="java-options" value="-Djava.security.policy=$ORACLE_HOME/j2ee/j2ee1/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false"/>

</category>
</module-data>
<start timeout="600" retry="2"/>

<stop timeout="120"/>

<restart timeout="720" retry="2"/>

<port id="default-web-site" range="12501-12600" protocol="ajp"/>

<port id="rmi" range="12401-12500"/>

<port id="rmis" range="12701-12800"/>

<port id="jms" range="12601-12700"/>

<process-set id="cluster_group" numprocs="1"/>

</process-type>


and when you run this you will see in your Windows task manager a process for the OC4J instance -- the javaw.exe highlighted below -- and typically two opmn.exe processes per Application Server instance:


If you end the process from the task manager you will get the pleasure of seeing OPMN in action - it will promptly re-start that OC4J instance. And if you look at the logs located at:

<ORACLE_HOME>\opmn\logs\ias_component_id_name~OC4J instance name~process_set_id_name~JVM_number>.log

in my case the ias-component id name was cluster_group, the OC4J instance name was j2ee1 and the process set id name was cluster group (from the opmn.xml above) and I had only one JVM running, that translates into:

D:\soasuite\opmn\logs\cluster_group~j2ee1~cluster_group~1.log

And wouldn't you know it - no stack trace. What's up? Well, it wasn't quite the right signal to OC4J for it to do the stack trace - ending a process in the task manager simply isn't a kill -3.

Let's make that stack trace happen! First, make sure that in your start-options for the OC4J instance you do not have -Xrs set as a parameter to the JVM:

<category id="start-parameters">
<data id="java-options" value="-Xrs ...

Just remove it and re-start the application server.

Next download a quick and dirty utility that makes the equivalent of kill -3 on Windows possible - check out:

http://www.latenighthacking.com/projects/2003/sendSignal/

and install the sendSignal.exe

Then, run it using the PID you see in the task manager above - in my case 10452:


And now you will be rewarded by going back to your OPMN log file (
D:\soasuite\opmn\logs\cluster_group~j2ee1~cluster_group~1.log) with a nice stack trace:


--------
06/10/04 11:54:33 Start process
--------
06/10/04 11:55:25 Oracle Containers for J2EE 10g (10.1.3.1.0) initialized

--------
06/10/04 12:21:02 Start process
--------
06/10/04 12:21:16 Oracle Containers for J2EE 10g (10.1.3.1.0) initialized
Full thread dump Java HotSpot(TM) Server VM (1.5.0_06-b05 mixed mode):

"HTTPThreadGroup-4" prio=6 tid=0x00862df0 nid=0x272c in Object.wait() [0x0bc1f000..0x0bc1fc98]
at java.lang.Object.wait(Native Method)
- waiting on <0x02af4e60> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode)
at EDU.oswego.cs.dl.util.concurrent.SynchronousChannel.poll(SynchronousChannel.java:376)
- locked <0x02af4e60> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:787)
at ....

the rest deleted for brevity. OPMN ever vigilant will start up another OC4J but you will have what you want, which is a stack trace. And there you go, stack traces on managed Oracle Application Server instances on Windows.

Tuesday, October 03, 2006

Stopping an MDB via JMX - Cluster

Moving up a tiny bit in sophistication, my next goal was to do my start/stop of my MDB in a cluster. To illustrate this, I have created a cluster of two OC4J instances each called j2ee1, each in a separate OracleAS instance and they tied together for administrative operations in a group called cluster_group.

First let's take a look at this. You can see on each instance (soa_j2ee and soasuite) I have there is an OC4J instance called j2ee1 (they can be different names but I kept them the same).


If I go to the bottom of ASControl and click on cluster_group - the default out of the box is called default_group, but I have customized my configuration - you will be taken to the administrative screen that lets you do administrative operations across multiple OC4J instances simulataneously. In this case it will be against my two OC4J instances called j2ee1 each on different app server instances:


Within cluster_group, if I click on administration, I will see that there is a cluster mbean JMX browser that sits underneath this environment:


If I drill down on this browser a bit I can see the result of my cluster setup - two JVMProxy's each representing a j2ee1 instance, one on each OracleAS instance:




Clicking on one of them you can see that I can get its JVMProxy MBean:

ias:j2eeType=JVMProxy,name=1,J2EEServerGroup=cluster_group,J2EEServer=j2ee1,
ASInstance=soa_j2ee.MLEHMANN-CA


and the other, differentiated by the OracleAS instance name from the other:

ias:j2eeType=JVMProxy,name=1,J2EEServerGroup=cluster_group,J2EEServer=j2ee1,
ASInstance=soasuite.mlehmann-ca.ca.oracle.com


With that I think we are ready to see if we can update our JMX client from before to walk the cluster and turn off/on the MDB.

Remember, I have deployed my MDB application myMDB to each one of these servers which you can see if you go to the applications page of the cluster_group where the combined list of applications of both servers is displayed:


The solution is provided by Steve Button again, this time in Groovy code, but the translation to Java is pretty easy from here:

http://buttso.blogspot.com/2006/05/locating-oc4j-instances-via-opmn.html

As you may be able to interpret, our job is to iterate through our two j2ee1 OC4J instances in cluster_group, grab the MBean server for each instance and then shut off the MDB like we did for the single instance. The full client is here [1]

First and most importantly, instead of connecting to a specific OC4J instance like we did in the single instance case we connect to the cluster and get its Mbean Server - the trick is instead of going to your OC4J instance name like I did before ("/j2ee1") go to "/cluster". The term /cluster tells the MBean server to go to the cluster MBean server. The port used here, again, is the OPMN request port.

JMXConnector clusterConnect = omdb.connect("service:jmx:rmi:///opmn://127.0.0.1:6006/cluster", "oc4jadmin", "welcome1");
MBeanServerConnection mbs = clusterConnect.getMBeanServerConnection();

Then, we query the OC4J server instances in the cluster. I know that my servers of interest are in the J2EEServerGroup cluster_group and I know what I really want is the JVMProxy for each of those OC4J instances. The JVMProxy MBean will give me the methods that return information about where each of the OC4J instance MBean servers are (host and port) thus allowing me to connect to each of them programmatically:

ObjectName query = new ObjectName("ias:j2eeType=JVMProxy,J2EEServerGroup=cluster_group,*"); Set mbeans = mbs.queryNames(query, null);

Next, we need to do some variable set up - a variable for the service URL for each instance, a connection variable for each, an MBean server for each OC4J instance and finally the MessageDrivenBeanMBeanProxy to turn the MDB off/on:


String serviceURL = "";
JMXConnector instanceConnect = null;
ObjectName instanceObjectName = null;
MessageDrivenBeanMBeanProxy instanceMDBMBean = null;
MBeanServerConnection instanceMbs = null;


Next, we walk through the cluster and for each instance instantiate the JVMProxy Proxy MBean (whose naming convention doesn't quite follow my previous recommendation but it is the right MBean):

for(Iterator iterator = mbeans.iterator(); iterator.hasNext();) {
ObjectName objName = (ObjectName) iterator.next();
JVMMBeanProxy jvm = (JVMMBeanProxy)MBeanServerInvocationHandler.newProxyInstance(mbs,
objName, JVMMBeanProxy.class, false);


From the JVMMBeanProxy we can construct the MBean service URL for that instance:

serviceURL = "service:jmx:rmi://" + jvm.getnode() + ":" + jvm.getrmiPort();

And then connect to it:

instanceConnect = omdb.connect(serviceURL, "oc4jadmin", "welcome1");
instanceMbs = instanceConnect.getMBeanServerConnection();

Then we can look up our MessageDrivenBean:

instanceObjectName = new ObjectName("oc4j:j2eeType=MessageDrivenBean,EJBModule=\"myMDB\",J2EEApplication=myMDB,J2EEServer=standalone,name=\"MessageTopicProcessor\""); instanceMDBMBean = (MessageDrivenBeanMBeanProxy)MBeanServerInvocationHandler.newProxyInstance(instanceMbs, instanceObjectName, MessageDrivenBeanMBeanProxy.class, false);

And last but not least stop it:

instanceMDBMBean.stop();

Running this and browsing the myMDB application shows that all of its MessageProcessor MDB's are stopped as shown below:



There you go. Not a trivial example but works like a champ :-)


[1] Cluster MBean MDB Stop via JMX

ppackage demo.oc4j.jmx;

import java.net.URL;

import java.util.ArrayList;
import java.util.Hashtable;

import java.util.Iterator;
import java.util.Set;

import javax.management.MBeanServerConnection;
import javax.management.MBeanServerInvocationHandler;
import javax.management.ObjectName;

import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;

import oracle.oc4j.admin.management.mbeans.proxies.JVMMBeanProxy;
import oracle.oc4j.admin.management.mbeans.proxies.MessageDrivenBeanMBeanProxy;

public class OperateOnMDBCluster {
public OperateOnMDBCluster() {
}



private JMXConnector connect (String URL, String username, String password) {

JMXConnector jmxCon = null;

try {

Hashtable credentials = new Hashtable();
credentials.put("login", username);
credentials.put("password", password);

// Properties required to use the OC4J ORMI protocol
Hashtable env = new Hashtable();
env.put(JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES, "oracle.oc4j.admin.jmx.remote");
env.put(JMXConnector.CREDENTIALS, credentials);

JMXServiceURL serviceUrl = new JMXServiceURL(URL);
jmxCon = JMXConnectorFactory.newJMXConnector(serviceUrl, env);

// Do it!
jmxCon.connect();

} catch (Exception ex) {
ex.printStackTrace();
}

return jmxCon;
}

public static void main(String[] args) {
try {
OperateOnMDBCluster omdb = new OperateOnMDBCluster();

JMXConnector clusterConnect = omdb.connect("service:jmx:rmi:///opmn://127.0.0.1:6006/cluster", "oc4jadmin", "welcome1");

MBeanServerConnection mbs = clusterConnect.getMBeanServerConnection();

// First get a list of active servers - note the trick of having to
// give it the ias prefix which is part of telling it
ObjectName query = new ObjectName("ias:j2eeType=JVMProxy,J2EEServerGroup=cluster_group,*");
Set mbeans = mbs.queryNames(query, null);

// Now walk through them and construct the JMX service connection from each
String serviceURL = "";
JMXConnector instanceConnect = null;
ObjectName instanceObjectName = null;
MessageDrivenBeanMBeanProxy instanceMDBMBean = null;
MBeanServerConnection instanceMbs = null;

for(Iterator iterator = mbeans.iterator(); iterator.hasNext();) {
ObjectName o = (ObjectName) iterator.next();
JVMMBeanProxy jvm = (JVMMBeanProxy)MBeanServerInvocationHandler.newProxyInstance(mbs, o, JVMMBeanProxy.class, false);
serviceURL = "service:jmx:rmi://" + jvm.getnode() + ":" + jvm.getrmiPort();

//Now lets get the connection to this specific MBeanServer and get that MDB MBean
instanceConnect = omdb.connect(serviceURL, "oc4jadmin", "welcome1");
instanceMbs = instanceConnect.getMBeanServerConnection();
instanceObjectName = new ObjectName("oc4j:j2eeType=MessageDrivenBean,EJBModule=\"myMDB\",J2EEApplication=myMDB,J2EEServer=standalone,name=\"MessageTopicProcessor\"");
instanceMDBMBean = (MessageDrivenBeanMBeanProxy)MBeanServerInvocationHandler.newProxyInstance(instanceMbs, instanceObjectName, MessageDrivenBeanMBeanProxy.class, false);
// And finally stop that sucker
System.out.println("Stopping MDB on: " + serviceURL);
instanceMDBMBean.stop();
}


// ObjectName myMDBObjectName = new ObjectName("oc4j:j2eeType=MessageDrivenBean,EJBModule=\"myMDB\",J2EEApplication=myMDB,J2EEServer=standalone,name=\"MessageTopicProcessor\"");
//
// MDBMBean.start();

System.out.println("Success!");

} catch (Exception ex) {
ex.printStackTrace();
} finally {
}
}

}






Stopping an MDB via JMX - Instance

Carrying on my ongoing thread on JMS and MDB, my next task was to figure out how to stop a MDB programmatically. If you browse around in the OracleAS 10.1.3 MBean browser you will see all sorts of operations you might want to perform programmatically and in my case, looking at the MDB I deployed earlier on this week (http://mike-lehmann.blogspot.com/2006/09/simple-mdb-with-oracle-database-jms.html) I was interested MBean operations available on it.

The trick behind MBeans - at least for starters, is simply finding the darn things. Fortunately, a lot of standard JMX tools can hook up to Oracle Application Server, including JConsole as Steve Button, Mr. JMX at Oracle, blogged about here - http://buttso.blogspot.com/2006/06/more-info-on-remote-jconsole.html.

In my case continuing the "take the easiest route" I simply used the MBean browser inside of ASControl. The steps are illustrated below where I first go to the administrative tab of ASControl, click the System MBean browser and lastly navigate to my MDB application (myMDB) and expand it to find my MDB MessageProcessor and the operations on available on it:







What I was interested in was starting and stopping that MDB within the application itself, and importantly I would like to do it programmatically in a single OC4J instance within an Oracle Applicaiton Server instance. It turns out this is pretty easy to do once you have a basic understanding of JMX. I will take the shortest route there rather than generalizing the solution here - just so you can see the bare minimum.

First you need to know the MBean name - at the top of the ASControl page for MessageTopicProcessor you will see the breakdown of the MDB name in JMX format:

oc4j:j2eeType=MessageDrivenBean,EJBModule="myMDB",J2EEApplication=myMDB,
J2EEServer=standalone,name="MessageTopicProcessor"

Then you write a bunch of boiler plate code to hook up to the MBean server and finally the few lines to lookup the Mbean and do the operation. The full code for doing this is in [1]. The 6 lines that
matter are these - they are pretty self explanatory once you see them:

First connect to that J2EE instance - in my case called j2ee1 - and get its MBean server. Note that the OPMN port used - 6006 is the request port of my OracleAS instance:

JMXConnector clusterConnect = omdb.connect("service:jmx:rmi:///opmn://127.0.0.1:6006/j2ee1", "oc4jadmin", "welcome1");
MBeanServerConnection mbs = clusterConnect.getMBeanServerConnection();

Then look up the MessageDrivenBean in the myMDB application:

ObjectName myMDBObjectName = new ObjectName("oc4j:j2eeType=MessageDrivenBean,
EJBModule=\"myMDB\",
J2EEApplication=myMDB,
J2EEServer=standalone,
name=\"MessageTopicProcessor\"");

Then instantiate a local proxy for that MBean:

MessageDrivenBeanMBeanProxy MDBMBean = (MessageDrivenBeanMBeanProxy)MBeanServerInvocationHandler.newProxyInstance(mbs, myMDBObjectName, MessageDrivenBeanMBeanProxy.class, false);

And finally, stop it:

MDBMBean.stop();

This is using what is called a dynamic proxy, a feature of JMX 1.2 that OracleAS 10.1.3 supports which gives you the ability to work the MBean methods like ordinary Java methods rather than marshalling up the number of arguments and argument types as previously.

In general (there turns out to be exceptions), the way you determine the dynamic proxy is simply take your MBean name you are looking up - in this case MessageDrivenBean and add a "MBeanProxy" on the end of it and away you go. For J2EEApplication, another common MBean people will want to manipulate, it would be J2EEApplicationMBeanProxy.

What's nice about dynamic Mbeans is in your IDE you actually will get code insight into the methods available on your OracleAS MBean (assuming you have admin_client.jar in the classpath). Check this out:


To run this client I just needed to add adminclient.jar to my classpath (it is part of OC4J and JDeveloper in the $ORACLE_HOME\j2ee\home\lib) and away I went. I have been told that dynamic proxies may still have a dependency on oc4j-internal.jar but am not sure that will stick when 10.1.3.1 goes production. Note adminclient.jar is also part of the client distributable that you can download from here:

http://www.oracle.com/technology/software/products/ias/htdocs/utilsoft_preview.html

In the case of MDB, not only can you start and stop them via MBeans in OracleAS 10.1.3.1 you can also control it via an annotation for enabling and disabling them. The reason I mention this is that start and stop operations are runtime operations and do not persist - that is if you stop the MDB and then bounce the container the MDB will come back in a started mode. The enable flag, on the other hand, is a persistent property turning the MDB off or on.

Using the previous MessageProcessor bean as an example, the MDB could be deployed as disabled/not-started as follows with the MessageDrivenDeployment property containing an extra "enabled" attribute:

@MessageDrivenDeployment(resourceAdapter = "simpleOemsRA" enabled="false")

and then later running a client similar to that illustrated, first enabling it and then starting it. Obviously real life will have permutations but the combination of JMX, JMX Consoles and annotations, the ability to do what you need whether at deployment time or runtime is clearly possible. Note this particular enable/disable feature is new in OracleAS 10.1.3.1 and is available at a patch to OracleAS 10.1.3.0 for those interested - corresponding to bug 4619599.

And that's that. Thanks to Steve Button whose JMX code I pillaged down to this tiny sample - in the near future he is working to put out a set of helper classes that generalize the solution below as part of a bigger scripting solution using Groovy on top of JMX. This was my poor man's way to get a quick and dirty example out and about while they work getting 10.1.3.1 out the door!


[1] Full Code:

package demo.oc4j.jmx;

import java.net.URL;

import java.util.ArrayList;
import java.util.Hashtable;

import java.util.Iterator;
import java.util.Set;

import javax.management.MBeanServerConnection;
import javax.management.MBeanServerInvocationHandler;
import javax.management.ObjectName;

import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;

import oracle.oc4j.admin.management.mbeans.proxies.JVMMBeanProxy;
import oracle.oc4j.admin.management.mbeans.proxies.MessageDrivenBeanMBeanProxy;

public class OperateOnMDBInstance {
public OperateOnMDBInstance() {
}



private JMXConnector connect (String URL, String username, String password) {

JMXConnector jmxCon = null;

try {

Hashtable credentials = new Hashtable();
credentials.put("login", username);
credentials.put("password", password);

// Properties required to use the OC4J ORMI protocol
Hashtable env = new Hashtable();
env.put(JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES, "oracle.oc4j.admin.jmx.remote");
env.put(JMXConnector.CREDENTIALS, credentials);

JMXServiceURL serviceUrl = new JMXServiceURL(URL);
jmxCon = JMXConnectorFactory.newJMXConnector(serviceUrl, env);

// Do it!
jmxCon.connect();

} catch (Exception ex) {
ex.printStackTrace();
}

return jmxCon;
}

public static void main(String[] args) {
try {
OperateOnMDBInstance omdb = new OperateOnMDBInstance();

JMXConnector clusterConnect = omdb.connect("service:jmx:rmi:///opmn://127.0.0.1:6006/j2ee1", "oc4jadmin", "welcome1");
MBeanServerConnection mbs = clusterConnect.getMBeanServerConnection();
ObjectName myMDBObjectName = new ObjectName("oc4j:j2eeType=MessageDrivenBean,EJBModule=\"myMDB\",J2EEApplication=myMDB,J2EEServer=standalone,name=\"MessageTopicProcessor\"");
MessageDrivenBeanMBeanProxy MDBMBean = (MessageDrivenBeanMBeanProxy)MBeanServerInvocationHandler.newProxyInstance(mbs, myMDBObjectName, MessageDrivenBeanMBeanProxy.class, false);
MDBMBean.stop();
System.out.println("Success!");

} catch (Exception ex) {
ex.printStackTrace();
} finally {
}
}

}