DSRA0304E errors in the SystemOut.log when server restarts.

[maverick_RD]

Hi Guys,
I have a serious problem . whenever the application server processes are killed instead of clean shutdown or when the server(machine) restarts(we have normal shell scripts in the system runlevels which stops the application server processes cleanly), i see the following errors in the application server log after the restart of the jvm. And for this i remove the transaction logs and partner logs for that particular application server profile.and another point to mention is we are in a clustered environment.
My question is, is this the right approach or do we have a better solution for this.
please seee the below lines from my SystemOut.log after the application server restart


[11/2/09 10:45:14:979 EST] 0000001d InternalOracl I DSRA8203I: Database product name : Oracle
[11/2/09 10:45:14:991 EST] 0000001d InternalOracl I DSRA8204I: Database product version : Oracle Database 11g Release 11.1.0.0.0 - Production
[11/2/09 10:45:14:992 EST] 0000001d InternalOracl I DSRA8205I: JDBC driver name : Oracle JDBC driver
[11/2/09 10:45:14:993 EST] 0000001d InternalOracl I DSRA8206I: JDBC driver version : 10.2.0.1.0
[11/2/09 10:45:14:994 EST] 0000001d InternalOracl I DSRA8212I: DataStoreHelper name is: ***@7c1a7c1a.
[11/2/09 10:45:14:995 EST] 0000001d WSRdbDataSour I DSRA8208I: JDBC driver type : ""
[11/2/09 10:45:14:999 EST] 0000001d WSRdbXaResour E DSRA0304E: XAException occurred. XAException contents and details are: The cause is : null.
[11/2/09 10:45:15:000 EST] 0000001d WSRdbXaResour E DSRA0302E: XAException occurred. Error code is: XAER_RMERR (-3). Exception is:
[11/2/09 10:45:15:001 EST] 0000001d XARminst E WTRN0037W: The transaction service encountered an error on an xa_recover operation. The resource was ***@71bd71bd. The error code was XAER_RMERR. The exception stack trace follows: javax.transaction.xa.XAException
at oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:526)
at com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl.recover(WSRdbXaResourceImpl.java:994)
at com.ibm.ws.Transaction.JTA.XARminst.recover(XARminst.java:138)
at com.ibm.ws.Transaction.JTA.XARecoveryData.recover(XARecoveryData.java:687)
at com.ibm.ws.Transaction.JTA.PartnerLogTable.recover(PartnerLogTable.java:524)
at com.ibm.ws.Transaction.JTA.RecoveryManager.resync(RecoveryManager.java:1859)
at com.ibm.ws.Transaction.JTA.RecoveryManager.run(RecoveryManager.java:2580)
at java.lang.Thread.run(Thread.java:810)

[11/2/09 10:45:15:050 EST] 0000001d InternalOracl I DSRA8203I: Database product name : Oracle
[11/2/09 10:45:15:071 EST] 0000001d InternalOracl I DSRA8204I: Database product version : Oracle Database 11g Release 11.1.0.0.0 - Production
[11/2/09 10:45:15:077 EST] 0000001d InternalOracl I DSRA8205I: JDBC driver name : Oracle JDBC driver
[11/2/09 10:45:15:078 EST] 0000001d InternalOracl I DSRA8206I: JDBC driver version : 10.2.0.1.0
[11/2/09 10:45:15:079 EST] 0000001d InternalOracl I DSRA8212I: DataStoreHelper name is: ***@cbd0cbd.
[11/2/09 10:45:15:080 EST] 0000001d WSRdbDataSour I DSRA8208I: JDBC driver type : ""
[11/2/09 10:45:15:085 EST] 0000001d WSRdbXaResour E DSRA0304E: XAException occurred. XAException contents and details are: The cause is : null.
[11/2/09 10:45:15:086 EST] 0000001d WSRdbXaResour E DSRA0302E: XAException occurred. Error code is: XAER_RMERR (-3). Exception is:
[11/2/09 10:45:15:086 EST] 0000001d XARminst E WTRN0037W: The transaction service encountered an error on an xa_recover operation. The resource was ***@60036003. The error code was XAER_RMERR. The exception stack trace follows: javax.transaction.xa.XAException
at oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:526)
at com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl.recover(WSRdbXaResourceImpl.java:994)
at com.ibm.ws.Transaction.JTA.XARminst.recover(XARminst.java:138)
at com.ibm.ws.Transaction.JTA.XARecoveryData.recover(XARecoveryData.java:687)
at com.ibm.ws.Transaction.JTA.PartnerLogTable.recover(PartnerLogTable.java:524)
at com.ibm.ws.Transaction.JTA.RecoveryManager.resync(RecoveryManager.java:1859)
at com.ibm.ws.Transaction.JTA.RecoveryManager.run(RecoveryManager.java:2580)
at java.lang.Thread.run(Thread.java:810)



I have seen couple of others having the same issue but not sure if we have to remove the tranlog,partnerlog everytime this happens.
Hope i am clear.

Thanks in advance.

RD

[Akash Bharti]

I have seen similar problem couple of times in WebSphere Portal, it basically happen if transactions are in-middle of processing and the JVM is killed. Also, I have seen similar errors in case of JVM hang or OOM.

Well, I think the approach you are using should be fine. Though, I am not sure if there is an ifix that adresses this particular issue. btw, whats the WAS version you are at ?? I remember getting rid of this problem moving to WAS 6.0.2.37 .

Also, please have a look at this IBM url, they also suggest the same solution in one of such case :
http://www-01.ibm.com/support/docview.wss?rs=688&uid=swg21288663

Thanks
Akash

[maverick_RD]

Thanks for the response Akash. I have seen the IBM technote earlier but thought if there is a better solution than recycling the transaction logs.
We are WebSphere V6.1.0.17.

Thanks
RD

[Jon Hawkes]

The exception is saying that WAS cannot contact Oracle to perform
transaction recovery. Recovery will be retried until the database responds.
Do you see these lines repeated continuously or do they eventually stop
appearing in the SystemOut log?

Removing the tranlogs is a very bad thing to do as it removes the
possibility of ever recovering the transactions that were in flight when the
machine went down.

If you really don't care about transaction recovery (and therefore data
integrity), you can turn off transaction logging as described here:
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.express.doc/info/exp/ae/tjta_settlog.html
As noted in the infocenter, this is not recommended in production systems.

Does your database come up before the appservers or vice versa? Is there
anything in the Oracle logs to suggest why the database might be throwing
the RMERR?

Post by maverick_RD
Hi Guys,
I have a serious problem . whenever the application server processes are
killed instead of clean shutdown or when the server(machine) restarts(we
have normal shell scripts in the system runlevels which stops the
application server processes cleanly), i see the following errors in the
application server log after the restart of the jvm. And for this i remove
the transaction logs and partner logs for that particular application
server profile.and another point to mention is we are in a clustered
environment.
My question is, is this the right approach or do we have a better solution for this.
please seee the below lines from my SystemOut.log after the application server restart
[11/2/09 10:45:14:979 EST] 0000001d InternalOracl I DSRA8203I: Database
product name : Oracle
[11/2/09 10:45:14:991 EST] 0000001d InternalOracl I DSRA8204I: Database
product version : Oracle Database 11g Release 11.1.0.0.0 - Production
[11/2/09 10:45:14:992 EST] 0000001d InternalOracl I DSRA8205I: JDBC
driver name : Oracle JDBC driver
[11/2/09 10:45:14:993 EST] 0000001d InternalOracl I DSRA8206I: JDBC
driver version : 10.2.0.1.0
[11/2/09 10:45:14:995 EST] 0000001d WSRdbDataSour I DSRA8208I: JDBC driver type : ""
XAException occurred. XAException contents and details are: The cause is
: null.
[11/2/09 10:45:15:001 EST] 0000001d XARminst E WTRN0037W: The
transaction service encountered an error on an xa_recover operation. The
javax.transaction.xa.XAException
at
oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:526)
at
com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl.recover(WSRdbXaResourceImpl.java:994)
at com.ibm.ws.Transaction.JTA.XARminst.recover(XARminst.java:138)
at
com.ibm.ws.Transaction.JTA.XARecoveryData.recover(XARecoveryData.java:687)
at
com.ibm.ws.Transaction.JTA.PartnerLogTable.recover(PartnerLogTable.java:524)
at
com.ibm.ws.Transaction.JTA.RecoveryManager.resync(RecoveryManager.java:1859)
at
com.ibm.ws.Transaction.JTA.RecoveryManager.run(RecoveryManager.java:2580)
at java.lang.Thread.run(Thread.java:810)
[11/2/09 10:45:15:050 EST] 0000001d InternalOracl I DSRA8203I: Database
product name : Oracle
[11/2/09 10:45:15:071 EST] 0000001d InternalOracl I DSRA8204I: Database
product version : Oracle Database 11g Release 11.1.0.0.0 - Production
[11/2/09 10:45:15:077 EST] 0000001d InternalOracl I DSRA8205I: JDBC
driver name : Oracle JDBC driver
[11/2/09 10:45:15:078 EST] 0000001d InternalOracl I DSRA8206I: JDBC
driver version : 10.2.0.1.0
[11/2/09 10:45:15:080 EST] 0000001d WSRdbDataSour I DSRA8208I: JDBC driver type : ""
XAException occurred. XAException contents and details are: The cause is
: null.
[11/2/09 10:45:15:086 EST] 0000001d XARminst E WTRN0037W: The
transaction service encountered an error on an xa_recover operation. The
javax.transaction.xa.XAException
at
oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:526)
at
com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl.recover(WSRdbXaResourceImpl.java:994)
at com.ibm.ws.Transaction.JTA.XARminst.recover(XARminst.java:138)
at
com.ibm.ws.Transaction.JTA.XARecoveryData.recover(XARecoveryData.java:687)
at
com.ibm.ws.Transaction.JTA.PartnerLogTable.recover(PartnerLogTable.java:524)
at
com.ibm.ws.Transaction.JTA.RecoveryManager.resync(RecoveryManager.java:1859)
at
com.ibm.ws.Transaction.JTA.RecoveryManager.run(RecoveryManager.java:2580)
at java.lang.Thread.run(Thread.java:810)
I have seen couple of others having the same issue but not sure if we have
to remove the tranlog,partnerlog everytime this happens.
Hope i am clear.
Thanks in advance.
RD