Summary: Recently at one of my client place, they suddenly started facing issue where In Fusion Middleware Control (/em) the instance components for Oracle HTTP Server, ‘ohs1’ and Oracle Web Cache, ‘webcache1’ were displayed as ‘Down’ status and no metrics were available for them.
Status was confirmed as Up using below command.
The Admin log showed below errors:-
<Unable to load wallets: java.security.cert.CertificateException: PKI-07005: Untrusted Cert Chain.> <BEA-000000> <Unable to fetch or parse webcache.xml from OraWorld:7001: java.security.cert.CertificateException: PKI-07005: Untrusted Cert Chain.> <BEA-000000> <Web Cache configuration is not available.> . . <BEA-000000> <OPMN mbeanstart failed with the following exception oracle.as.management.opmn.optic.OpticBadConnectException: Error flushing write connection to OPMN at oracle.as.management.opmn.optic.OpmnPhone.sendRequest(OpmnPhone.java:774) Caused By: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: PKI-07005: Untrusted Cert Chain. at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174) Caused By: java.security.cert.CertificateException: PKI-07005: Untrusted Cert Chain. at oracle.security.pki.ssl.g.a(Unknown Source) at oracle.security.pki.ssl.g.b(Unknown Source)
a) After much analysis, the root cause was found to be default wallet used by OPMN. It was replaced by a wallet containing the real company details. Thats why, when the client (FMW EM) initiated an SSL handshake with OPMN, it no longer recognised the chain of trust presented by OPMN, which lead to an SSL handshake failure.
b) $ORACLE_INSTANCE/config/OPMN/opmn/opmn.xml was modified to point to the new wallet.
c) By default, the location defined for the demo wallet is given in opmn.xml by:
<ssl enabled="true" wallet-file="$ORACLE_INSTANCE/config/OPMN/opmn/wallet"/>
After changes, it was something like this.
<ssl enabled="true" wallet-file="$ORACLE_INSTANCE/config/OPMN/opmn/OraWorld"/>
d) FMW em communicates with the Web Tier OPMN using SSL. So to resolve this issue, we had to made corresponding change to the client wallet used by the Weblogic Server domain (located at $DOMAIN NAME/opmn/$INSTANCE NAME/wallet/cwallet.sso)
e) Shutdown the Weblogic Admin Server and copy the OPMN wallet file over to the domain wallet location, for example:
cd $DOMAIN NAME/opmn/$INSTANCE NAME/wallet cp $ORACLE_INSTANCE/config/OPMN/opmn/new_wallet_location/cwallet.sso .
f) Restart the Weblogic Server Admin Server and WebTier Instance.
g) Check the Em console now, the status should be up with all up-to-date metrics.
If you encounter any issues or need any help with Identity Management, feel free to contact me on Sumit@OraWorld.co.uk