In the past few months we have seen some cases where the out of the box Enterprise Manager (Fusion Middleware Control) ceases to function in an Identity Management deployment. In a recent example, we’ve seen this JSP error:
A common troubleshooting step in this case is to remove the tmp directory from $DOMAIN_HOME/servers/AdminServer/, but in our experience this did not solve the issue. In the end, the solution was to redeploy the EM ear file along with all of its library dependencies. In this blog post, we will walk you through this process.
DISCLAIMER: Before attempting these steps, you should backup your environment (especially domain home). We make no guarantees about this process and you should perform these steps in your own dev/test environment before attempting in any production environment. Also note that the steps below apply only to an 11.1.2 environment. They will likely vary slightly for PS1, PS2, etc.
The Application you’ll be working with here is EM. The libraries are:
1) Before you begin, make sure the only server running in your domain is the Admin Server.
2) You need to verify where each Application and Library is deployed. You will need this information later when you redeploy so you know where to target each one. Run wlst.sh and execute these commands:
print 'app targets' for appName in emApps:
print 'lib targets'
for libName in emLibs:
You should see output similar to this (this is just a part of the output you will see):
3) Now that you have the targets for each deployment, you need to stop em, undeploy it, and then undeploy all of the Libraries.
Note if some of these are in use by other applications, you will want to take it on a case-by-case basis. In some cases, such as the oam-admin app, it is safe to proceed even with this error (continue even though one of the Libraries is not undeployed). In other cases, such as apm, you can undeploy the app then redeploy it when you’re finished. In the case of DMS, we found that simply stopping it allowed its dependencies to be undeployed. Again, be very cautious with the process and MAKE A BACKUP.
4) Run listApplicatons() to confirm that the Applications and Libraries are undeployed.
5) Stop the Admin Server
6) Remove these directories from $DOMAIN_HOME/servers/AdminServer/tmp/_WL_user/:
7) Restart the Admin Server and redeploy the Libraries and EM Application, making sure you target the same servers for each item as you noted previously. You will have to deploy all of the Library dependences before deploying EM. When you deploy EM, make sure you start it as well.
8) If you stopped or undeployed any apps in step 3 that depended on the Libraries you will have to redeploy and/or start those as well.
9) Finally you can test EM in the browser and make sure it is working. We also recommend restarting the Admin Server as well just as a verification step to see all the Libraries and Apps are performing properly upon restart.
Remember, backup backup backup. Don’t attempt this on a prod environment until you have tested it in a pre-production environment.
For more information on the WLST deploy/undeploy commands, see this page.
Questions, comments or concerns? Feel free to reach out to us below, or email us at IDMWORKS to learn more about how you can protect your organization and customers.