I recently encountered an issue at a client that I thought might be useful to share with others. We were implementing OAM 126.96.36.199 using the F5 (v11.5.2) OAM APM module to act as the webgate in our test environment. Our initial implementation used Open security mode to validate the connection worked. We then progressed to Simple mode to test SSL. This point is where we hit a wall. We kept seeing that the F5 would not initialize because it could not contact the Access Server. Very generic errors such as “Access Server you specified is currently down. Please check your Access Server” and “failed to connect” were seen in the F5 logs. The oblog.log log did not have any errors at all.
A connection test showed that a TCP connection between the F5 and OAM was possible and this was confirmed when we reverted to Open mode.
The F5 actually uses a 10g webgate implementation. After downloading and examining the 10g webgate package documentation (http://www.oracle.com/technetwork/middleware/ias/downloads/101401-099957.html), the Certification Release 1 documentation mentions a bug that has been fixed that sounded similar to our issue. Bug 13387353 – WebGate: simple mode handshake fails with JDK 6u28 and later. Examining that bug in the Oracle Support portal, led to support article 1513143.1.
The description in this support article states that security fixes in the latest JDKs may cause issues with the Simple Mode Handshake. There is a patch to be applied to the webgate listed which doesn’t apply to us since F5 uses their own webgate implementation via the OAM SDK. The fix mentions that you need to disable CBC protection on the webgate side as a workaround with a specific Java flag. Unfortunately, F5 doesn’t use Java but rather C++ for their implementation so the Java flag will not work. On the other hand, we can control the OAM server side connection.
The following code can be added to setDomainEnv.sh under domain_home/bin on the OAM server(s):
Once you restart the server(s) you should be able to perform the handshake between the F5 and OAM. You may need to reconfigure/restart the APM module but this should fix the handshake issue. Be aware that this does indeed disable CBC protections that Oracle put into the JDK due to the BEAST vulnerability but you can mitigate those consequences using other techniques.
Hopefully, this will help anyone else who encounters this issue.