So I ran into an issue with OAM 11gR2PS2 where I needed to modify the retry limit for the authentication scheme. Previous versions of OAM used a system level value that you could define in oam-config.xml (per https://support.oracle.com/epmos/faces/DocumentDisplay?id=1360866.1). When I tried that for PS2, I still saw the default 5 retries.
I then stumbled onto an article talking about someone exhibiting the behavior I desired when they didn’t want it (https://support.oracle.com/epmos/faces/DocumentDisplay?id=1570598.1)
As a result, in my authentication scheme, I added OverrideRetryLimit=1 into my Challenge Parameters. Once you click apply, the value takes effect.
The benefit to this modification, and this applies to previous versions of OAM that support Challenge Parameters as well, is that you can change this value depending on the authentication scheme used. So different scenarios may have different values. Hope that helps.