So in the APM Captcha settings the "Display CAPTCHA After Number of Logon Attempts Equals" actually means that the login attempt must be rejected by the auth server. I think "logon attempt" in this instance means "until failure" - so the entire attempt to logon - not just the first username and password entry.
So if the "Display CAPTCHA After Number of Logon Attempts Equals" setting is set to 1 then the user will need to see the "Your session could not be established" page once - i.e. their Logon Attempt fails/is rejected, and then when the user clicks on "To open a new session, please click here" in order to start a second Logon Attempt they will then see the captcha displayed.
If the APM Captcha "Display CAPTCHA After Number of Logon Attempts Equals" setting is set to 2, then the user will need to see the "Your session could not be established" page twice before the captcha appears - ie be rejected twice by the auth server.