when we set the client authentication to "request" or "require", the result is ssl error.
The self-signed certificate, I presume, is defined in the client SSL profile. This certificate is only really relevant to the client. When the client initiates an SSL session with the server (BIG-IP VIP), one of the first things the server does is send its certificate (the public certificate in the client SSL profile). The client (browser) must then decide if it trusts that certificate by way of a few checks:
- Is the certificate valid (unexpired, valid attributes)?
- Is the certificate trusted - can a complete chain from this certificate to a root certificate be established, and does the browser have access (and explicitly trust) the root certificate and potentially all CA certificates in between?
If you use an IP address to access an SSL site, you'll most often get a security warning in the browser that the certificate is not trusted. This is usually because you've violated check 1 above - the name you asked for didn't match the name in the certificate. If you don't have an explicit trust established with the CA that issued the server's certificate, that's another reason for a security warning. In your case, since you're using a self-signed certificate, you'd necessarily have to install that certificate in the browser's trust store to avoid the security warnings.
As for client certificate authentication, you need either the request or require options set in the client SSL profile to be able to prompt the user for a client certificate. They both, more or less, perform the same function - requesting a certificate from the client during the SSL negotiation. The most significant difference is how each deals with what the client sends. You can think of this process as a reverse of the server certificate and browser check process. In this case the server (BIG-IP) must be able to validate the certificate presented by the client. Is it a valid certificate? Can a chain of trust be established between the client's certificate and an explicitly trusted set of CA certificates? The require option is a definitive check. All of the tests must pass. The request option, however, is a "soft" check. It will generally not fail if any of the tests fail.
So given the above, please answer these questions:
-
Just to clarify, with a simple client SSL profile applied to the VIP, no client certificate authentication enabled, you can get to the application via HTTPS://, correct?
-
If yes above, do you get a security warning in the browser that the server certificate is not trusted? Or have you installed that certificate in the browser's trust store?
-
What specifically happens when you enable the request option in the client SSL profile?
-
Do you have any iRules applied to the VIP while testing that are looking for certificate attributes? Are you seeing anything in the LTM logs that may indicate a problem?