Hi,I'm using a layered virtual configuration:Tier1: Virtual applying SNI-Routing (only SSL persistence profile and LTM policy as described in https://www.devcentral.f5.com/kb/technicalarticles/sni-routing-with-big-ip/282018)Tier2: Virtual applies SSL termination and delivering the actual application, with the required profiles, iRules, .... If the required, an additional LTM policy is applied for URI-based routing and forwards to Tier3 VS.Tier3 (optional, if required): Virtual delivers specific applications, like microservices, usually no monolithical apps.This configuration is very robust and I'm working with it successfully since years. Important: The tier1 uses one single IP address and a single port. So all tier2 and tier3 virtuals MUST be externally available through the same IP address and port.Now I have to publish the first HTTP/2 applications over this concept and see strange behavior of the BIG-IP.User requests www.example.com. IP and port point to tier1 virtual.Tier1 LTM policy forwards the requests, based on the SNI, to tier2 virtuals "vs-int_www.example.com".Within www.example.com there are references to piwik.example.com, which is another tier2 virtual, behind my tier1 virtual.User requests piwik.example.com. IP and port point to tier1 virtual.Tier1 LTM policy forwards the requests to "vs-int_www.example.com" instead of "vs-int_piwik.example.com". Probably not based on SNI, but on the existing TCP connection.I'm afraid, that this bahvior is a result of HTTP/2, especially because of the persistent TCP connection. I assume that, because the connection ID (gathered from browser devtools) for requests to www.example.com and piwik.example.com is identical. From the perspective of the browser I wouldn't expect such a behavior, because the target hostname differs.I didn't configure HTTP/2 in full-proxy mode, as described in several articles. I've just enabled it on the client-side. I would be very happy for any input on that. Thanks in advance!

it's called coalescing in http/2.do your ssl certificates contain SAN?

Woah...that's completely new for me and seems to break my layered virtual concept, based on SNI forwarding. In my case a wildcard certificate is in place. Not sure if this will result in the same behavior, but from the description and my observations, I would assume that the behavior is identical - generally every CN is also in the SAN list these days.Unfortunately, I wasn't able to find specific information, related to the BIG-IP. From my understanding I won't be able to disable this behavior in the BIG-IP, i.e. via HTTP/2 profile or LTM policy or even via iRule, right?The only solution I cloud think of is to get rid of the wildcard certificate and make sure, that each "app", for which a dedicated tier2 virtual exists, uses only domain-specific certificates, without any additional SANs, that should be handled by other tier2 virtuals. Did I get this right?

the coalescing in controlled by client.in your case, the wildcard certificate basically causes client to coalesce the connections.the solution is simply do ssl offload/termination in tier1 vserver (assign client(side) ssl profile with the wildcard cert in it)

How I read this HTTP/2 breaks SNI based routing in layered VS setup. Bad, but thanks for this important hint! Another reason why I don't like http/2.

simply do the ssl temination in tier1 vserver.the new udp based http/3 uses same coalescing mechanism.it's faster than tcp based http v1.1/v2.i don't think it should be avoided as youtube, facebook and many cdn use it.

svs

Cirrostratus

Jun 24, 2024

Incosistent forwarding of HTTP/2 connections with layered virtual

Hi,

I'm using a layered virtual configuration:

Tier1: Virtual applying SNI-Routing (only SSL persistence profile and LTM policy as described in https://www.devcentral.f5.com/kb/technicalarticles/sni-routing-with-big-ip/282018)
Tier2: Virtual applies SSL termination and delivering the actual application, with the required profiles, iRules, .... If the required, an additional LTM policy is applied for URI-based routing and forwards to Tier3 VS.
Tier3 (optional, if required): Virtual delivers specific applications, like microservices, usually no monolithical apps.

This configuration is very robust and I'm working with it successfully since years. Important: The tier1 uses one single IP address and a single port. So all tier2 and tier3 virtuals MUST be externally available through the same IP address and port.

Now I have to publish the first HTTP/2 applications over this concept and see strange behavior of the BIG-IP.

User requests www.example.com. IP and port point to tier1 virtual.
Tier1 LTM policy forwards the requests, based on the SNI, to tier2 virtuals "vs-int_www.example.com".
Within www.example.com there are references to piwik.example.com, which is another tier2 virtual, behind my tier1 virtual.
User requests piwik.example.com. IP and port point to tier1 virtual.
Tier1 LTM policy forwards the requests to "vs-int_www.example.com" instead of "vs-int_piwik.example.com". Probably not based on SNI, but on the existing TCP connection.

I'm afraid, that this bahvior is a result of HTTP/2, especially because of the persistent TCP connection. I assume that, because the connection ID (gathered from browser devtools) for requests to www.example.com and piwik.example.com is identical. From the perspective of the browser I wouldn't expect such a behavior, because the target hostname differs.

I didn't configure HTTP/2 in full-proxy mode, as described in several articles. I've just enabled it on the client-side.

I would be very happy for any input on that. Thanks in advance!

zamroni777
Nacreous
Jun 24, 2024
it's called coalescing in http/2.
do your ssl certificates contain SAN?
- svs
  Cirrostratus
  Jun 26, 2024
  Woah...that's completely new for me and seems to break my layered virtual concept, based on SNI forwarding. In my case a wildcard certificate is in place. Not sure if this will result in the same behavior, but from the description and my observations, I would assume that the behavior is identical - generally every CN is also in the SAN list these days.
  Unfortunately, I wasn't able to find specific information, related to the BIG-IP. From my understanding I won't be able to disable this behavior in the BIG-IP, i.e. via HTTP/2 profile or LTM policy or even via iRule, right?
  The only solution I cloud think of is to get rid of the wildcard certificate and make sure, that each "app", for which a dedicated tier2 virtual exists, uses only domain-specific certificates, without any additional SANs, that should be handled by other tier2 virtuals. Did I get this right?
  - zamroni777
    Nacreous
    Jun 27, 2024
    the coalescing in controlled by client.
    in your case, the wildcard certificate basically causes client to coalesce the connections.
    the solution is simply do ssl offload/termination in tier1 vserver (assign client(side) ssl profile with the wildcard cert in it)
Juergen_Mang
MVP
Jun 27, 2024
How I read this HTTP/2 breaks SNI based routing in layered VS setup. Bad, but thanks for this important hint! Another reason why I don't like http/2.
- zamroni777
  Nacreous
  Jun 27, 2024
  simply do the ssl temination in tier1 vserver.
  the new udp based http/3 uses same coalescing mechanism.
  it's faster than tcp based http v1.1/v2.
  i don't think it should be avoided as youtube, facebook and many cdn use it.
  - svs
    Cirrostratus
    Jun 28, 2024
    I agree that we can't ignore that, just because it breaks our beloved concepts and new technologies often require new methods and concepts. That's fine.
    However, my main caveat is, that I need to re-encrypt between tier1 and tier2 virtual, if I don't want to create a security gap or provoke legitimate discussions with the guys from information security, by transporting the traffic unencrypted between both virtuals. I'm talking about the usage of tcpdump on the BIG-IP. Yes, I'm aware, that each encryption on the BIG-IP can be dumped and decrypted. But there's a difference from the information security point of view, whether I need to invest some effort to get unencrypted traffic (which could be formally prohibitted) or if my configuration is "open like barn door".
    zamroni777are you aware that this will solve the issue? Do you have practical experience with this configuration? The descriptions from F5 require customized (server) SSL profiles to "full-proxy HTTP/2". Why do you think that enabling HTTP/2 on tier2 virtual wont result in the same issue? If the same TCP connection is used, why should TLS be re-negotiated by bringing the termination to the tier1 virtual? This would be required to correctly forward the connection to the correct tier2 virtual via SNI-Routing. It's still the same (outer) TCP connection and probably the same (wildcard) certificate...
svs
Cirrostratus
Jun 24, 2024
I should add, that I came to the above conclusion, because of the information from the ASM/AWF event log. I see "invalid hostnames", like "piwik.example.com" in the policy for "www.example.com" appearing. The behavior of the forwarding may be correct, but wrongly shown in the event logs of the WAF. 😵
- svs
  Cirrostratus
  Jun 24, 2024
  It was definetely not a false positve. There was an issues, which the customer didn't see until now. 🙃
  However, the fix was to disable HTTP/2 on the tier2 virtual. So there is definetely an issue with the layered virtual forwarding, when HTTP/2 is enabled. My primary intention to use layered virtual / SNI-routing is to save IP addresses from the public space. I'm still confused about the fact, that the browser doesn't create a new TCP connection for another hostname, although the IP address and port is identical... 😕
CayOnWay
Nimbostratus
Jul 22, 2024
Hi svs,
we possibly have a solution in place. Our Web Application Yard is a set of iRules and an API controller to allow dynamic host reader routing. IF you still haven't solved the problem, I suggest to perform a simple poc on one of your web applications.
Ciao Cay