Forum Discussion

svs's avatar
svs
Icon for Cirrostratus rankCirrostratus
Jun 24, 2024

Incosistent forwarding of HTTP/2 connections with layered virtual

Hi,

I'm using a layered virtual configuration:

  1. Tier1: Virtual applying SNI-Routing (only SSL persistence profile and LTM policy as described in https://www.devcentral.f5.com/kb/technicalarticles/sni-routing-with-big-ip/282018)
  2. Tier2: Virtual applies SSL termination and delivering the actual application, with the required profiles, iRules, .... If the required, an additional LTM policy is applied for URI-based routing and forwards to Tier3 VS.
  3. Tier3 (optional, if required): Virtual delivers specific applications, like microservices, usually no monolithical apps.

This configuration is very robust and I'm working with it successfully since years. Important: The tier1 uses one single IP address and a single port. So all tier2 and tier3 virtuals MUST be externally available through the same IP address and port.

Now I have to publish the first HTTP/2 applications over this concept and see strange behavior of the BIG-IP.

  1. User requests www.example.com. IP and port point to tier1 virtual.
  2. Tier1 LTM policy forwards the requests, based on the SNI, to tier2 virtuals "vs-int_www.example.com".
  3. Within www.example.com there are references to piwik.example.com, which is another tier2 virtual, behind my tier1 virtual.
  4. User requests piwik.example.com. IP and port point to tier1 virtual.
  5. Tier1 LTM policy forwards the requests to "vs-int_www.example.com" instead of "vs-int_piwik.example.com". Probably not based on SNI, but on the existing TCP connection.

I'm afraid, that this bahvior is a result of HTTP/2, especially because of the persistent TCP connection. I assume that, because the connection ID (gathered from browser devtools) for requests to www.example.com and piwik.example.com is identical. From the perspective of the browser I wouldn't expect such a behavior, because the target hostname differs.

I didn't configure HTTP/2 in full-proxy mode, as described in several articles. I've just enabled it on the client-side.

 

I would be very happy for any input on that. Thanks in advance!

asm
awf
http2
https
layered policies
layered virtual
LTM
ltm policy
security policy
waf

7 Replies

Sort By
  • Juergen_Mang's avatar
    Juergen_Mang
    Icon for MVP rankMVP
    Jun 27, 2024

    How I read this HTTP/2 breaks SNI based routing in layered VS setup. Bad, but thanks for this important hint! Another reason why I don't like http/2.

    • zamroni777's avatar
      zamroni777
      Icon for Cumulonimbus rankCumulonimbus
      Jun 27, 2024

      simply do the ssl temination in tier1 vserver.

      the new udp based http/3 uses same coalescing mechanism.
      it's faster than tcp based http v1.1/v2.
      i don't think it should be avoided as youtube, facebook and many cdn use it.

  • zamroni777's avatar
    zamroni777
    Icon for Cumulonimbus rankCumulonimbus
    Jun 24, 2024

    it's called coalescing in http/2.

    do your ssl certificates contain SAN?

     

    • svs's avatar
      svs
      Icon for Cirrostratus rankCirrostratus
      Jun 26, 2024

      Woah...that's completely new for me and seems to break my layered virtual concept, based on SNI forwarding. In my case a wildcard certificate is in place. Not sure if this will result in the same behavior, but from the description and my observations, I would assume that the behavior is identical - generally every CN is also in the SAN list these days.

      Unfortunately, I wasn't able to find specific information, related to the BIG-IP. From my understanding I won't be able to disable this behavior in the BIG-IP, i.e. via HTTP/2 profile or LTM policy or even via iRule, right?

      The only solution I cloud think of is to get rid of the wildcard certificate and make sure, that each "app", for which a dedicated tier2 virtual exists, uses only domain-specific certificates, without any additional SANs, that should be handled by other tier2 virtuals. Did I get this right?

      • zamroni777's avatar
        zamroni777
        Icon for Cumulonimbus rankCumulonimbus
        Jun 27, 2024

        the coalescing in controlled by client.

        in your case, the wildcard certificate basically causes client to coalesce the connections.

        the solution is simply do ssl offload/termination in tier1 vserver (assign client(side) ssl profile with the wildcard cert in it)

  • svs's avatar
    svs
    Icon for Cirrostratus rankCirrostratus
    Jun 24, 2024

    I should add, that I came to the above conclusion, because of the information from the ASM/AWF event log. I see "invalid hostnames", like "piwik.example.com" in the policy for "www.example.com" appearing. The behavior of the forwarding may be correct, but wrongly shown in the event logs of the WAF. 😵

    • svs's avatar
      svs
      Icon for Cirrostratus rankCirrostratus
      Jun 24, 2024

      It was definetely not a false positve. There was an issues, which the customer didn't see until now. 🙃

      However, the fix was to disable HTTP/2 on the tier2 virtual. So there is definetely an issue with the layered virtual forwarding, when HTTP/2 is enabled. My primary intention to use layered virtual / SNI-routing is to save IP addresses from the public space. I'm still confused about the fact, that the browser doesn't create a new TCP connection for another hostname, although the IP address and port is identical... 😕