Forum Discussion

ahadem's avatar
ahadem
Icon for Nimbostratus rankNimbostratus
May 13, 2024

F5 not sending traffic to Web pool

Hello All,

 

I am having issues with a new configured F5 big-IP that everything works fine as follows.

 

  1. traffic from the client is coming to the firewall which is then natted to the private network. (works)
  2. the Load balancer ( Virtual server) IP is accessible and request is sent to the virtual server. 
  3. and from the big ip to the pool is not sent.
  4. connection between the F5 to the pool is fine and vice versa and pool and nodes are available (green).
  5. connection between web-server and F5 is through Https (443).

 

configuration F5 as follows:

 

F5  Virtual IP : 192.168.1.41

self IP:   int 1 : 10.10.10.14

self IP   int 2 : 192.168.1.41

 

web server pool :  10.10.10.X  range with class c subnet. 

SSL is configured between the client to F5 as clientssl and between the server and F5 as serverssl. 

source address translation is automap. 

 

I am having trouble why it doesn't work and is trying to find out the problem. 

  • If you can provide the associated configuration of the F5 it would help significantly in troubleshooting this issue. How have you verified that connections are not being sent out from the F5 to the pool members? If you could try running the following tcpdump it might point you to the issue.

     

    tcpdump -nni 0.0:nnp host <virtual server IP>

     

    The tcpdump should follow the SNAT that is occurring to see where the connection is not functioning properly.

  • ahadem's avatar
    ahadem
    Icon for Nimbostratus rankNimbostratus

    I captured the tcpdump packet and analyzed it, the request is coming from the client and three-way handshake is done between the client and the F5 but after nothing happens and when I look into the pool statistics nothing is there and it's all zero. 

     

    have captured packets multiple times and checked packet by packet and the connection is RST at last. 

     

    apm oauth db-instance /Common/oauthdb {
        description "Default OAuth DB."
    }
    apm policy customization-source /Common/modern { }
    apm policy customization-source /Common/standard { }
    apm report default-report {
        report-name sessionReports/sessionSummary
        user /Common/admin
    }
    ilx global-settings {
        debug-port-blacklist { 47019 54321 60000 }
    }
    ltm default-node-monitor {
        rule none
    }
    ltm node /Common/f5STGWEB01 {
        address 10.10.10.15
        monitor /Common/https_443
    }
    ltm node /Common/web1 {
        address 10.10.10.21
        monitor /Common/https_443
    }
    ltm pool /Common/webpool {
        members {
            /Common/f5STGWEB01:443 {
                address 10.10.10.15
            }
            /Common/web1:443 {
                address 10.10.10.21
            }
        }
        monitor /Common/https and /Common/https_443
    }
    ltm snat-translation /Common/192.168.1.41 {
        address 192.168.1.41
        inherited-traffic-group true
        traffic-group /Common/traffic-group-1
    }
    ltm snatpool /Common/snatpool1 {
        members {
            /Common/192.168.1.41
        }
    }
    ltm virtual /Common/virtualserver1 {
        creation-time 2024-04-30:17:02:17
        description "virtual server is the load balancer server"
        destination /Common/192.168.1.41:443
        ip-protocol tcp
        last-modified-time 2024-05-08:17:01:47
        mask 255.255.255.255
        pool /Common/webpool
        profiles {
            /Common/F5SANCert {
                context clientside
            }
            /Common/serverssl {
                context serverside
            }
            /Common/tcp { }
        }
        serverssl-use-sni disabled
        source 0.0.0.0/0
        source-address-translation {
            type automap
        }
        translate-address enabled
        translate-port enabled
    }
    ltm virtual-address /Common/192.168.1.41 {
        address 192.168.1.41
        arp enabled
        icmp-echo enabled
        mask 255.255.255.255
        traffic-group /Common/traffic-group-local-only
    }
    ltm profile client-ssl /Common/F5SANCert {
        app-service none
        cert-key-chain {
            f5-F5-With-SAN_SANChainCert_0 {
                cert /Common/f5-F5-With-SAN
                chain /Common/SANChainCert
                key /Common/f5-F5-With-SAN
            }
        }
        defaults-from /Common/clientssl
        inherit-ca-certkeychain true
        inherit-certkeychain false
        options { dont-insert-empty-fragments no-tlsv1.3 no-tlsv1.1 no-sslv3 no-tlsv1 }
    }
    ltm profile client-ssl /Common/F5f5 {
        app-service none
        cert-key-chain {
            f5-F5-certificate_F5f5Chain_0 {
                cert /Common/f5-F5-certificate
                chain /Common/F5f5Chain
                key /Common/f5-F5-certificate
            }
        }
        defaults-from /Common/clientssl
        inherit-ca-certkeychain true
        inherit-certkeychain false
    }
    ltm profile client-ssl /Common/clientssl {
        alert-timeout indefinite
        allow-dynamic-record-sizing disabled
        allow-expired-crl disabled
        allow-non-ssl disabled
        app-service none
        authenticate once
        authenticate-depth 9
        bypass-on-client-cert-fail disabled
        bypass-on-handshake-alert disabled
        c3d-client-fallback-cert none
        c3d-drop-unknown-ocsp-status drop
        c3d-ocsp none
        ca-file none
        cache-size 262144
        cache-timeout 3600
        cert /Common/default.crt
        cert-extension-includes { basic-constraints subject-alternative-name }
        cert-key-chain {
            default {
                cert /Common/default.crt
                key /Common/default.key
            }
        }
        cert-lifespan 30
        cert-lookup-by-ipaddr-port disabled
        chain none
        cipher-group none
        ciphers DEFAULT
        client-cert-ca none
        crl none
        crl-file none
        data-0rtt disabled
        generic-alert enabled
        handshake-timeout 10
        inherit-ca-certkeychain false
        inherit-certkeychain false
        key /Common/default.key
        max-active-handshakes indefinite
        max-aggregate-renegotiation-per-minute indefinite
        max-renegotiations-per-minute 5
        maximum-record-size 16384
        mod-ssl-methods disabled
        mode enabled
        notify-cert-status-to-virtual-server disabled
        ocsp-stapling disabled
        options { dont-insert-empty-fragments no-tlsv1.3 }
        passphrase none
        peer-cert-mode ignore
        peer-no-renegotiate-timeout 10
        proxy-ssl disabled
        proxy-ssl-passthrough disabled
        renegotiate-max-record-delay indefinite
        renegotiate-period indefinite
        renegotiate-size indefinite
        renegotiation enabled
        retain-certificate true
        secure-renegotiation require
        server-name none
        session-mirroring disabled
        session-ticket disabled
        session-ticket-timeout 0
        sni-default false
        sni-require false
        ssl-c3d disabled
        ssl-forward-proxy disabled
        ssl-forward-proxy-bypass disabled
        ssl-forward-proxy-verified-handshake disabled
        ssl-sign-hash any
        strict-resume disabled
        unclean-shutdown enabled
    }
    ltm profile client-ssl /Common/mynewcertificate {
        alert-timeout indefinite
        allow-dynamic-record-sizing disabled
        allow-non-ssl disabled
        app-service none
        cache-size 262144
        cache-timeout 3600
        cert-key-chain {
            MyCertificate_0 {
                cert /Common/MyCertificate
                key /Common/MyCertificate
            }
        }
        cipher-group none
        ciphers DEFAULT
        data-0rtt disabled
        defaults-from /Common/clientssl
        generic-alert enabled
        handshake-timeout 10
        inherit-ca-certkeychain true
        inherit-certkeychain false
        max-active-handshakes indefinite
        max-aggregate-renegotiation-per-minute indefinite
        max-renegotiations-per-minute 5
        maximum-record-size 16384
        mod-ssl-methods disabled
        mode enabled
        notify-cert-status-to-virtual-server disabled
        ocsp-stapling disabled
        options { dont-insert-empty-fragments no-tlsv1.3 }
        peer-no-renegotiate-timeout 10
        proxy-ssl disabled
        proxy-ssl-passthrough disabled
        renegotiate-max-record-delay indefinite
        renegotiate-period indefinite
        renegotiate-size indefinite
        renegotiation enabled
        secure-renegotiation require
        server-name none
        session-mirroring disabled
        session-ticket disabled
        session-ticket-timeout 0
        sni-default false
        sni-require false
        ssl-sign-hash any
        strict-resume disabled
        unclean-shutdown enabled
    }
    ltm profile server-ssl /Common/do-not-remove-without-replacement {
        app-service none
    }
    net dns-resolver /Common/f5-aws-dns {
        forward-zones {
            amazonaws.com {
                nameservers {
                    8.8.8.8:53 { }
                }
            }
            idservice.net {
                nameservers {
                    8.8.8.8:53 { }
                }
            }
            shpapi.com {
                nameservers {
                    8.8.8.8:53 { }
                }
            }
        }
        route-domain /Common/0
    }
    net dns-resolver /Common/internaldns {
        forward-zones {
            dns {
                nameservers {
                    192.168.1.11:53 { }
                    192.168.1.12:53 { }
                }
            }
        }
        route-domain /Common/0
    }
    net route /Common/default {
        gw 192.168.1.1
        mtu 1500
        network default
    }

  • you can have decrypted tcpdump using this procedure which the result will be useful for analysis
    https://clouddocs.f5.com/training/community/adc/html/class4/module1/lab10.html
    https://my.f5.com/manage/s/article/K31793632

    even with non decrypted tcpdump, wireshark can show tls session setup sequence.

    at least in old version, health monitor uses management layer's network routing, which is different from what ltm traffic uses.
    so, health monitor OK doesnt always mean ltm traffic can reach backend server.

    • ahadem's avatar
      ahadem
      Icon for Nimbostratus rankNimbostratus

      what route path normally ltm uses and how it makes the packet route decisions? if you could elaborate more on that.

      • zamroni777's avatar
        zamroni777
        Icon for Nacreous rankNacreous

        the data plane uses routing config from the webgui Network tab

         

  • Can you please check the routes? As I can see, there's only one route default route for 192.168.x.x  Can you also check? Do you need any more routes for 10 dot subnet  for the internal interface 

     can you also check that you're using auto map in your VIP,   you can try SNAT , so could you please try apply SNAT ? 

    • ahadem's avatar
      ahadem
      Icon for Nimbostratus rankNimbostratus

      I tried SNAT as well and it did not work and there is only one route that I can add is the default route which I have added and I can not add a route for 10.10.x.x because it has a direct connection to it which is normally the self IP. 

       

      • If client ip should be transition and is not routable THROUGH the bigip you need SNAT activated with automap (than it uses float ip in HA or selfip in standalone) or you configure a snat pool

         

        If i see this correct, you give your virtual server the same IP as your pool member?