Proxy Protocol v2 Initiator
Problem this snippet solves: Proxy Protocol v1 related articles have already been posted on DevCentral, but there is no v2 support iRule code available. A customer wanted to support Proxy Protocol v2, so I wrote an iRule code for supporting v2. Proxy protocol for the BIG-IP (f5.com) How to use this snippet: Back-end server must handle Proxy header prior data exchange. Code : when CLIENT_ACCEPTED { # DEBUG On/Off set DEBUG 0 set v2_proxy_header "0d0a0d0a000d0a515549540a" # v2 version and command : 0x21 - version 2 & PROXY command set v2_ver_command "21" # v2 address family and transport protocol : 0x11 - AF_INET (IPv4) & TCP protocol set v2_af_tp "11" # v2 Address Size : 0x000C - 12 bytes for IPv4 + TCP set v2_address_length "000c" # Get TCP port - 2 byte hexadecimal format set src_port [format "%04x" [TCP::client_port]] set dst_port [format "%04x" [TCP::local_port]] # Get Src Address and convert to 4 byte hexadecimal format foreach val [split [IP::client_addr] "."] { append src_addr [format "%02x" $val] } # Get Dst Address and convert to 4 byte hexadecimal format foreach val [split [IP::local_addr] "."] { append dst_addr [format "%02x" $val] } # Build proxy v2 data set proxy_data [binary format H* "${v2_proxy_header}${v2_ver_command}${v2_af_tp}${v2_address_length}${src_addr}${dst_addr}${src_port}${dst_port}"] if { $DEBUG } { binary scan $proxy_data H* proxy_dump log local0. "[IP::client_addr]:[TCP::client_port]_[IP::local_addr]:[TCP::local_port] - proxy_data dump : $proxy_dump" } } when SERVER_CONNECTED { TCP::respond $proxy_data }Residential and Mobile Proxy Networks – The good and the not-so-good!
Keeping your privacy and identity under control in today's online world is critical, whether you're up to good or not-so-good things. That’s where residential and mobile proxies networks come in. These networks help hide your real IP address by making it look like your internet traffic is coming from regular people’s devices instead of data centers or well-known VPNs and proxies. These networks may resemble the TOR network, conceived initially to anonymize the internet using a decentralized network model to route traffic through volunteer-operated servers. Still, despite their similar nature, they have different architecture and drivers and are operated by private companies. Residential and Proxy Networks: A residential proxy network routes internet traffic through IP addresses assigned by the Internet Service Providers (ISPs) to homeowners. This makes traffic appear to be coming from a regular residential user when it hits its target. You can easily google the search term “residential proxy” and find that many companies are offering these services, allowing users to access geographically restricted content, perform web scraping without getting blocked by IP reputation systems, conduct competitive analysis without revealing their identity, and perform all sorts of cyberattacks, ranging from the reconnaissance phase up to the data exfiltration phase when the breach already has taken place. These services can also be leveraged for legitimate purposes like ad verification, market research, and SEO monitoring. Mobile Proxy Networks: Mobile proxy networks use IP addresses assigned to mobile devices by mobile carriers. These proxies provide an even higher level of legitimacy because mobile IPs rotate frequently and are associated with actual mobile devices or sometimes with regional CGNAT pools. This makes them particularly useful for tasks that require high anonymity and dynamic IP changes. Typically, the same companies that offer residential proxy services also have an offer for mobile proxy services. These are often used to test mobile apps and websites, manage social media accounts, bypass geographical restrictions on mobile content, and, let’s not forget, perform cyberattacks. How these networks operate Affiliation and Recruitment Programs Residential and mobile proxy companies often offer affiliate programs to incentivize developers to integrate their SDK into mobile apps, TV apps, browser extensions, VPN apps, etc. These programs allow developers to earn commissions or other benefits by integrating the SDK and becoming a network node to proxy traffic when remotely instructed by their “command-and-control” network. Figure 1 These affiliation programs can be a vital source of revenue for developers who still struggle to generate enough revenue from their applications. Figure 2 Some companies are stricter; others are not so much, but ultimately, it all depends on one’s ability to monitor what is being proxied by these SDKs to be able to prevent becoming part of a malicious Botnet, and this is a hard task to expect from a regular end-user. Below is a fragment from one of the SDK developer’s End-User Agreements. Figure 3 Traffic Flow Figure 4 Utilization for malicious activities: While residential and mobile proxies have legitimate uses, they are also increasingly used for cyberattacks. Here are some of the ways these networks are utilized for malicious activities: Web Scraping and Data Theft: Illegitimate Scraping: Today, with the AI hype, more than ever, DATA is GOLD and not only cybercriminals use residential mobile proxies to perform large-scale web scraping, extracting sensitive or proprietary information from websites without being detected or blocked. Credential Stuffing and Account Takeover: By blending their traffic using a mix of clean residential and mobile IPs and masking their identity, attackers can use stolen credentials to gain unauthorized access to user accounts across multiple platforms. Most importantly, they can validate the large dataset of credentials to ensure that when they sell it, a warranty is provided for the buyers. At the end of the day, Cybercriminals also need to keep their reputation, right? Carding: A very similar mechanism to Credential Stuffing applies to Carding, but here, cybercriminals can stealthily validate credit card numbers to make sure each one has not been flagged as compromised and is active for selling and being utilized by fraudsters. Gift Card Abuse: Fraudsters love Gift Cards because of their untraceable nature. Imagine combining that with the ability to brute-force numbers, validate and balance-check compromised ones. Distributed Denial of Service (DDoS) Attacks: Traffic Diversion: Residential and mobile proxies help in distributing attack traffic across numerous IP addresses, making it challenging for defenders to mitigate DDoS attacks effectively. Ad Fraud: Click Fraud: Attackers use these proxies to simulate legitimate clicks on ads, defrauding advertisers by generating fake traffic. Impression Fraud: By repeatedly loading advertisements through residential proxies, fraudsters can inflate the number of ad impressions, misleading advertisers about the reach and effectiveness of their ads. Bypassing Geo-blocks and Anti-fraud Mechanisms: Content Manipulation: Malicious actors use proxies to bypass geographical restrictions and access region-specific content or services. Avoiding Detection: Proxies help in evading anti-fraud systems designed to detect and block suspicious activities, thereby facilitating various forms of online fraud. Residential and mobile proxy networks provide a legitimate service for enhancing online privacy and enabling activities like market research and ad verification. However, their misuse for cyberattacks poses significant challenges for cybersecurity professionals. Understanding the dual-use nature of these technologies is essential for developing effective countermeasures and ensuring the internet remains a safe and secure environment. F5’s Bot and Fraud prevention solutions can distinguish between human-originated requests and software-originated requests by leveraging the ability to collect untamperable client-side signals. This unique capability is layered with our surveillance network, which tracks residential and mobile proxies using proprietary mechanisms. This offers our customers complete visibility and protection against malicious traffic originating from different sources, regardless of whether the attacker is blending their attacks using clean residential or mobile IP addresses.
Intermittent Net::ERR_CONNECTION_RESET Error and Incomplete Loading over HTTPS
I have an F5 load balancing setup configured with two servers. My MVC web application, which incorporates Kendo UI, Jquery, and bootstrapping, is hosted on an IIS server with an SSL certificate. However, when accessing the application via HTTPS from outside the server, it often or sometimes results in a 'net::ERR_CONNECTION_RESET' error, with intermittent failures to load javascript and CSS files to the client browser. Strangely, upon reloading the page, the assets load properly, and the page functions as expected. This issue did not occur when the application was accessed via HTTP, where it worked properly without any issues. What could be the reason behind this problem?
Loadbalancing a 2-stage proxy environment - persistence problems
Hello fellow F5-experts, My situation: I try to loadbalance Web-Proxy traffic. In genenal it seems to work, but we've run into some problems relating to websites with weird session handling. I sketched up the environment in the following Image A clients uses VS-A as it's HTTP proxy. This VS loadbalances to two of our own proxies in Pool A. Those are not transparent, so outgoing traffic toward their upstream proxy (VS-B) will have the Pool A node's IP as source address. The Proxies use the HTTP: Connect method. VS-B on the other hand, has a Pool B attached with a lot (as in really a lot) other proxy servers, that are not under our control (but are trustworthy, and have to be used. No way around it). VS-A: Type Standard HTTP-Profile: http-transparent (to be able to use LB method Fastest (node)) Source Address Translation: none Default Persistence Profile: source_addr Pool A: LB Method: Fastest (node) VS-A: Type Standard HTTP-Profile: http-transparent (to be able to use LB method Fastest (node)) Source Address Translation: SNAT Default Persistence Profile: none Pool B: LB Method: Fastest (node) Problem: On some websites, people complain about loosing their sessions. I tracked it down to the VS-B, which can not persist connections to a website to the same node in Pool B. Since the VS-B does not see the original client IP, but only the 2 proxy IP's I have no idea how to establish a propper persistence. For a very important website, i wrote an iRule that "hard-binds" to a single Node in Pool B, based on the HTTP:URI. My question: Is my config any viable? Or what should I change? I read a lot of article here, but I never had the feeling, that those met my situation with the "2-stage" proxy environment. I was thinking about adding a http header with the original client IP by an iRule in VS-A so I have something to make a persistence decision in VS-B, but I'm not sure how to do that. I also often read about applying the OneConnect profile combined with proxy loadbalancing, but I don't think I really understood the reason and/or benefit, nor do I know where to apply it in this environment. I hope I made it some kind of clear what I try to accomplish and where the problems are, since English is not my mother language and I'm a but rusted using it. Any help or hint is very much appreciated. Thanks in advance, ichnafi EDIT:(28th Feb) I just found out, that the desired website can be acquired from the HTTP::host header even by Vs-B. So would it be possible to establish a persinstence for the combination client IP and HTTP::host header? EDIT2: Possible Solution found (1st Mar) I currently endet up with an iRule creating a universal persistence based on the http::host header. the iRule is then bound to a universal Persistense-Profile. The iRule looks lilke this: when HTTP_REQUEST { persist uie [HTTP::host] } I'm thinking of maybe creating a custom header that contains a combination of client_ip and http:host value to get an even more definite persistence, but not quite sure about that. What do you guys think?
FTp proxy via iRule
Hello, I'm trying to do an anonymous ftp connection through an F5 irule. My logs of the irule look like this: Rule /Common/iRule_ftp_proxy <CLIENT_ACCEPTED>: client FTP accepted Rule /Common/iRule_ftp_proxy <CLIENT_DATA>: client payload - USER anonymous@193.190.198.27 Rule /Common/iRule_ftp_proxy <CLIENT_DATA>: sitename:193.190.198.27 - cmd:USER - uid:anonymous Rule /Common/iRule_ftp_proxy <CLIENT_DATA>: address 193.190.198.27 port 21 Rule /Common/iRule_ftp_proxy <SERVER_CONNECTED>: connected to server Rule /Common/iRule_ftp_proxy <SERVER_DATA>: server payload 220-Welcome to the Belnet public FTP server ftp.belnet.be !This server is located in Brussels, Belgium and operated by Belnet, the BelgianEducation and Research Network. If you have any problem, question or mirrorrequest, please send them to ftpmaint@belnet.be.This archive is available through the following means:RSYNC rsync://rsync.belnet.be (IPv4)HTTP http://ftp.belnet.be (IPv4 + IPv6)FTP ftp://ftp.belnet.be (IPv4 + IPv6) Rule /Common/iRule_ftp_proxy <SERVER_DATA>: server found 220 ok Rule /Common/iRule_ftp_proxy <SERVER_DATA>: server payloadNote: opening too many parallel connections to this host is considered an abuse.All access is logged.Currently used storage capacity : 34T / 100T on /ftp220 193.190.198.27 FTP server ready Rule /Common/iRule_ftp_proxy <SERVER_DATA>: server payload 331 Anonymous login ok, send your complete email address as your password When I check with a wireshark capture on my client; the '331 Anonymous login ok, send your complete email address as your password' never reaches the client. Without the proxy, it does reach the client and filezilla knows how to deal with it. This is my server_data in the irule; when SERVER_DATA { if { $static::debug } { log local0. "server payload [TCP::payload]" } if { [TCP::payload] starts_with "220" }{ if { $static::debug } { log local0. "server found 220 ok" } TCP::respond "USER $uid\r\n" TCP::payload replace 0 [TCP::payload length] "" } TCP::release TCP::collect } I also tried modifying the rule like this: when SERVER_DATA { if { $static::debug } { log local0. "server payload [TCP::payload]" } if { [TCP::payload] starts_with "220" }{ if { $static::debug } { log local0. "server found 220 ok" } TCP::respond "USER $uid\r\n" TCP::payload replace 0 [TCP::payload length] "" } if { [TCP::payload] starts_with "331" }{ TCP::respond "PASS $uid@example.com\r\n" TCP::payload replace 0 [TCP::payload length] "" } TCP::release TCP::collect } This get's me one step further, but I'm still not able to connect. When I check with a wireshark capture on my client; the '331 Anonymous login ok, send your complete email address as your password' never reaches the client. Without the proxy, it does reach the client and filezilla knows how to deal with it. Rule /Common/iRule_ftp_proxy <SERVER_DATA>: server payload 230 Anonymous access granted, restrictions apply Why is the '331 server_data' not being forwarded to my client so it can respond to it, or is this the task of the proxy? And why is the '230 server data' not reaching the client? CheersFTP proxy data connection
I've used this ftp proxy for ftp connection. https://devcentral.f5.com/codeshare/ftp-proxy-v10-and-up My iRule when CLIENT_ACCEPTED { TCP::respond "220 Welcome to the F5 FTP Proxy v10\r\n" log local0. "client accepted" TCP::collect } when CLIENT_DATA { set ftplogin [TCP::payload] log local0. "login is $ftplogin" if { $ftplogin starts_with "USER" } { You can set it up to use any DNS server you want. set dnslookup "10.191.31.60" set cuser [TCP::payload] scan $cuser {%[^@]@%s} garbage sitename scan $garbage %s%s cmd uid log local0. "$sitename" set ips [RESOLV::lookup @/OpusCapita/vs_dns_test -a $sitename] set ips $sitename if {$ips eq "" } { Input wasn't an IP address, take some default action? log local0. "$ips Input wasn't an IP address" reject } else { TCP::payload replace 0 [TCP::payload length] "" set ftp_serv "[lindex $ips 0]" node [lindex $ips 0] [TCP::local_port] log local0. "member adresa [lindex $ips 0] port [TCP::local_port]" } } TCP::release } when SERVER_CONNECTED { TCP::collect } when SERVER_DATA { set serv_data [TCP::payload] if { $serv_data contains "220" } { TCP::respond "USER $uid\r\n" log local0. "user= "USER $uid\r\n" " TCP::payload replace 0 [TCP::payload length] "" } TCP::release } User can connect to my FP proxy VIP and connect to dest ($sitename) server. At the moment don´t use DNS. But when user try trasfer some data to FTP server. User see only timeout and no data sent. Can you please check where can be problem? my ftp profile Name proxy_ftp Parent Profile ftp ranslate Extended Enabled Inherit Parent Profile Enabled Data Port 0 And log from ftp server [Cway2:/home/jtoivola/TEMP] date ; ftp 82.180.230.19 121 Fri May 20 12:31:06 EEST 2016 Connected to 82.180.230.19. 220 Welcome to the F5 FTP Proxy v10 Name (82.180.230.19:jtoivola): cwjuha1@82.180.220.75 331 Password required for cwjuha1. Password: 230 User CWay2005Juha1 logged in. Remote system type is UNIX. ftp-child [32279] <05/20-12:37:47> TECH-DBG from Server-PI (3): '230 User CWay2005Juha1 logged in.' ftp-child [32279] <05/20-12:37:47> USER-INF 'SYST' from 82.180.230.19 ftp-child [32279] <05/20-12:37:47> TECH-DBG from Server-PI (3): '215 UNIX Cway' ftp-child [32279] <05/20-12:37:51> USER-INF 'PORT 82.180.230.19:51418' from 82.180.230.19 ftp-child [32279] <05/20-12:37:51> USER-INF 'STOR testi' from 82.180.230.19 ftp-child [32279] <05/20-12:37:51> TECH-DBG from Server-PI (3): '227 Entering Passive Mode (10,190,8,6,225,252)' ftp-child [32279] <05/20-12:37:51> TECH-DBG got SRV-PASV 10.190.8.6:57852 for 82.180.230.19:28912 ftp-child [32279] <05/20-12:37:51> TECH-INF 'STOR testi' sent for 82.180.230.19 ftp-child [32279] <05/20-12:37:51> TECH-DBG from Server-PI (3): '150 ASCII data connection for testi (82.180.220.75,36979).' ftp-child [32279] <05/20-12:37:55> USER-INF Transfer for 82.180.230.19 completed: STOR 'testi' read 0/4 byte/sec ftp-child [32279] <05/20-12:37:55> TECH-DBG from Server-PI (3): '552 testi: An invalid argument value was given.'
Websites do not load correctly when load balancing via proxy
We currently have a pair of BIG-IPs with 11.5 running in our DC. One of the services we want to load balance is a pair of Cisco WSAs (IronPort) which function as web proxies. When a client connects via the BIG-IP's VIP to access the Proxies we have the problem that not all of the content is loaded. This problem does not change if we take one of the WSAs out of the pool so that we can be sure we always go via the same proxy. It is also working fine when the clients go via one of the proxies directly. [UPDATE] The http-WSA-proxy profile is based obn the fastL4 but has XFF enabled. Does anyone have an idea what we are missing and why we are not receiving the complete page? High level traffic flow: Client <> BIG-IP Cluster <> Firewall <> 2x Cisco WSA Web Proxy <> Firewall <> Internet LTM config: ltm virtual vs_NAME { destination VIP%RD:webcache ip-protocol tcp mask 255.255.255.255 partition NAME persist { source_addr_mirror { default yes } } pool NAME profiles { /Common/fastL4 { } http-WSA-proxy { } } source 0.0.0.0/0 source-address-translation { type automap } vlans { NAME-VIPs } vlans-enabled vs-index 17 }
Any iRules that acts as Virtual Server for By-Pass Cert
F5 APM with SWG module, so this F5 acts as Proxy and Intercept Cert. I have a problem about intercept certificate some website cannot use it, then I solved that problem by create the new virtual machine and fixed the destination of each website's IP. (nslookup) But I think it's not a good solution, because If some website occurs like this problem more, I have to add more virtual server. So I try to use iRules to by-pass the destination by using iRules. when CLIENT_ACCEPTED { if { [ IP::Addr [IP::local_addr] equals "xxx.xxx.xxx.xxx" ] } { SSL::disable } } But it's did not work, please could you suggest me for the iRules command.
Proxy Protocol Receiver
Problem this snippet solves: iRule for BIG-IP to receive PROXY protocol (v1 and v2) header in TCP Payload and remove it before forwarding remaining TCP Payload to server side pool member. How to use this snippet: Enable iRule on virtual server where upstream proxy will be sending PROXY header. Testing done using proxied IPv6 and IPv4 HTTP connections from HAProxy using Proxy Protocol v1 and v2. Use of client or server SSL profiles slated for testing and validation since ELB promotes use of Proxy Protocol as a solution for customers that don't want to have ELB terminate HTTPS traffic but do want servers to see original IP addresses. Code : #PROXY Protocol Receiver iRule # c.jenison at f5.com (Chad Jenison) # v2.0 - Added support for PROXY Protocol v2, control for v1,v2 or lack of proxy via static:: variables set in RULE_INIT # v2.1 - Fix for skipping bytes in v2 code when RULE_INIT { set static::allowProxyV1 0 set static::allowProxyV2 1 set static::allowNoProxy 0 } when CLIENT_ACCEPTED { TCP::collect } when CLIENT_DATA { binary scan [TCP::payload 12] H* v2_protocol_sig if {$static::allowProxyV1 && [TCP::payload 0 5] eq "PROXY"} { set proxy_string [TCP::payload] set proxy_string_length [expr {[string first "\r" [TCP::payload]] + 2}] scan $proxy_string {PROXY TCP%s%s%s%s%s} tcpver srcaddr dstaddr srcport dstport log "Proxy Protocol v1 conn from [IP::client_addr]:[TCP::client_port] for an IPv$tcpver stream from Src: $srcaddr:$srcport to Dst: $dstaddr:$dstport" TCP::payload replace 0 $proxy_string_length "" } elseif {$static::allowProxyV2 && $v2_protocol_sig eq "0d0a0d0a000d0a515549540a"}{ binary scan [TCP::payload] @12H* v2_proxyheaderremainder binary scan [TCP::payload] @12H2H* v2_verCommand v2_remainder if {$v2_verCommand == 21}{ binary scan [TCP::payload] @13H2 v2_addressFamilyTransportProtocol if {$v2_addressFamilyTransportProtocol == 11} { binary scan [TCP::payload] @16ccccccccSS v2_sourceAddress1 v2_sourceAddress2 v2_sourceAddress3 v2_sourceAddress4 v2_destAddress1 v2_destAddress2 v2_destAddress3 v2_destAddress4 v2_sourcePort1 v2_destPort1 set v2_sourceAddress "[expr {$v2_sourceAddress1 & 0xff}].[expr {$v2_sourceAddress2 & 0xff}].[expr {$v2_sourceAddress3 & 0xff}].[expr {$v2_sourceAddress4 & 0xff}]" set v2_destAddress "[expr {$v2_destAddress1 & 0xff}].[expr {$v2_destAddress2 & 0xff}].[expr {$v2_destAddress3 & 0xff}].[expr {$v2_destAddress4 & 0xff}]" set v2_sourcePort [expr {$v2_sourcePort1 & 0xffff}] set v2_destPort [expr {$v2_destPort1 & 0xffff}] log "Proxy Protocol v2 conn from [IP::client_addr]:[TCP::client_port] for an IPv4 Stream from Src: $v2_sourceAddress:$v2_sourcePort to Dst: $v2_destAddress:$v2_destPort" binary scan [TCP::payload] @14S address_size set skip_bytes [expr 16 + $address_size] TCP::payload replace 0 $skip_bytes "" } elseif {$v2_addressFamilyTransportProtocol == 21} { binary scan [TCP::payload] @16H4H4H4H4H4H4H4H4 v2_v6sourceAddress1 v2_v6sourceAddress2 v2_v6sourceAddress3 v2_v6sourceAddress4 v2_v6sourceAddress5 v2_v6sourceAddress6 v2_v6sourceAddress7 v2_v6sourceAddress8 binary scan [TCP::payload] @32H4H4H4H4H4H4H4H4 v2_v6destAddress1 v2_v6destAddress2 v2_v6destAddress3 v2_v6destAddress4 v2_v6destAddress5 v2_v6destAddress6 v2_v6destAddress7 v2_v6destAddress8 binary scan [TCP::payload] @48SS v2_v6sourcePort1 v2_v6destPort1 set v2_v6sourcePort [expr {$v2_v6sourcePort1 & 0xffff}] set v2_v6destPort [expr {$v2_v6destPort1 & 0xffff}] set v2_v6sourceAddress "$v2_v6sourceAddress1:$v2_v6sourceAddress2:$v2_v6sourceAddress3:$v2_v6sourceAddress4:$v2_v6sourceAddress5:$v2_v6sourceAddress6:$v2_v6sourceAddress7:$v2_v6sourceAddress8" set v2_v6destAddress "$v2_v6destAddress1:$v2_v6destAddress2:$v2_v6destAddress3:$v2_v6destAddress4:$v2_v6destAddress5:$v2_v6destAddress6:$v2_v6destAddress7:$v2_v6destAddress8" log "Proxy Protocol v2 conn from from [IP::client_addr]:[TCP::client_port] for an IPv6 Stream from Src: $v2_v6sourceAddress:$v2_v6sourcePort to Dst: $v2_v6destAddress:$v2_v6destPort" binary scan [TCP::payload] @14S address_size set skip_bytes [expr 16 + $address_size] TCP::payload replace 0 $skip_bytes "" } else { log "v2_proxy conn from [IP::client_addr]:[TCP::client_port] - possible unknown/malformed transportProtocol or addressFamily" reject } } elseif {$v2_verCommand == 20}{ log "Proxy Protocol v2 and LOCAL command from [IP::client_addr]:[TCP::client_port]; skipping" binary scan [TCP::payload] @14S address_size set skip_bytes [expr 16 + $address_size] TCP::payload replace 0 $skip_bytes "" binary scan [TCP::payload] H* local_remainder } else { log "Proxy Protocol Protocol Signature Detected from [IP::client_addr]:[TCP::client_port] but protocol version and command not legal; connection reset" reject } } elseif {$static::allowNoProxy} { log "Connection from [IP::client_addr]:[TCP::client_port] allowed despite lack of PROXY protocol header" } else { reject log "Connection rejected from [IP::client_addr]:[TCP::client_port] due to lack of PROXY protocol header" } TCP::release } Tested this on version: 12.0
