WAF evasion techniques for Command Injection
Let’s talk about Command Injection; I’m going to talk about this specifically from the perspective of Web Application Firewalls (like BIG-IP Advanced WAF, BIG-IP Next WAF, F5 Distributed Cloud WAF and so on) but these concepts are generally applicable anywhere user-input is used to construct commands run on the system, directly or indirectly. So, what is Command Injection? To quote OWASP, who put it very nicely: Command injection is an attack in which the goal is execution of arbitrary commands on the host operating system via a vulnerable application. Command injection attacks are possible when an application passes unsafe user supplied data (forms, cookies, HTTP headers etc.) to a system shell. In this attack, the attacker-supplied operating system commands are usually executed with the privileges of the vulnerable application. Command injection attacks are possible largely due to insufficient input validation. This attack differs from Code Injection, in that code injection allows the attacker to add their own code that is then executed by the application. In Command Injection, the attacker extends the default functionality of the application, which execute system commands, without the necessity of injecting code. Like I say, in this case I’m going to talk about command injection to web applications, but they can happen in almost any piece of software that works on untrusted user input. Perhaps the most famous example of a command injection vulnerability is Shellshock, a suite of vulnerabilities in the Unix Bash shell and if you needed any proof that they can be hard things to find as a defender, Shellshock lived, undiscovered (or at least undisclosed, we’ve no way of proving that no malicious entities knew of the bug!) for 25 years from 1989 to 2014, in one of the most widely used pieces of software in the world. The original Shellshock vulnerability involved a maliciously crafted environment variable containing (malicious) commands after a function definition, e.g. env x=’() { :;}; echo vulnerable’ bash -c “echo test” On a vulnerable system, running the above commands would display “vulnerable” because of Bash continuing to execute the (injected) commands following the function definition. Injecting a command here requires the"| use of two specific characters plus the command, the semi-colon and space characters. If you imagine a web application passing commands to bash on a vulnerable system, you’ll see that it would be possible to block this attack simply by blocking requests containing semi-colon or space (or indeed having a signature for the full function definition and trailing semi-colon of “{ :;};”) Bypassing protections Any time there is a WAF in front of a vulnerable system – and sometimes even when there isn’t – an attacker must try to evade the rules preventing them from simply injecting their chosen command. You’ll often see this when attackers or scanners are looking for SQL injection vulnerabilities in web applications, replacing characters like ‘ with %27 or the space with %20 (and many other tricks), or using chunks of existing text with the SUBSTRING() function to construct queries without having to use the actual text. Many of the same tricks work for command injection vulnerabilities, and I’d like to talk about a specific example here because it’s one I hadn’t considered until it turned up in some real life traffic.. Bypassing WAF signatures using Environment Variables Remember I just said you could construct SQL queries using sub-strings of existing text? Well, if your target system is Windows-based and you know there’s a command injection vulnerability but you’re unable to exploit it due to character blocks or similar restrictions, then good news! Windows environment variables might be what you’re looking for.. Environment variables exist in most operating systems, and Microsoft ones are no exception – they date back to DOS and were one of the enhancements Microsoft brought to the table over and above CP/M (unlike the 8.3 filenames, which came right from CP/M!); their behaviour has been pretty much the same throughout, and there have always been a number of ‘default’ environment variables like the PATH, TMP & TEMP. Current versions of Windows add a number of additional default environment variables like PROGRAMDATA, PROGRAMFILES etc. Windows also allows the shell to return just a part of the environment variable value using the following syntax: %VARIABLE:~start_pos,end_pos% How is this useful to us, you ask? Let’s say you know you can inject a command, but you need a space in your command line; you want to inject “ping 127.0.0.1” but the space is dropped or the request is blocked by a WAF looking for “ping <IP>”, well then you just need an environment variable you know will have a space in it! %PROGRAMFILES%, by default, is going to be set to C:\Program Files on most systems, which has a space right there in the middle! All we need to do to get to it is use it as %PROGRAMFILES:~10,1%, for example: ping%PROGRAMFILES:~10,1%127.0.0.1 Go ahead, fire up a command prompt, and try it out! You could even construct the whole command that way: %PROGRAMFILES:~3,1%%SYSTEMROOT:~4,2%%PROGRAMFILES:~6,1%%PROGRAMFILES:~10,1%127.0.0.1 Again, fire up a command prompt and give that a try! Protect against bypasses with BIG-IP Advanced WAF Now here’s the good news: ASM includes signatures, by default, for all of those useful Windows environment variables (and the same for many other systems, too), so if you were to try the above on a vulnerable system with the right signatures in the policy, you’d still be blocked – like this: All these signatures are part of the Predictable Resource Location Signatures signature set, so you’ll want to make sure you either have all signatures of Medium or above, or at least this set assigned to your policy: Summary Command Injection is a huge topic, much bigger than I can talk about in one blog post here, but hopefully this shows you one way an attacker might try to evade protection in front of a vulnerable Windows system, and some ways in which you can protect it — BIG-IP Advanced WAF or F5 Distributed Cloud WAF both have signatures for this kind of evasion.The Business Partner Exchange - An F5 Distributed Cloud Services Demonstration
Large enterprises face challenges when deploying applications at scale, including managing application sprawl, segregating partner and customer traffic, and maintaining consistent security policies. To address these issues, comprehensive traffic management, policy enforcement, and resource allocation are essential for seamless and secure application deployment. The Business Partner Exchange demo illustrates how F5 distributed cloud services with Equinix effectively addresses these challenges.Simplify Network Segmentation for Hybrid Cloud
Introduction Enterprises have always had the need to maintain separate development and production environments. Operational efficiency, reduction of blast radius, security and compliance are generally the common objectives behind separating these environments. By dividing networks into smaller, isolated segments, organizations can enhance security, optimize performance, and ensure regulatory compliance. This article demonstrates a practical strategy for implementing network segmentation in modern multicloud environments that also connect on-prem infrastructure. This uses F5 Distributed Cloud (F5 XC) services to connect and secure network segments in cloud environments like Amazon Web Services (AWS) and on-prem datacenters. Need for Segmentation Network segmentation is critical for managing complex enterprise environments. Traditional methods like Virtual Routing and Forwarding (VRFs) and Multiprotocol Label Switching (MPLS) have long been used to create isolated network segments in on-prem setups. F5 XC ensures segmentation in environments like AWS and it can extend the same segmentation to on-prem environments. These techniques separate traffic, enhance security, and improve network management by preventing unauthorized access and minimizing the attack surface. Scenario Overview Our scenario depicts an enterprise with three different environments (prod, dev, and shared services) extended between on-prem and cloud. A 3rd party entity requires access to a subset of the enterprise's services. This article, covers the following two networking segmentation use-cases: Hybrid Cloud Transit Extranet (servicing external 3 rd party partners/customers) Hybrid Cloud Transit Consider an enterprise with three distinct environments: Production (Prod), Development (Dev), and Shared Services. Each environment requires strict isolation to ensure security and performance. Using F5 XC Cloud Connect, we can assign each VPC a network segment effectively isolating the VPC’s. Segments in multiple locations (or VPC’s) can traverse F5 XC to reach distant locations whether in another cloud environment or on-prem. Network segments are isolated by default, for example, our Prod segment cannot access Shared. A segment connector is needed to allow traffic between Prod and Shared. The following diagram shows the VPC segments, ensuring complete "ships in the night" isolation between environments. In this setup, Prod, Dev, and Shared Services environments operate independently and are completely isolated from one another at the control plane level. This ensures that any issues or attacks in one environment do not affect the others. Customer Requirement: Shared Services Access Many enterprises deploy common services across their organization to support internal workloads and applications. Some examples include DHCP, DNS, NTP, and NFS, services that need to be accessible to both Prod and Dev environments while keeping Prod and Dev separate from each other. Segment Connectors is a method to allow communication between two isolated segments by leaking the routes between the source and destination segments. It is important to note that segment connector can be of type Direct or SNAT. Direct allows bidirectional communication between segments whereas the SNAT option allows unidirectional communication from the source to the destination. Extending Segmentation to On-Premises Enterprises already use segmented networks within their on-premises infrastructure. Extending this segmentation to AWS involves creating similar isolated segments in the cloud and establishing secure communication channels. F5 XC allows you to easily extend this segmentation from on-prem to the cloud regardless of the underlay technology. In this scenario, communication between the on-premises Prod segment and its cloud counterpart is seamless, and the same also applies for the Dev segment. Meanwhile Dev and Prod stay separate ensuring that existing security and isolation is preserved across the hybrid environment. Extranet In this scenario an external entity (customer/partner) needs access to a few applications within our Prod segment. There are two different ways to enable this access, Network-centric and App-centric. Let’s refer to the external entity as Company B. In order to connect Company B we generally need appropriate cloud credentials, but Company B will not share their cloud credentials with us. To solve this problem, F5 XC recommends using AWS STS:AssumeRole functionality whereby Company B creates an AWS IAM Role that trusts F5 XC with the minimum privileges necessary to configure Transit Gateway (TGW) attachments and TGW route table entries to extend access to the F5 XC network or network segments. Section 1 – Network-centric Extranet Many times, partners & customers need to access a unique subset of your enterprise’s applications. This can be achieved with F5 XC’s dedicated network segments and segment connectors. With a segment connector for the external and prod network segments, we can give Company B access to the required HTTP service without gaining broader access to other non-Prod segments. Locking Down with Firewall Policies We can implement a Zero Trust firewall policy to lock down access from the external segment. By refining these policies, we ensure that third-party consumers can only access the services they are authorized to use. Our firewall policy on the CE only allows access from the external segment to the intended application on TCP/80 in Prod. [ec2-user@ip-10-150-10-146 ~]$ curl --head 10.1.10.100 HTTP/1.1 200 OK Server: nginx/1.24.0 (Ubuntu) Date: Thu, 30 May 2024 20:50:30 GMT Content-Type: text/html Content-Length: 615 Last-Modified: Wed, 22 May 2024 21:35:11 GMT Connection: keep-alive ETag: "664e650f-267" Accept-Ranges: bytes [ec2-user@ip-10-150-10-146 ~]$ ping -O 10.1.10.100 PING 10.1.10.100 (10.1.10.100) 56(84) bytes of data. no answer yet for icmp_seq=1 no answer yet for icmp_seq=2 no answer yet for icmp_seq=3 ^C --- 10.1.10.100 ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 3153ms After applying the new policies, we confirm that the third-party access is restricted to the intended services only, enhancing security and compliance. This demonstrates how F5 Distributed Cloud services enable networking segmentation across on-prem and cloud environments, with granular control over security policies applied between the segments. Section 2 - App-centric Extranet In the scenario above, Company B can directly access one or more services in Prod with a segment connector and we’ve locked it down with a firewall policy. For the App-centric method, we’ll only publish the intended services that live in Prod to the external segment. App-centric connectivity is made possible without a segment connector by using load balancers within App Connect that target the application within the Prod segment and advertises its VIP address to the external segment. The following illustration shows how to configure each component in the load balancer. Visualization of Traffic Flows The visualization flow analysis tool in the F5 XC Console shows traffic flows between the connected environments. By analyzing these flows, particularly between third-party consumers and the Prod environment, we can identify any unintended access or overreach. The following diagram is for a Network-centric connection flow: This following diagram shows an App-centric connection flow using the load balancer: Product Feature Demo Conclusion Effective network segmentation is a cornerstone of secure and efficient cloud environments. We’ve discussed how F5 XC enables hybrid cloud transit and extranet communication. Extranet can be done with either a network centric or app-centric deployment. F5 XC is an end to end platform that manages and orchestrates end-to-end segmentation and security in hybrid-cloud environments. Enterprises can achieve comprehensive segmentation, ensuring isolation, secure access, and compliance. The strategies and examples provided demonstrate how to implement and manage segmentation across hybrid environments, catering to diverse requirements and enhancing overall network security. Additional Resources More features and guidance are provided in the comprehensive guide below, where showing exactly how you can use the power and flexibility of F5 Distributed Cloud and Cloud Connect to deliver a Network-centric approach with a firewall and an App-centric approach with a load balancer. Create and manage segmented networks inyour own cloud and on-prem environments, and achieve the following benefits: Ability to isolate environments within AWS Ability to extend segmentation to on-prem environments Ability to connect external partners or customers to a specific segment Use Enhanced Firewall Policies to limit access and reduce the blast radius Enhance the compliance and regulatory requirements by isolating sensitive data and systems Visualize and monitor the traffic flows and policies across segments and network domains Workflow Guide - Secure Network Fabric (Multi-Cloud Networking) YouTube: Using network segmentation for hybrid-cloud and extranet with F5 Distributed Cloud Services DevCentral:Secure Multicloud Networking Article Series GitHub: S-MCN Use-case Playbooks (Console, Automation) for F5 Distributed Cloud Customers F5.com: Product Information Product Documentation Network Segmentation Cloud Connect Network Segment Connectors App Security App Networking CE Site ManagementAccess Troubleshooting: BIG-IP APM OIDC integration
Introduction Troubleshooting Access use cases can be challenging due to the interconnected components used to achieve such use cases. A simple example for Active Directory authentication can go through below challenges, DNS resolution of Domain Controller (DC) configured. Reachability between F5 and DC. Communication ports used. Domain account privileges. Looking at the issue of non-working Active Directory (AD) authentication is a complex task, yet looking at each component to verify the functionality is much easier and shows output the influence further troubleshooting actions. Implementation and troubleshooting We discussed the implementation of OpenID Connect over here Let's discuss here how we can troubleshoot issues in OIDC implementation, here's a summary of the main points we are checking Role Troubleshooting main points OAuth Authorization Server DNS resolution for the authentication destination. Routing setup to the authentication system. Authentication configurations and settings. Scope settings. Token signing and settings. OAuth Client DNS resolution for the authorization server. Routing setup. Token settings. Authorization attributes and parameters. OAuth Resource Server Token settings. Scope settings Looking at the main points, you can see the common areas we need to check while troubleshooting OAuth / OIDC solutions, below are the troubleshooting approach we are following, Check the logs. APM logging provides a comprehensive set of logs, the main logs to be checked apm, ltm and tmm. DNS resolution and check DNS resolver settings. Routing setup. Authentication methods settings. OAuth settings and parameters. Check the logs The logs are your true friends when it comes to troubleshooting. We start by creating debug logging profile Overview > Event logs > Setting. Select the target Access Policy to apply the debug profile. Case 1: Connection reset after authentication In this case the below is the connection sequence, User accessing through F5 acting as Client + RS. Users are redirected to OAuth provider for authentication. User is redirected back to F5 but connection resets at this point. Troubleshooting steps: Checking logs by clicking the session ID fromAccess > Overview. From the below logs we can see the logon was successful but somehow the Authorization code wasn’t detected. One main reason would be mismatched settings between Auth server and Client configurations. In our setup I’m using provider flow type as Hybrid and format code-idtoken. Local Time 2024-06-11 06:47:48 Log Message /Common/oidc_google_t1.app/oidc_google_t1:Common:204adb19: Session variable 'session.logon.last.result' set to '1' Partition Common Local Time 2024-06-11 06:47:49 Log Message /Common/oidc_google_t1.app/oidc_google_t1:Common:204adb19: Authorization code not found. Partition Common Checking back the configuration to validate the needed flow type: adjust flow type at the provider settings to beAuthorization Code instead of Hybrid. Case 2: Expired JWT Keys In this case the below is the connection sequence, User accessing through F5 acting as Client + RS. Users are redirected to OAuth provider for authentication. User is redirected back to F5 with Access denied. Troubleshooting steps: Checking logs by clicking the session ID from Access > Overview. From the below logs we can see the logon was successful but somehow the Authorization code wasn’t detected. One main reason can be the need to rediscover JWT keys. Local Time 2024-06-11 06:51:06 Log Message /Common/oidc_google_t1.app/oidc_google_t1:Common:848f0568: Session variable 'session.oauth.client.last.errMsg' set to 'None of the configured JWK keys match the received JWT token, JWT Header: eyJhbGciOiJSUzI1NiIsImtpZCI6ImMzYWJlNDEzYjIyNjhhZTk3NjQ1OGM4MmMxNTE3OTU0N2U5NzUyN2UiLCJ0eXAiOiJKV1QifQ' Partition Common The action to be taken would be to rediscover the JWT keys if they are automatic or add the new one manually. Head toAccess ›› Federation : OAuth Client / Resource Server : Provider Select the created provider. Click Discover to fetch new keys from provider Save and apply the new policies settings. Case 3: OAuth Client DNS resolver failure In this case the below is the connection sequence, User accessing through F5 acting as Client + RS. Users are redirected to OAuth provider for authentication. User is redirected back to F5 with Access denied. Troubleshooting steps: Checking logs by clicking the session ID fromAccess > Overview. From the below logs we can see the logon was successful but somehow the Authorization code wasn’t detected. Another reason for such behavior can be the DNS failure to reach to OAuth provider to validate JWT keys. Local Time 2024-06-12 19:36:12 Log Message /Common/oidc_google_t1.app/oidc_google_t1:Common:fb5d96bc: Session variable 'session.oauth.client.last.errMsg' set to 'HTTP error 503, DNS lookup failed' Partition Common Checking DNS resolver Network ›› DNS Resolvers : DNS Resolver List Validate resolver config. is correct. Check route to DNS server Network ›› Routes Note, DNS resolver uses TMM traffic routes not the management plane system routing. Case 4: Token Mismatch In this case the below is the connection sequence, User accessing through F5 acting as Client + RS. Users are redirected to OAuth provider for authentication. User is redirected back to F5 with Access denied. Troubleshooting steps: Checking logs by clicking the session ID fromAccess > Overview. We will find the logs showing Bearer token is received yet no token enabled at the client / resource server connections. Local Time 2024-06-21 07:25:12 Log Message /Common/f5_local_client_rs.app/f5_local_client_rs:Common:c224c941: Session variable 'session.oauth.client./Common/f5_local_client_rs.app/f5_local_client_rs_oauthServer_f5_local_provider.token_type' set to 'Bearer' Partition Common Local Time 2024-06-21 07:25:12 Log Message /Common/f5_local_client_rs.app/f5_local_client_rs:Common:c224c941: Session variable 'session.oauth.scope./Common/f5_local_client_rs.app/f5_local_client_rs_oauthServer_f5_local_provider.errMsg' set to 'Token is not active' Partition Common We need to make sure client and resource server have JWT token enabled instead of opaque and proper JWT token is selected. Case 5: Audience mismatch In this case the below is the connection sequence, User accessing through F5 acting as Client + RS. Users are redirected to OAuth provider for authentication. User is redirected back to F5 with Access denied. Troubleshooting steps: Checking logs by clicking the session ID fromAccess > Overview. We will find the logs stating incorrect or unmatched audience. Local Time 2024-06-23 21:32:42 Log Message /Common/f5_local_client_rs.app/f5_local_client_rs:Common:42ef6c51: Session variable 'session.oauth.scope.last.errMsg' set to 'Audience not found : Claim audience= f5local JWT_Config Audience=' Partition Common Case 6: Scope mismatch In this case the below is the connection sequence, User accessing through F5 acting as Client + RS. Users receive authorization error with wrong scope. Troubleshooting steps: Checking logs by clicking the session ID fromAccess > Overview. Scope name is mentioned in the logs, in this case I named it “wrongscope” You will see scope includes openid string, this is because we have openid enabled. Change the scope to the one configured at the provider side. Local Time 2024-06-24 06:20:28 Log Message /Common/oidc_google_t1.app/oidc_google_t1:Common:edacbe31:/Common/oidc_google_t1.app/oidc_google_t1_act_oauth_client_0_ag: OAuth: Request parameter 'scope=openid wrongscope' Partition Common Case 7: Incorrect JWT Signature In this case the below is the connection sequence, User accessing through F5 acting as Client + RS. Users are redirected to OAuth provider for authentication. User is redirected back to F5 with Access denied. Troubleshooting steps: Checking logs by clicking the session ID fromAccess > Overview. We will find the logs showing Bearer token is received yet no token enabled at the client / resource server connections. Local Time 2024-06-21 07:25:12 Log Message /Common/f5_local_client_rs.app/f5_local_client_rs:Common:c224c941: Session variable 'session.oauth.scope./Common/f5_local_client_rs.app/f5_local_client_rs_oauthServer_f5_local_provider.errMsg' set to 'Token is not active' Partition Common When trying to renew the JWT key we see this error in the GUI. An error occurred: Error in processing URL https://accounts.google.com/.well-known/openid-configuration. The message is - javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target We need at this step to validate the used CA bundle and if we need to allow the trust of expired or self-signed JWT tokens. General issues In addition to the listed cases above, we have some general issues: DNS failure at client side not able to reach whether the F5 virtual server or OAuth provider to provide authentication information. In this case, please verify DNS configurations and Network setup on the client machine. Validate HTTP / SSL / TCP profiles at the virtual server are correctly configured. Related Content DNS Resolver Overview BIG-IP APM deployments using OAuth/OIDC with Microsoft Azure AD may fail to authenticate OAuth and OpenID Connect - Made easy with Access Guided Configurations templates Request and validate OAuth / OIDC tokens with APM F5 APM OIDC with Azure Entra AD Configuring an OAuth setup using one BIG-IP APM system as an OAuth authorization server and another as the OAuth client
