BIG-IQ Central Management API automation and programmability - Workflow - Python
Summary In conjunction with the announcement of BIG-IQ 5.0 we are excited to bring the field greater flexibility when centrally managing BIG-IP devices utilizing well defined workflows and the BIG-IQ iControl REST API. We understand that automation and programmability is becoming more the norm these days as the network is evolving into a software defined application aware environment. This article will provide a basic example for first establishing trust while adding a device to inventory, second importing device configuration into BIG-IQ's configuration database and finally adding device to a license pool all utilizing the BIG-IQ rest api uri provided by the interface. Response is parsed in JSON format so there are dependencies required. This is intended to be an introduction to leveraging the API directly. We are in the process of developing a python object abstraction for the BIG-IQ CM REST API to provide even greater simplicity when leveraging automation for central management programmability. But we thought this level of detail may be appropriate to get the ball rolling. The BIG-IQ CM SDK will be structured similar to version 0.1.7 BIG-IP iControl REST API described here: BIGIP SDK - https://devcentral.f5.com/s/articles/f5-friday-python-sdk-for-big-ip-18233 BIGIP SDK Docs - https://f5-sdk.readthedocs.io/en/v0.1.7/ Let's get started ... READY ... SET ... GO ... Here are the API URIs' used as reference for both Python and Perl examples below: Trust:https://'bigiq-ip'/mgmt/cm/global/tasks/device-trust Discovery:https://'bigiq-ip'/mgmt/cm/global/tasks/device-discovery Import: ADC - https://'bigiq-ip'/mgmt/cm/adc-core/tasks/declare-mgmt-authority Import: Firewall - https://'bigiq-ip'/mgmt/cm/firewall/tasks/declare-mgmt-authority License: Pools - https://'bigiq-ip'/mgmt/cm/shared/licensing/pools License: Members: - https://'bigiq-ip'/mgmt/cm/shared/licensing/pools/'member_uuid'/members Using a python class abstraction (libraries can be located in /lib/) for Discovery, Import and Licensing, these tasks are a single call from one place which will provide the flexibility to create scripts defined by workflow. For example, lets - 1. Negotiate trust certificates for a newly deployed BIGIP. 2. Discover the BIGIP device and add to inventory. 3. Import ADC configuration and resolve all differences to BIGIQ’s management database. 4. Import Firewall configuration and resolve all differences to BIGIQ’s management database. 5. Add this newly discovered BIGIP to a provisioned license pool using the base-registration-key as a filter. Using the following python libraries, we can accomplish these tasks defined as methods in each class definition: ## Discovery ../lib/discovery.py: class Discover(object) def device_trust(dev_id): def ltm_discover(modules adc_core, firewall, asm, apm etc..): ## Import ../lib/import.py: class Import(object) def import_ltm(dev_id): def import_security(dev_id, afm=True): ## License ../lib/license.py : class License(object) def license_pool(config): Now it’s just a matter of putting the puzzle together in a script that will accomplish the workflow described above. BIG-IQ and BIG-IP device address and credentials are read in from a configuration file (../config/..) the script first and passed into def workflow for processing. Each method is called with argument <config> dictionary to perform the task required. After completion each will return True or False depending on result from rest response code and some evaluation. ## Import libraries to leverage methods described above from discovery import Discover from import_module import Import from license import License ## main def "workflow" def workflow (Discover, License, Import, config): ## Trust "establish certificate exchange" result_trust = Discover.device_trust(config) ## Discover "add a BIGIP to inventory" result = Discover.ltm_discover(config, result_trust) ## Import ADC "import adc configuration into database" result = Import.import_ltm(config, result_trust) ## Import AFM, ASM, APM "import module configuration into database" result = Import.import_sec(config, result_trust, afm=True) ## Add BIGIP member to license pool result = License.license_pool(config) ## Python main body if __name__ == '__main__': # read config file from ../config/.. config = {} with open (file) as infile: config[str(key)] = val.strip('\n') ## create an instance of each imported class defined in the libraries Discover = Discover(config) Import = Import(config) License = License(config) ## call the main program method "workflow" workflow (Discover, Import, License, config) If you are interested in this code for collaboration or solution, search on key words "bigiq" "api" "python" in code share section on dev central or you can refer to the reference link: Device trust, discovery and import using python requests - supported in Python version 2.7.9 and greater. https://devcentral.f5.com/s/articles/big-iq-trust-discovery-and-import-of-big-ip-using-rest-api-python-oo-973 Grants a license to a BIG-IP from an already-activated purchaced pool. https://devcentral.f5.com/s/articles/big-iq-licensing-of-big-ip-using-rest-api-python-oo-974 We will also be adding to github and will update this article once completed. So please come back and visit us soon and often for additional content.BIG-IQ Central Management API automation and programmability - BULK Discovery, Import and Re Import - Perl
Summary In conjunction with the announcement of BIG-IQ 5.0 we are excited to bring the field greater flexibility when centrally managing BIG-IP devices utilizing well defined workflows and the BIG-IQ iControl REST API. We understand that automation and programmability is becoming more the norm these days as the network is evolving into a software defined application aware environment. This article is an addendum to “BIG-IQ Central Management API automation and programmability – Python” and will demonstrate bulk device trust, discovery to populate resolver groups and import bigip configuration of many BIG-IP device as defined in a csv configuration file. This automation can be run as a standalone utilities that will run directly in the BIG-IQ shell for adding devices to inventory in a sequential yet automated fashion. API Reference Trust: https://'bigiq-ip'/mgmt/cm/global/tasks/device-trust Discovery: https://'bigiq-ip'/mgmt/cm/global/tasks/device-discovey Import: ADC - https://'bigiq-ip'/mgmt/cm/adc-core/tasks/declare-mgmt-authority Import: Firewall - https://'bigiq-ip'/mgmt/cm/firewall/tasks/declare-mgmt-authority Let’s get started – When using the BIG-IQ it is suggested to make a directory called scripts under /shared and securely copy this distribution into /shared/scripts/. Contents: ../config/bulk_discovery.csv ../src/perl/bulkDiscovery.pl ../src/perl/bulkReImport.pl Everything is predicated by a main loop which will invoke each task by calling supporting perl subroutines self-contained in the script. All rest calls, using curl (https://curl.haxx.se/), made are highlighted below. Establishment of device trust is completed in the main loop while the process of discovery and import configurations are completed in subroutine blocks within the script. #====================================================== # Main loop # Process trust, discovery, and imports #====================================================== for $bigip (@bigips) { my %postBodyHash = ("address"=>$bigiq, "userName"=>$user, "password"=>$pw,"clusterName"=>"", "useBigiqSync"=>"false", "name"=>"trust_$mip"); my $postBody = encode_json(\%postBodyHash); my $trustCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X POST -d \'$postBody\' https://localhost/mgmt/cm/global/tasks/device-trust"; if (discoverModules($bigiq, $machineId)) { if (importModules($bigiq)) { } } # upgrade the framework if nessasary if (handleFrameworkUpdade ($trustTask, $bigip)) { } } end of all devices #====================================================== # Discover specified modules. #====================================================== sub discoverModules { my %postBodyHash = ("moduleList" => \@moduleList, "status" => "STARTED"); # POST a new discovery task $postBodyHash{"deviceReference"}{"link"} = "cm/system/machineid-resolver/$machineId"; my $newDiscoverTaskCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X POST -d \'$postBodyJson\' \"https://localhost/mgmt/cm/global/tasks/device-discovery\""; } end of discoverModules #====================================================== # A subroutine for importing individual module. #====================================================== sub importModule { # POST a new import task $postBodyHash{"deviceReference"}{"link"} = "cm/system/machineid-resolver/$machineId"; my $postBody = encode_json(\%postBodyHash); my $newImportTaskCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X POST -d \'$postBodyJson\' \"$dmaUrl\""; # if we ecounter conflicts, we mark them to use BigIQ, patch the task back to started, and poll again if (($currentStep eq "PENDING_CONFLICTS") or ($currentStep eq "PENDING_CHILD_CONFLICTS")) if (resolveConflicts($mip, $module, $currentStep, $importSelfLink, @conflicts)) } # end of importModule #========================================================== # sub routine calling importModule to colocate all modules #========================================================== sub importModules { $ltmSuccess = importModule($mip, "ltm", "https://localhost/mgmt/cm/adc-core/tasks/declare-mgmt-authority", %postBodyHash); $asmSuccess = importModule($mip, "asm", "https://localhost/mgmt/cm/asm/tasks/declare-mgmt-authority", %postBodyHash); } And last but not least Re Import of BIGIP configuration objects for greater than one BIGIP device. This script can be run periodically based on Linux cron to ensure your device configurations managed by BIGIQ are up to date. On occasion other Element Management Systems could modify BIGIP object base and BIGIQ should be aware of these changes. If you refer to the below main loop, the discovery and import call's are the same. So two things actually happen that differs from inital bulk discovery and import. 1. Trust establishment is removed as it already contains mutaul certificate trust. 2. We test if the discovery and import tasks exists, if they do we can just PATCH discovery and import tasks to enforce a re import. That's about it. Refer to the code snippet below. #====================================================== # Main loop # Process Re Discovery, and Imports #====================================================== for $bigip (@bigips) { ## get the device properties my $deviceCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X GET https://localhost/mgmt/shared/resolver/device-groups/cm-bigip-allBigIpDevices/devices"; ## call disc routine using ip and machine id. if (discoverModules($bigiq, $machineId)) { ## call import routine using up and machine id. if (importModules($bigiq, $machineId)) } } } # end for devices Just to re iterate the above the discovery and import routines used for Re Import just PATCH the existing task created during inital discovery and import. Here are the PATCH requests. #====================================================== # Discover specified modules. #====================================================== sub discoverModules { ## get the discovery task based on the machineId my $findDiscoverTaskCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X GET \"https://localhost/mgmt/cm/global/tasks/device-discovery?\\\$filter=deviceReference/link+eq+\'*$machineId*\'+and+status+eq+\'FINISHED\'\""; ## If it exists PATCH the task if (defined $discoveryTask->{"items"}[0]) { # PATCH the existing discovery task my $discoveryTaskSelfLink = $discoveryTask->{"items"}[0]->{"selfLink"}; $postBodyJson = encode_json(\%postBodyHash); my $discoverCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X PATCH -d \'$postBodyJson\' $discoveryTaskSelfLink"; } #====================================================== # A subroutine for importing individual module. #====================================================== sub importModule { ## get import task based on the machineid my $findImportTaskCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X GET \"$dmaUrl?\\\$filter=deviceReference/link+eq+\'*$machineId*\'\""; ## If exists PATCH the task if (defined $findImportTask->{"items"}[0]) { # PATCH the existing import task $importTaskLink = $findImportTask->{"items"}[0]->{"selfLink"}; my $importCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X PATCH -d \'$postBodyJson\' $importTaskLink"; } #======================================== # sub routine for calling importModule to collocate all modules. #======================================== sub importModules { $ltmSuccess = importModule($mip, $machineId, "ltm", "https://localhost/mgmt/cm/adc-core/tasks/declare-mgmt-authority", %postBodyHash); $asmSuccess = importModule($mip, machineId, "asm", "https://localhost/mgmt/cm/asm/tasks/declare-mgmt-authority", %postBodyHash); } If you are interested in this code for collaboration or solution, seach on key words "bigiq" "api" "python" or "perl" in code share section on dev central or here is the reference link: https://devcentral.f5.com/s/articles/big-iq-big-ip-rest-api-bulk-device-discovery-perl-972 We will also create a repository on github for easy accessability. Please visit us soon and often for periodic updates.
WordPress REST API Vulnerability: Violating Security’s Rule Zero
It's an API economy. If you don't have an API you're already behind. APIs are the fuel driving organizations' digital transformation. We've all heard something similar to these phrases in the past few years. And while they look like marketing, they taste like truth. Because APIs really are transforming organizations and have taken hold as the de facto method of integration - internally, externally, laterally, and vertically. APIs enable mobile apps and things, web apps and middleware to communicate, collaborate, and extricate digital gold (that's data, by the way). So when there’s a vulnerability discovered in an API, it turns heads. Especially if it’s a vulnerability that is easily exploitable (this one is) and affects a significant number of publicly accessible resources (it does). #WordPress REST API vulnerability is already abused in defacement campaigns https://t.co/5i9pYKW4tH by @danielcid If you’re running WordPress versions 4.7.0 or 4.7.1 you are vulnerable. Stop reading this and go patch it right now. If you can’t for some reason and you’ve got a BIG-IP ASM you can apply these rules right now to protect vulnerable sites. I’m not kidding. I’ll wait right here. Okay, now that we’ve got that out of the way, I want to talk about APIs and security for a moment because there seems to be some general misunderstandings about REST APIs and security that this vulnerability just happens to illustrate perfectly. HTTP: The Foundation on which REST APIs are Built REST stands for Representational State Transfer. Don’t worry too much about what that means, because what you really need to understand is that it’s an architectural style. If you were going to list it along with other methods of achieving the same thing (transfer of data between two endpoints) you might list: RPC, CORBA, and SOAP. A REST API call is an HTTP request where the URI endpoint is typically indistinguishable from a web URI. The request looks the same. Really, which of these two URIs is a call to an API: GET /w/thing/snowmobile/id GET /a/thing/snowmobile/id See what I mean? No difference from outside. There’s no standard out there that requires a REST API contain some identifying attribute or HTTP header that makes it different than a traditional web request. The Content-Type suggested by the JSON specification is application/json but like the Pirate Code, those are more guidelines than actual rules. Thus, one cannot rely solely on Content-Type to identify a REST API from a standard HTTP request. In fact, good old x-www-form-urlencoded is often used to invoke API calls from clients. For example, here’s an HTTP request to an Express-based API I’m working on (for fun, cause I still do that) using Postman: GET /api/user/1 HTTP/1.1 Host: 192.168.0.57:8080 Content-Type: application/x-www-form-urlencoded Cache-Control: no-cache Postman-Token: 6d2247a1-9923-4774-6b86-7ba334bd497e And here’s one that does a POST, to send data to the API endpoint: POST /api/login HTTP/1.1 Host: 192.168.0.57:8080 Content-Type: application/x-www-form-urlencoded Cache-Control: no-cache Postman-Token: 32a29369-5d9d-6952-7de6-8aa4782d694d name=Webmistress&passwd=xxxx Now, these are API calls. And I’m betting that you noticed a couple of things: The REST API uses HTTP verbs like GET, POST, PUT, and DELETE. The URI is – and hold on to your hats now – a standard HTTP URI. The Content-Type does not help us determine whether this is a request to an API or a traditional web app. These first two points are really important because it’s this basic foundation that lets us breathe (a little) easier when it comes to securing APIs. Because we already know a whole lot about HTTP and the myriad ways in which it can be exploited. REST has become the de facto standard for communication because it’s far simpler than its predecessors and it relies solely on well-understood protocols and platforms, primarily HTTP. Building a REST API, then, means using HTTP to define a set of interfaces through which mobile apps, things, and web apps will communicate. These interfaces, in aggregate, comprise an API. And basically, they’re a set of URI endpoints invoked via HTTP and executed on by a server-side application*. The fact that the architectural design is REST, and it’s an API, is irrelevant. The same code could have been written to support a traditional web-based system, because the problem isn’t with the API or REST, it’s with the code that’s processing the input provided. The WordPress vulnerability is not a vulnerability in its API, per se, but in the way the API implementation handles standard, well-understood HTTP mechanisms. API implementations are often accomplished via frameworks (like Express, can you tell I’m a fan?) that take the tedium out of splitting apart the URI and distilling the paths into “routes” that are then used to call the appropriate function. In addition, this code is responsible for grabbing the query parameters (everything that comes after the ? in a URI, including the key-value pairs) and stuffing them into variables that can be used to do things like update databases, retrieve forum posts, and insert new content into the system. In this case, the back-end code for the API improperly handles those variables (including some poorly considered authorization logic), failing to properly validate and sanitize it. In Dungeons and Dragons we have the concept of “Rule Zero” and it comes before every other rule out there. It is foundational. There is a similar Rule Zero in security, and it is this: THOU SHALT NOT TRUST USER INPUT. EVER. It is Rule Zero that has been violated and thus introduces this vulnerability to WordPress. Now, you can certainly dig into the details and walk through the code (I like the type-casting logic that really makes this vulnerability work and the lazy sanitization attempt) but the point here is that REST API security relies a lot on the same, well-understood WEB security long preached by OWASP and every other security vendor out there. In fact, one could say it starts with strong, basic web security, the foundation of which is built upon rule zero. Yes, there are other concerns with APIs with respect to security, and REST introduces some of them because it eschews state. That means it requires a more active or external means of authentication and authorization. And many security folks are still getting up to speed on JSON, as are many of the security services inserted into the data path between endpoints that inspect, scan, and scrub app layer data. But if you’re trying to figure out where to start securing APIs, one of the best places is back to the beginning by laying down a strong set of secure coding practices that begins with Security’s Rule Zero. * Increasingly server-side applications are microservices and in the case of APIs, serverless (Function as a Service) is also gaining traction. And still they rely on endpoint invocation via an HTTP URI. Just sayin. The more things change, the more the stay the same.Generation of Utility Billing Report using BIG-IQ's API
This script generate the utility billing report based of a list of regKey.This script only applies to utility-type licenses used for F5's subscription and/or ELA programs. The regkey listed in the csv file or in the command line are the registration key associated with the license pool you wish to report on. Reports can be automatically submitted to F5 or manually created. /!\ This feature will be included with BIG-IQ 6.1 /!\ Automatic report submission requires BIG-IQ to access api.f5.com. BIG-IQ makes a REST call over SSL to api.f5.com to upload the report. Manual report submission is used in cases where BIG-IQ cannot reach api.f5.com. In this workflow, the customer generates the report, extracts it, then emails it to F5 (/shared/scripts/F5_Billing_License_Report.<date>.json). Pre-Requisist: Make sure you can access api.f5.com port 443 if automatic reporting is chosen (nslookup api.f5.com, ping api.f5.com, telnet api.f5.com 443). The script has to be executed locally from the BIG-IQ. Installation: The script mustbe installed in BIG-IQ under /shared/scripts: # mkdir /shared/scripts # chmod +x /shared/scripts/licenseUtilityReport.pl Allowed command line options: -h Help -c Path to CSV file with all regKey - REQUIRED if not using -k -k regKey(s) separated by , - REQUIRED if not using -c -r Report option automatic or manual - OPTIONAL (default is automatic) Command example automatic report: # cd /shared/scripts # ./licenseUtilityReport.pl -k DRLPZ-JISKU-VPUPT-HZMMV-LERVPYQ,GYCWI-FOUEZ-YMWPX-LYROB-PXTKMTG CSV file example manual report: # ./licenseUtilityReport.pl -c listregkey.csv-r manual # cat listregkey.csv DRLPZ-JISKU-VPUPT-HZMMV-LERVPYQ GYCWI-FOUEZ-YMWPX-LYROB-PXTKMTG Cron table example (every 1st of the month at 10am) for both above examples: # crontab -e (edit: type 'o' to add a new line, then 'ESC' and ':x' to save and quit) # crontab -l (list) 0 10 1 * * /usr/bin/perl /shared/scripts/licenseUtilityReport.pl -k DRLPZ-JISKU-VPUPT-HZMMV-LERVPYQ 0 10 1 * * /usr/bin/perl /shared/scripts/licenseUtilityReport.pl -c /shared/script/listregkey.csv-r manual ┌───────────── minute (0 - 59) │ ┌───────────── hour (0 - 23) │ │ ┌───────────── day of month (1 - 31) │ │ │ ┌───────────── month (1 - 12) │ │ │ │ ┌───────────── day of week (0 - 6) (Sunday to Saturday; │ │ │ │ │ 7 is also Sunday on some systems) │ │ │ │ │ │ │ │ │ │ ** * * * In case you need toobfuscate the Utility Billing Report before sending it, use manual export, then use the following script to mask IP/Macaddresses and Hostnames from the report. Then submit/email report to F5 (Please contact F5 Support for assistance). # ./f5_sanitize_usage_report.sh F5_Billing_License_Report.<date>.json Location of the scriptson GitHub:https://github.com/f5devcentral/f5-big-iq-pm-teamApplication Programming Interface (API) Authentication types simplified
API is a critical part of most of our modern applications. In this article we will walkthrough the different authentication types, to help us in the future articles covering NGINX API Connectivity Manager authentication and NGINX Single Sign-on.NGINX Management Suite API Connectivity Manager - Modern API driven Applications
Introduction API based applications benefits NGINX Management Suite API Connectivity Manager capabilities API Connectivity Manager use case API Connectivity Manager use case overview API Connectivity Manager traffic flows API Connectivity Manager lab & implementation References Introduction API based applications benefits Before we dive into our API gateway use case, we will go one step back and check why the move to API driven applications, below are some of the benefits for this move: Loose coupling: API-based applications can be built and maintained independently, allowing for faster development and deployment cycles. Reusability: APIs can be reused across multiple applications, reducing the need to duplicate code and effort. Scalability: API-based architecture allows for easier scaling of individual services, rather than having to scale the entire application. Flexibility: APIs allow for different client applications to consume the same services, such as web, mobile, and IoT devices. Interoperability: APIs facilitate communication between different systems and platforms, enabling integration with third-party services and data sources. Microservices: API-based architecture allows developers to build small, modular services that can be developed, deployed, and scaled independently. NGINX Management Suite API Connectivity Manager capabilities NGINX Management Suite API Connectivity Manager adds to the capabilities of the API driven applications a secure approach to authenticate, access and developing those API based applications. API Connectivity Manager is used to connect, secure, and govern our APIs. In addition, API Connectivity Manager lets us separate infrastructure lifecycle management from the API lifecycle, giving the IT/Ops teams and application developers the ability to work independently. API Connectivity Manager provides the following features: Create and manage isolated Workspaces for business units, development teams, and so on, so each team can develop and deploy at its own pace without affecting other teams. Create and manage API infrastructure in isolated workspaces. Enforce uniform security policies across all workspaces by applying global policies. Create Developer Portals that align with your brand, with custom color themes, logos, and favicons. Onboard your APIs to an API Gateway cluster and publish your API documentation to the Dev Portal. Let teams apply policies to their API proxies to provide custom quality of service for individual applications. Onboard API documentation by uploading an OpenAPI spec. Publish your API docs to a Dev Portal while keeping your API’s backend service private. Let users issue API keys or basic authentication credentials for access to your API. Send API calls by using the Developer Portal’s API Reference documentation. API Connectivity Manager use case API Connectivity Manager use case overview In our case we will have three teams, Infrastructure team, this one will be responsible for setting up the infrastructure, domains and access policies. API team, this one will be responsible for setting up the API documentation, QoS and gateway for both production and developer portals. Application team, this one will be responsible for learning the APIs through the developer portal and use the APIs through the production portal. Authentication in our case is done via two methods, API Key authentication for API version 1. OAuth2 introspection for API version 2. Note, More Authentication methods can be used (JSON Web Token Assertion) included in the following tutorial. API authentication more detailed discussion can be found here Application Programming Interface (API) Authentication types simplified Additional features like API rate limiting can be applied as well, here's a toturial to enable that feature. API Connectivity Manager traffic flows In our use case will have three flows, Management flow, illustrated below. Metrics and events collection flow, illustrated below Data flow illustrated below NGINX tutorial on how to streamline API operations with API Connectivity Manager, API Connectivity Manager lab & implementation ِThe steps we are going to follow with some useful tutorial videos are highlighted below, Setup backend API application (This step has been already done for you in the lab). Setup API Connectivity Manager infrastructure and policies. Enable API Key Authentication via the following Youtube toturial Enable API Key Authentication with API Connectivity Manager. Publish APIs and Documentation through API Connectivity Manager. Test APIs through API Developer Portal The detailed lab guide and the implementation videos Cloud labs detailed guide https://clouddocs.f5.com/training/community/nginx/html/class10/class10.html UDF lab can be found here as well https://udf.f5.com/b/ed5ffb71-bcce-47ec-9d9f-307441e4c12c#documentation Below a recorded Lab walkthrough by our awesome guru Matt_Dierick References API Connectivity Manager NGINX Management Suite NGINX Docs API Connectivity Manager UDF Lab
Advanced WAF v16.0 - Declarative API
Since v15.1 (in draft), F5® BIG-IP® Advanced WAF™ canimport Declarative WAF policy in JSON format. The F5® BIG-IP® Advanced Web Application Firewall (Advanced WAF) security policies can be deployed using the declarative JSON format, facilitating easy integration into a CI/CD pipeline. The declarative policies are extracted from a source control system, for example Git, and imported into the BIG-IP. Using the provided declarative policy templates, you can modify the necessary parameters, save the JSON file, and import the updated security policy into your BIG-IP devices. The declarative policy copies the content of the template and adds the adjustments and modifications on to it. The templates therefore allow you to concentrate only on the specific settings that need to be adapted for the specific application that the policy protects. ThisDeclarative WAF JSON policyis similar toNGINX App Protect policy. You can find more information on theDeclarative Policyhere : NAP :https://docs.nginx.com/nginx-app-protect/policy/ Adv. WAF :https://techdocs.f5.com/en-us/bigip-15-1-0/big-ip-declarative-security-policy.html Audience This guide is written for IT professionals who need to automate their WAF policy and are familiar with Advanced WAF configuration. These IT professionals can fill a variety of roles: SecOps deploying and maintaining WAF policy in Advanced WAF DevOps deploying applications in modern environment and willing to integrate Advanced WAF in their CI/CD pipeline F5 partners who sell technology or create implementation documentation This article covershow to PUSH/PULL a declarative WAF policy in Advanced WAF: With Postman With AS3 Table of contents Upload Policy in BIG-IP Check the import Apply the policy OpenAPI Spec File import AS3 declaration CI/CD integration Find the Policy-ID Update an existing policy Video demonstration First of all, you need aJSON WAF policy, as below : { "policy": { "name": "policy-api-arcadia", "description": "Arcadia API", "template": { "name": "POLICY_TEMPLATE_API_SECURITY" }, "enforcementMode": "blocking", "server-technologies": [ { "serverTechnologyName": "MySQL" }, { "serverTechnologyName": "Unix/Linux" }, { "serverTechnologyName": "MongoDB" } ], "signature-settings": { "signatureStaging": false }, "policy-builder": { "learnOnlyFromNonBotTraffic": false } } } 1. Upload Policy in BIG-IP There are 2 options to upload a JSON file into the BIG-IP: 1.1 Either youPUSHthe file into the BIG-IP and you IMPORT IT OR 1.2 the BIG-IPPULLthe file froma repository (and the IMPORT is included)<- BEST option 1.1PUSH JSON file into the BIG-IP The call is below. As you can notice, it requires a 'Content-Range' header. And the value is 0-(filesize-1)/filesize. In the example below, the file size is 662 bytes. This is not easy to integrate in a CICD pipeline, so we created the PULL method instead of the PUSH (in v16.0) curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/file-transfer/uploads/policy-api.json' \ --header 'Content-Range: 0-661/662' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --header 'Content-Type: application/json' \ --data-binary '@/C:/Users/user/Desktop/policy-api.json' At this stage,the policy is still a filein the BIG-IP file system. We need toimportit into Adv. WAF. To do so, the next call is required. This call import the file "policy-api.json" uploaded previously. AnCREATEthe policy /Common/policy-api-arcadia curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Type: application/javascript' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{ "filename":"policy-api.json", "policy": { "fullPath":"/Common/policy-api-arcadia" } }' 1.2PULL JSON file from a repository Here, theJSON file is hosted somewhere(in Gitlab or Github ...). And theBIG-IP will pull it. The call is below. As you can notice, the call refers to the remote repo and the body is a JSON payload. Just change the link value with your JSON policy URL. With one call, the policy isPULLEDandIMPORTED. curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{ "fileReference": { "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json" } }' Asecond versionof this call exists, and refer to the fullPath of the policy.This will allow you to update the policy, from a second version of the JSON file, easily.One call for the creation and the update. As you can notice below, we add the"policy":"fullPath" directive. The value of the "fullPath" is thepartitionand thename of the policyset in the JSON policy file. This method is VERY USEFUL for CI/CD integrations. curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{ "fileReference": { "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json" }, "policy": { "fullPath":"/Common/policy-api-arcadia" } }' 2. Check the IMPORT Check if the IMPORT worked. To do so, run the next call. curl --location --request GET 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ You should see a 200 OK, with the content below (truncated in this example). Please notice the"status":"COMPLETED". { "kind": "tm:asm:tasks:import-policy:import-policy-taskcollectionstate", "selfLink": "https://localhost/mgmt/tm/asm/tasks/import-policy?ver=16.0.0", "totalItems": 11, "items": [ { "isBase64": false, "executionStartTime": "2020-07-21T15:50:22Z", "status": "COMPLETED", "lastUpdateMicros": 1.595346627e+15, "getPolicyAttributesOnly": false, ... From now, your policy is imported and created in the BIG-IP. You can assign it to a VS as usual (Imperative Call or AS3 Call).But in the next session, I will show you how to create a Service with AS3 including the WAF policy. 3. APPLY the policy As you may know, a WAF policy needs to be applied after each change. This is the call. curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/apply-policy/' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{"policy":{"fullPath":"/Common/policy-api-arcadia"}}' 4. OpenAPI spec file IMPORT As you know,Adv. WAF supports OpenAPI spec (2.0 and 3.0). Now, with the declarative WAF, we can import the OAS file as well. The BEST solution, is toPULL the OAS filefrom a repo. And in most of the customer' projects, it will be the case. In the example below, the OAS file is hosted in SwaggerHub(Github for Swagger files). But the file could reside in a private Gitlab repo for instance. The URL of the projectis :https://app.swaggerhub.com/apis/F5EMEASSA/Arcadia-OAS3/1.0.0-oas3 The URL of the OAS file is :https://api.swaggerhub.com/apis/F5EMEASSA/Arcadia-OAS3/1.0.0-oas3 This swagger file (OpenAPI 3.0 Spec file) includes all the application URL and parameters. What's more, it includes the documentation (for NGINX APIm Dev Portal). Now, it ispretty easy to create a WAF JSON Policy with API Security template, referring to the OAS file. Below, you can notice thenew section "open-api-files"with the link reference to SwaggerHub. And thenew templatePOLICY_TEMPLATE_API_SECURITY. Now, when I upload / import and apply the policy, Adv. WAF will download the OAS file from SwaggerHub and create the policy based on API_Security template. { "policy": { "name": "policy-api-arcadia", "description": "Arcadia API", "template": { "name": "POLICY_TEMPLATE_API_SECURITY" }, "enforcementMode": "blocking", "server-technologies": [ { "serverTechnologyName": "MySQL" }, { "serverTechnologyName": "Unix/Linux" }, { "serverTechnologyName": "MongoDB" } ], "signature-settings": { "signatureStaging": false }, "policy-builder": { "learnOnlyFromNonBotTraffic": false }, "open-api-files": [ { "link": "https://api.swaggerhub.com/apis/F5EMEASSA/Arcadia-OAS3/1.0.0-oas3" } ] } } 5. AS3 declaration Now, it is time to learn how we cando all of these steps in one call with AS3(3.18 minimum). The documentation is here :https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/declarations/application-security.html?highlight=waf_policy#virtual-service-referencing-an-external-security-policy With thisAS3 declaration, we: Import the WAF policy from a external repo Import the Swagger file (if the WAF policy refers to an OAS file) from an external repo Create the service { "class": "AS3", "action": "deploy", "persist": true, "declaration": { "class": "ADC", "schemaVersion": "3.2.0", "id": "Prod_API_AS3", "API-Prod": { "class": "Tenant", "defaultRouteDomain": 0, "API": { "class": "Application", "template": "generic", "VS_API": { "class": "Service_HTTPS", "remark": "Accepts HTTPS/TLS connections on port 443", "virtualAddresses": ["10.1.10.27"], "redirect80": false, "pool": "pool_NGINX_API_AS3", "policyWAF": { "use": "Arcadia_WAF_API_policy" }, "securityLogProfiles": [{ "bigip": "/Common/Log all requests" }], "profileTCP": { "egress": "wan", "ingress": { "use": "TCP_Profile" } }, "profileHTTP": { "use": "custom_http_profile" }, "serverTLS": { "bigip": "/Common/arcadia_client_ssl" } }, "Arcadia_WAF_API_policy": { "class": "WAF_Policy", "url": "http://10.1.20.4/root/as3-waf-api/-/raw/master/policy-api.json", "ignoreChanges": true }, "pool_NGINX_API_AS3": { "class": "Pool", "monitors": ["http"], "members": [{ "servicePort": 8080, "serverAddresses": ["10.1.20.9"] }] }, "custom_http_profile": { "class": "HTTP_Profile", "xForwardedFor": true }, "TCP_Profile": { "class": "TCP_Profile", "idleTimeout": 60 } } } } } 6. CI/CID integration As you can notice, it is very easy to create a service with a WAF policy pulled from an external repo. So, it is easy to integrate these calls (or the AS3 call) into a CI/CD pipeline. Below, an Ansible playbook example. This playbook run the AS3 call above. That's it :) --- - hosts: bigip connection: local gather_facts: false vars: my_admin: "admin" my_password: "admin" bigip: "10.1.1.12" tasks: - name: Deploy AS3 WebApp uri: url: "https://{{ bigip }}/mgmt/shared/appsvcs/declare" method: POST headers: "Content-Type": "application/json" "Authorization": "Basic YWRtaW46YWRtaW4=" body: "{{ lookup('file','as3.json') }}" body_format: json validate_certs: no status_code: 200 7. FIND the Policy-ID When the policy is created, a Policy-ID is assigned. By default, this ID doesn't appearanywhere. Neither in the GUI, nor in the response after the creation. You have to calculate it or ask for it. This ID is required for several actions in a CI/CD pipeline. 7.1 Calculate the Policy-ID Wecreated this python script to calculate the Policy-ID. It is an hash from the Policy name (including the partition). For the previous created policy named"/Common/policy-api-arcadia",the policy ID is"Ar5wrwmFRroUYsMA6DuxlQ" Paste this python codein a newwaf-policy-id.pyfile, and run the commandpython waf-policy-id.py "/Common/policy-api-arcadia" Outcome will beThe Policy-ID for /Common/policy-api-arcadia is: Ar5wrwmFRroUYsMA6DuxlQ #!/usr/bin/python from hashlib import md5 import base64 import sys pname = sys.argv[1] print 'The Policy-ID for', sys.argv[1], 'is:', base64.b64encode(md5(pname.encode()).digest()).replace("=", "") 7.2 Retrieve the Policy-ID and fullPath with a REST API call Make this call below, and you will see in the response, all the policy creations. Find yours and collect thePolicyReference directive.The Policy-ID is in the link value "link": "https://localhost/mgmt/tm/asm/policies/Ar5wrwmFRroUYsMA6DuxlQ?ver=16.0.0" You can see as well, at the end of the definition, the "fileReference"referring to the JSON file pulled by the BIG-IP. And please notice the"fullPath", required if you want to update your policy curl --location --request GET 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Range: 0-601/601' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ { "isBase64": false, "executionStartTime": "2020-07-22T11:23:42Z", "status": "COMPLETED", "lastUpdateMicros": 1.595417027e+15, "getPolicyAttributesOnly": false, "kind": "tm:asm:tasks:import-policy:import-policy-taskstate", "selfLink": "https://localhost/mgmt/tm/asm/tasks/import-policy/B45J0ySjSJ9y9fsPZ2JNvA?ver=16.0.0", "filename": "", "policyReference": { "link": "https://localhost/mgmt/tm/asm/policies/Ar5wrwmFRroUYsMA6DuxlQ?ver=16.0.0", "fullPath": "/Common/policy-api-arcadia" }, "endTime": "2020-07-22T11:23:47Z", "startTime": "2020-07-22T11:23:42Z", "id": "B45J0ySjSJ9y9fsPZ2JNvA", "retainInheritanceSettings": false, "result": { "policyReference": { "link": "https://localhost/mgmt/tm/asm/policies/Ar5wrwmFRroUYsMA6DuxlQ?ver=16.0.0", "fullPath": "/Common/policy-api-arcadia" }, "message": "The operation was completed successfully. The security policy name is '/Common/policy-api-arcadia'. " }, "fileReference": { "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json" } }, 8 UPDATE an existing policy It is pretty easy to update the WAF policy from a new JSON file version. To do so, collect from the previous call7.2 Retrieve the Policy-ID and fullPath with a REST API callthe"Policy" and"fullPath"directive. This is the path of the Policy in the BIG-IP. Then run the call below, same as1.2 PULL JSON file from a repository,but add thePolicy and fullPath directives Don't forget to APPLY this new version of the policy3. APPLY the policy curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{ "fileReference": { "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json" }, "policy": { "fullPath":"/Common/policy-api-arcadia" } }' TIP : this call, above, can be used in place of the FIRST call when we created the policy "1.2PULL JSON file from a repository". But be careful, the fullPath is the name set in the JSON policy file. The 2 values need to match: "name": "policy-api-arcadia" in the JSON Policy file pulled by the BIG-IP "policy":"fullPath" in the POST call 9 Video demonstration In order to help you to understand how it looks with the BIG-IP, I created this video covering 4 topics explained in this article : The JSON WAF policy Pull the policy from a remote repository Update the WAF policy with a new version of the declarative JSON file Deploy a full service with AS3 and Declarative WAF policy At the end of this video, you will be able to adapt the REST Declarative API calls to your infrastructure, in order to deploy protected services with your CI/CD pipelines. Direct link to the video on DevCentral YouTube channel : https://youtu.be/EDvVwlwEFRw
