LTM iControlREST API request fails with HTTP 502 Bad Gateway
Hi,
We are using LTM v11.5.2 (on a 5250v's) and successfully using the REST API for all of our F5 Automation needs.
We have been using this REST call to gather info about every LTM Pool Member. GET https://{{F5-BIGIP-LTM}}/mgmt/tm/ltm/pool?expandSubcollections=true
This has been working fine until recently when the of pools configured on one LTM went from 2200 to 3335 (as we migrated one of Cisco ACE environments over to F5). After that, the above REST call fails with this HTTP response:
Bad Gateway!
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /mgmt/tm/ltm/pool.
Reason: Error reading from remote server
Error 502
myltm.example.com
Thu Apr 2 09:26:15 2015
After some experimentation, we were able to workaround the error by breaking the request down a smaller query size using this query param: $top=. We found that setting that to 3300 was successful. Larger than that produced the error above. So it appears that we have hit some REST capacity boundary. We think it is particular to REST because when we do the equivalent tmsh command (tmsh show ltm recursive pool members), we have no problem.
We have a few questions:
- Is the "Bad Gateway" error, in fact, indicative of a REST API capacity limit being hit?
- Is there any way for us to pre-determine where these REST API capacity boundaries are, short of hard experience like this?
- The "Bad Gateway" error that the LTM turns back is less than helpful. Should we expect the LTM REST server to recognize its limitations and give back a more reasonable error if a response capacity limit is reached?
FYI...A couple of other pieces of info about the problem.
When I do this curl command:
curl -kX GET -H "Content-Type: application/json" -H "Authorization: Basic –omitted--" -H "Cache-Control: no-cache" https://myltm.example.com/mgmt/tm/ltm/pool?expandSubcollections=trueI see that this message gets put to the restjavad-audit.0.log log within about 2-3 seconds:
[I][9479][03 Apr 2015 12:46:12 UTC][ForwarderPassThroughWorker][run] {"user":"automation","method":"GET","uri":"","status":200,"from":"x.y.z.a"}
The curl command continues to wait for results for about 65-70 seconds then ends with the Bad Gateway result.
Looking a bit more I found errors being logged here: /var/log/httpd/httpd_errors (EST times)
Apr 3 08:47:14 myltm-p err httpd[16202]: [error] [client 169.184.47.60] (20014)Internal error: proxy: error reading status line from remote server localhost Apr 3 08:47:14 myltm-p err httpd[16202]: [error] [client 169.184.47.60] proxy: Error reading from remote server returned by /mgmt/tm/ltm/pool
Here are some log messages from the same time (UTC times!) in /var/log/icrd
Apr 3 12:46:08 myltm-p notice icrd_child: 16837,16837, RestServer, INFO,Start: Begin start process. Apr 3 12:46:08 myltm-p notice icrd_child: 16837,16837, RestServer, INFO,Start: Begin listening on socket. Apr 3 12:46:08 myltm-p notice icrd_child: 16837,16837, RestServer, INFO,Start: Begin scheduled task thread. Apr 3 12:46:08 myltm-p notice icrd_child: 16837,16837, RestServer, INFO,Start: Begin socket thread. Apr 3 12:46:08 myltm-p notice icrd_child: 16837,16837, RestServer, INFO,Start: Begin process servicing threads. Apr 3 12:46:08 myltm-p notice icrd_child: 16837,16837, RestQueue, INFO,Start: Begin process servicing threads. Apr 3 12:46:08 myltm-p notice icrd_child: 16837,16837, RestServer, INFO,Start: start workers. Apr 3 12:46:08 myltm-p notice icrd_child: 16837,16837, RestServer, INFO,Start: start process done. Apr 3 12:46:08 myltm-p notice icrd_child: 16837,16837, RestRequestSender, INFO,RestRequestSender starting Apr 3 12:46:08 myltm-p notice icrd_child: 16837,16837, RestQueue, INFO,Start: Begin process servicing threads. Apr 3 12:46:46 myltm-p notice icrd_child[16837]: 16837,16837, RestRequestSender, INFO,RestRequestSender stopping Apr 3 12:46:46 myltm-p err icrd: 5557,13989, RestServer, SEVERE,Unknown socket for write event fd:10 Apr 3 12:46:46 myltm-p notice icrd_child[16837]: 16837,16837, RestServer, INFO,Shutdown: Begin shutdown process. Apr 3 12:46:46 myltm-p notice icrd_child[16837]: 16837,16837, RestServer, INFO,Shutdown: stop listening socket. Apr 3 12:46:46 myltm-p notice icrd_child[16837]: 16837,16837, RestServer, INFO,Shutdown: stop scheduled task thread. Apr 3 12:46:46 myltm-p notice icrd_child[16837]: 16837,16837, RestServer, INFO,Shutdown: stop process servicing threads. Apr 3 12:46:46 myltm-p notice icrd_child[16837]: 16837,16837, RestServer, INFO,Shutdown: shutdown workers. Apr 3 12:46:46 myltm-p notice icrd_child[16837]: 16837,16837, RestServer, INFO,Shutdown: Starting shutdown process done.
Thanks, Eric