Your assumption is almost true, the 100ms of delay as you meassured is the ICMP Round Trip Time (RTT) for establishing a Clientside Connection (from your Ping Machine to BigIP Clientside Listener). As you may have heard, F5 is a Full Proxy device, therefore it's not all of the delay. On top of that, you will have to add the delay which comes from establishing the Serverside Connection (from BigIP to Pool Member Listener).
In regards to prioritizing the production traffic over irrelevant pings and appliance management, that also holds true with BigIP. For instance, if you consume the appliance resource by extensive use of the Graphical User Interface, your management sessions will be disconnected before the impact hits real production traffic. With that said, in any LAN environment, network-level delay in excess of 100ms should never occur even for the non-production Ping tests. Is the Internet in between your Ping Machine and F5 clientside listener? Other than that, your appliance really has to be under a heavy load for that to happen.
To monitor the true delay of your TCP or UDP application, I recommend to have a look around the DevCentral and try some of the logging iRules, either local appliance logging or remote syslog - whatever suits you better. There are a plenty of examples for both.