Hi,
I am not quite sure I get all your questions, but I'll give it a try. Do not hesitate to precise your request if not.
1 - HTTP::retry is to be used in HTTP Response and Response_Data events only. Hence, it can process a 500 error received by the backend, but probably not a RESET (L4)
2 - If you have LTM license, F5 supports many more LB Methods than Round Robin.
3 - As mentionned, if the server response is reset, you will not trigger HTTP_RESPONSE events, hence, I don't think you can use the HTTP::retry command. It could be donewhen receiving specific HTTP Error status from the backend
4 - This part is handled by TCP profiles (e.g. parameter Maximum Syn Retransmissions for SYN packet which is 3 by default, and maximum segment restransmission which is 8 by default)
5 - HTTP Retry resubmits a request. As in the examples given in clouddocs, if you want to interfere with the pool member selection you will need to code it in the iRules.
I hope this somehow helps you, otherwise, do not hesitate to complete, or detail precisely the problem you are trying to solve.
Have a nice day.
Yoann