I'm no expert when it comes to theorcrafting or WoW mathematics, but I believe the 20% increase is multiplicative (is that the correct term?). 20% increase on the chance you had before, rather than a flat 20% more each time.
Anyway, rather than multiplying 20 x 5 attempts to get 100%, it's 20% of the previous product. So for example, let's say there's a 10% chance to trap a pet (I really don't know the exact value, but let's just pretend for the sake of this example
20% of 10% = 2%. Add that to the 10% and you get 12%. The trap fails, so you try again. 20% of 12% = 2.4%. Add that to the 12% and you get 14.4%. Continue that same pattern until you're either successful with the trap or reach 100%.
Also, it's possible that whenever a pet heals and its health goes above the necessary value required to trap a pet, the trap chance resets.
Someone please correct me if I'm wrong!