Just adding my experience on catching these:
Blue - was in Townlong finishing up zone quests on an alt and saw the swarm spawned in the small lake by the Temple. Tried my luck and got the carp after around 15 mins.
White - fished the Octopus swarm off the coast of Jade Forest, caught after 10 mins.
Green - caught after roughly 5 hours in Valley: 3 hours flying up and down the Yah-Zhe river stopping at every pool, then another 2 hours from open water standing on Fish Fellreed's dock.
Red - 1 hour in Dread Wastes, 2 hours fishing the swarm in Kun Lai (finally got the sea turtle after giving up on that years ago), and another 3 hours yesterday at the Temple Lake in Townlong. After 6 hours still no luck, but today I plan on camping Townlong until this last one is in my bags!
EDIT: updating so I have one spot tracking my efforts. Another 6 hours at the Temple lake in Townlong... no red fishie yet. Brings total to 12 hours on red alone.
EDIT2: just got the red after 1 hour at Temple lake fishing open water during swarms (too lazy to move to pools), so total 13 hours on red.
Final tally: roughly 18.5 hours to get all 4 carp. Zowie!