Pet battle XP can seem a little weird
However, I can say for sure that the amount of XP you get is determined by your pet's level vs the opponent's level and how many pets you used in a battle, in that order.
This means that the amount of XP you get is fixed when you consider your level vs the opponent's level. From that fixed amount, it's divided evenly depending on how many pets you used to fight. If you only use 1, then that 1 gets all the XP. If you use 2, each get half. If you use 3, each gets a third.
The part where it seems weird is that if you have 2 different pets fight and they're very different level, then they each get half the amount of that level vs the opponent's level.
So, for example (made up numbers, but it's easy enough to verify it in game), if you have a level 23 and a level 5 that both participate, then they each get half the XP, but the level 23 gets half of the 256 she would have gotten if she had finished the fight solo, and the level 5 gets half of the 750XP (plus difficulty bonus!) she would have gotten if she had fought solo. So you'll see one pet get 128, and the other pet get 454.
Also, I have noticed when power leveling pets that there does seem to be a cap. For instance, if you take a level 3 and participate in a fight vs level 24's, then go and fight again at level 5 (because you'll get 2 levels from the first fight), the amount you get at level 5 is actually higher than the amount you got as a level 3. It seems to reach a peak, and then start to drop off normally.