Ah good. So this seems to be the problem I'm having too.
My goal is to be able to join the kids on my PS3 with my vita in a 1v1v1 but can't seem to join. I'm also looking for a way to basically have the Vita be a 3rd controller/screen while logged in on the same account.
Giving this a bump since it appears to still be a problem.
It is a known issue that you cannot use the Vita as a controller yet. So forget that for now.
The only way is to use a Vita and PS3 simultaneously, use a different PSN id and sign into it and invite from the PS3. That in THEORY, should work, I haven't yet tried it with my Vita and PS3.
Playstation God of War Stars Battle Bore y'all !!!