Very importaint question.
Are you swaping memory cards while the Vita is still powered on?
I read the Userguide and it says that you have to turn the system off before swapping memory cards otherwise you can get a corrupted card.
Playstation Vita Online Manual http://manuals.playstation.net/document/en/psvita/
Insert or remove the memory card for the PS Vita system only when the power is off. Inserting or removing the memory card while the power is on, including when your system is in standby mode, can cause data corruption or loss.