On the PS3, make sure the AR Server option (Options > Game > AR Server) is ON.
To connect to your PS3, you need to have the game running, and be outside the Hideout for the Vita to connect. The Vita and PS3 both need to be connected to the same network.
Once you are outside the Hideout on the PS3, select the AR Treasures option from the main menu of Sly Cooper on the Vita.
After it loads up, point the camera at the TV screen and press the R button to take a snapshot of the screen. If successful, it will show an alternate color 'scan' of the current game area. Press the L button to save it and the Vita will now show the game world through the alternate view indicating where treasures are hidden.
At first I was having the same problem as everyone else. Just like others have said make sure remote play is set to ON on your PS3. Then go into Sly 4 on your PS3 AND Vita, go out of the safehouse and jump somewhere safe, then go on your Vita and try to connect, here's the important part though, make sure your Vita is face your screen, if that doesn't work face it towards your PS3. It worked for me after that, if that isn't working for you then I'm sorry, I've got nothing. Hope this helps, this is a pretty awesome function!