It has taken a second playthrough to see the subtle "Sarah" moments throughout the game. Joel unconsciously touches his watch in cut scenes and free walk he will look at it. Joel reaction to the small grave at the Dam followed up with Tommy giving him the sports photo of Sarah and himself.
Joel vocal response when he gives it back to Tommy spoke volumes of his reaction to actually seeing a photo of Sarah, something he never thought he would see again. It seemed the catalyst to his need to push Ellie onto Tommy and out of his life. The pain and fear of him realizing he was becoming emotionally attached to Ellie and his terror of not being able to protect her.
Ellie's emotional rant telling him "Everyone I've cared about has Died or left me, everyone but you!" It visually blindsided Joel, made him realize despite his best efforts Ellie had bonded with him.
The subtlety's only grow after that, you just have to play it again to see them.