All of that doesn't matter if the game runs like crap. Granted most games will run at least at 30 (only most). I care because I can deal with 30 fps, I just prefer 60 because if their are any drops I probbly won't notice it as much. I like fluidity as do all gamers.
If you have 60 FPS and get frame drops then it'll be around 45FPS. With last gen consoles the frame drops would be around 15-25FPS which was obviously an issue. Smoother looking games is what any gamer should look for. But to answer the question, you and everyone else should care.
Graphics are nice. Gameplay, storyline, character development, and functionality are so much more important to me than graphics.
Fun factor is also extremely important along with a sense of fantasy. I don't play games to have them be real. I play games for the adventures, stories, and for them to be fake. If they remade Battle Toads on today's consoles it would be amazing and I would spend $60 for that game all over again.