I don't think it is unfair at all. Federer played his A game in both finals but Murray simply didn't play well, serve, as per usual, playing a significant role in those defeats.
2 finals, 2 poor performances.
He's a damn fine player but he does have a brittle temperament.