The 28-year-old right-back has put pen to paper on a two-year deal after leaving Preston at the end of last season.
He had made 72 appearances for the Lilywhites, having joined them from Rochdale in January 2019.
Blues boss Danny Cowley said: “Joe brings Championship experience from a successful Preston team and has also performed very well in League One.
“He didn’t play as much as he would have liked last season and so I think he’ll come feeling energised and looking forward to the challenge that playing for Pompey brings.
“Joe is a boy who really loves to win and the ambition that he has aligns with the ambition of the club.
“We’re a pretty youthful squad and he offers a lot of experience and, importantly, he’s also a great character.
“He’ll be a role model in the way he conducts himself, so he can help in the development and support of our young players.
“Joe has played at Fratton Park before and knows how passionate our fans are, so he was really eager to sign.
Rafferty – who has represented Republic of Ireland at youth level – progressed through the ranks at Liverpool.
He signed for Rochdale in the summer of 2012 and featured more than 250 times for them, helping to secure League Two promotion.
