John doesn't really notice the distance at all. He's never been one to mind personal space a great deal, and this is all so comfortable he's definitely just filtering it out. It's nice, so it doesn't matter much.

"I am kind of curious to see what sort of family produced you."

Which means to say that he would be glad to visit, and not only for that reason, although it's definitely a part of it. But there's no reason he wouldn't want to, really. He likes Sherlock, after all.