In 1871 your John Hannah was a Harbour master boatman aged 27 born in Douglas.
The 1881 John, prisoner aged 36, was a Dock Labourer, born in German.
None of the rest of the family in the 1871 census (Jane, John, Margaret, Matthew and Patrick) appear in the 1881 census.
This suggests that the family had left the Island.
Patrrick Hannah reappears in 1891 and 1901.
In 1891 he is 27 born and living in Douglas, as were other family members (Mary Ellen, wife 24, Margaret Jane daughter 6, Mary Ann daughter 4, Thomas E son 2, and John N son under 1 month). Per Brian' marriage index, Atrick Hannay married Mary Ellen Martin in Braddan on July 17, 1884 (note spelling and that Patrick's "P" is missing)

My guess is that they went to Ireland, Scotland, the US or the southern hemisphere and Patrick returned alone.

Bruce