My G G Grandparents Richard Cavendish and Elizabeth (formerly Moore) at the 1871 census are shown as having Robert S Moore, nephew aged 5, staying with them as a lodger. He was still with them at the 1881 census. Could he be your Grandfather? In 1871 they were living at St. Georges Walk, 1881 Shaws Brow, both Douglas.

Mike