I found the family of James D. Hannay on 1861 census in Douglas, Conchan:
Alexander Hannay, 50, head, accountant, born Liverpool,
Isabella, 49, wife, born Douglas, IOM
Edwin, 21, son, printer,
James D., 16, son, coach trimmer
Lewis G., 10, son, scholar
Elizabeth Loughan, 60, lodger, plain sewer.
I hope that helps!
Janet.