I think you may have a couple of generations/families mixed up.

1841, Andreas verra Moar
Thomas Ellison, 45, agricultural labourer
Jane, 45
Daniel, 15
Robert, 13
Frederick, 10
Charles, 8

1851, Jurby, Kerrowcroie
Thomas Ellison, head, married, 55, agricultural labourer, Andreas
Jane, wife, married, 60, Ballaugh
Ellenor, daughter, unmarried, 29, Andreas
Fredercik, son, unmarried, 21, agricultural labourer, Andreas
Charles, son, unmarried, 19, agricultural labourer, Andreas
Mary Hogg, grand daughter, 9, scholar, Andreas
Susan Quarry, grand daughter, 5, scholar, Andreas

1861, Ramsey, King Street
Charles Ellison, head, married, 26, engineer, Andreas
Jane, wife, married, 26, Lezayre
Thomas, father, widower, 67, labourer, Andreas
Louisa, daughter, 6 months, Ramsey