Val - I couldn't find either of them in the 1861 census of either the IoM or England but you may find the following interesting.

1851 IoM census Lezayre, address given as "Windmill"
Charles Brown head 47 Miller Journeyman b. England
Maria Brown wife 41 b. England
Alexander Brown son 18 b. England
Horatio Nelson Brown son 16 b. England
Charles Brown son 12 b. England
John Brown son 9 b. England
Edwin Brown son 5 b. IoM Peel

1871 census Maughold
Horatio N Brown head 36 Innkeeper b. England
Ellen Brown wife 32 b. England
Howard Brown son 9 b. Ramsey
Charles E Brown son 6 b. Ramsey
Annie F daur 7 b. Ramsey
Lilly daur 5 b. Ramsey

I know the age given for Ellen does not agree exactly with the 1881 census but my experience is that they often don't.

1871 census Douglas
Edwin Brown head 25 Corn Miller b. Peel
Isabella Brown wife 24 b. Rushen
Horatio N. son 2 b. Braddan

Mike