try http://www.ee.surrey.ac.uk/Contrib/manx/towns/castltwn/index.htm
and in particularhttp://www.ee.surrey.ac.uk/Contrib/manx/towns/castltwn/map2.htm
the Mill was a corn mill (still standing but rebult) to south of the mill pound
(see map and look at top - you will find much on interest in the referenced texts on that page
Few people owned their own house until middle or later of 20th c. - you rented the house - often built by local builders from early 19th C as an investment