you have a good chance - it was used
(a) as a barracks
(b) as a gaol
(c) as a lunatic asylum

Are there any names spring to mind ? as Castle Rushen accounts (in Manx Museum) will often give info re anyone connected with the place pre 1765