A good description is given by A Rigby's overview of the main phases:

1. The old square tower, probably built about the middle of the 13th century, and partly destroyed by Robert Bruce in 1313. The remains of this tower formed the nucleus of the Castle of the next period.

2. The 14th century Castle, incorporating the old tower, probably built by Sir William de Monacute about 1344.

3. The Castle, as further fortified for protection against cannon, said to have been done by Cardinal Wolsey early in the 16th century.

4. The Castle domesticated by the erection of the Derby House in the main ward and alterations to the sally port, etc. Since the fourth period the Castle has been used as a prison and lunatic asylum, and was considerably altered for these purposes. The restoration consisted largely in removing recent additions and restoring parts altered to fit the buildings for the above uses, so that the effects of this period being practically obliterated, it need not be taken into consideration.

Gateway

The photograph, dating from c.1905, shows the main gateway behind which is Derby House, the building erected by James VII Earl of Derby c.1645.

It also shows some of the Victorian accretions (to the right of the gateway) which were cleared away by Goverenor Raglan c.1910.

The stange shaped building on the left is the Baillie Scott designed police station.

Outer and Inner Gatehouses

The photograph, dating from c.1905, shows the 14^th century outer gatehouse, behind which within the curtain wall is the earlier 14^th century inner gatehouse

References

The most informative is probably the Armitage Rigby account published posthumously from his notes, it unfortuneately breaks off somewhat unfinished
Rev Joseph George Cumming, MA FGS, The Story of Rushen Castle and Rushen abbey in the Isle of Man London: Bell & Daldy 1857
J.M. (John Moore) Jeffcott, Castle Rushen Manchester: Guardian Printing office [c1893]
Castle Rushen: its history and legends Douglas: Brown & Son [1880]
Castle Rushen: its history and legends Birmingham: Munns and Allen [1895]
Joseph E. (Edward) Douglas, Castle Rushen. This account of Castle Rushen was compiled for official use by Joseph E. Douglas ; illustrations are from drawings made by E.C. Quayle. Douglas : Brown & Sons, 1925 [reprints 1935; 1946]
A Rigby Castle Rushen Proc IoMNHAS vol 1 #3 p51-53, 1907
A. (Armitage) Rigby, Castle Rushen: a historical and descriptive account. Douglas: Victoria Press. 1927
Maurice Holmes,Castle Rushen Isle of Man Official Guide [Douglas: Gov Prop Trustees ?] 1962
Norman and Margaret Gale, Castle Rushen: Official Guide [c.1970]

W.R. Serjeant Castle Rushen Gaol Proc IoMNHASoc VI #3 pp277/294 1963