Hugh's family line was taking a turn in the background of my researches for a few months while I went down other tracks, until the issue of Christian names caught my attention. However, you're quite right, if I'm very lucky someone might be able to help me with this 'mysterious' branch of the family. This is what I've got so far:

Sarah d. Hugh & Margt Cavendish, Douglas, Boatman, bapt. August 21st 1859 St Matthew’s
Nessy d. of Hugh & Margt Cavendish, Boatman, bapt. Jany 5, 1862 St Matthew’s

(1861) 10 Paradise/Parade St, Douglas. RG 9/4418 f.27 p.8 sch. 28 ED6
Hugh Cavendish, H, m, 32, Boatman, b. Douglas [1829]
Margaret, W, m, 33, Boatman’s wife, Braddan [1828]
John, s, 9, scholar, Douglas [1852]
James, s, 7, “ “ [1854]
Margaret, d, 4, “ [1857]
Sarah, d, 1, “ [1860]

Sarah died 1861.

(1871) 4 Barrack Street, Douglas. RG10/5776 f35 p5 ED2 sch33 (cd71)
Hugh Cavendish, H, m, 48, Boatman, IOM [1823]
Rose, w, m, 32, Shopkeeper, Ireland [1839]
Margaret, d, 15, scholar, IOM [1856]

(1881) 18 Church Street, Douglas. RG11/5603 f30 p17
Hugh Cavendish, H, m, 57, Boatman (on seas), Douglas [1824]
Rose Cavendish, wife, 41, Green Grocer, Ireland [1840]

Brown’s Directory 1881/2: Cavendish, Hugh, labourer, 18 Church-street

Will: CAVENDISH, Hugh, 1891, Dgs, 12, 0106512
which only mentions wife Rose and that Hugh “died 24th December 1890”

Cavendish, Rose, 70, buried Braddan 1908 c132

I've got what I think are sons John & James in censuses from 1871.

Marriages:-
16 March 1873, John Cavendish, porter, Douglas, s. of Hugh Cavendish, boatman,
m. Georgina Craine, Douglas, d. of James Craine, labourer, St Barnabas REG

1885 Apr 12th Kirk Braddan Book 9073/3/12 p.245 No.489
Margaret Cavendish, full age, Sp, 8 Tynwald Terr, Douglas, d. Hugh Cavendish, porter &

William Shimmin, full age, B, Lab., 12 Edward Ln, Douglas, s. William Shimmin, lab.
Wit: ~~~~~~~~~ & Elizabeth McComby her X mark

I've got Margaret & family in 1891, and births & her burial from Brian's indexes.

As you can see, I've no parents, baptism, marriages or burial for Hugh Cavendish, but he was definitely there!