Hi Shirley,
I think there was a slip of the key in a previous message - easy to do - the Every St census entry was from 1871 not 1861. Your 1856-James is a 14 year old Joiner's apprentice born IOM, son of 1835-James born Coleford, Glos.
It is confusing that in the 1881 census 1856-James, husband of Henrietta, is down as a cab driver from Glos, although in 1891 he is a plasterer from IOM:
RG12/2954 f.105 p.41? sch.244
65 Wye St, Everton, Liverpool
James Williams, h, m, 34, Plasterer, born IOM (1857)
Henrietta Williams, w, m, 35, Liverpool
Esther, d, s, 14, tailoress, Liverpool
Lily, d, 4, Liverpool
John H., s, 9mo., Liverpool
William, s, 9, scholar, Liverpool
James, s, 8, Liverpool (1883)
The 1901 census has him:
RG13/3479 f.37 p.6, Everton
James Williams, h, m, 45, Builder, born IOM Port Erin, (1856)
Henrietta Wiliams, w, m, 45, Liverpool
Esther, d, s, 23, dressmaker, L'pool
William, s, s, 18, bricklayer, L'pool
James, s, s, 16, appt joiner, L'pool (1885)
Lily, d, s, 14, L'pool
John, s, s, 10, Everton
Albert, s, s, 9, Everton
Henrietta, d, s, 8, Everton
Marian, d, s, 2, Everton
As someone pointed out, in the 1871 census (I couldn't find them in 1861 either) the wife of 36-year old James, a Groom from Coleford, Glos, 28-year old Sarah from Scotland, would have been only 14 at the time of James' birth!
Other evidence besides censuses seems essential - there is a family on the 1881 census which could almost be James & Sarah as the pattern of names/ages of the eldest children agree, but not other significant details, and probably you could find hundreds of others similar.
RG11/3666 f.93/4 p.20/21 sch.102
122 Killshaw St, Everton
Jas Williams, h, m, 49, Cabman, Herefordshire, Walford
Sarah Williams, w, m, 32, Liverpool, Prescot
Suzanna, d, u, 18, charwoman, Glos. Coleford
Christopher, d, u, 11, scholar, Lancs West Derby
Sarah Ann, d, u, 10, scholar, Lancs West Derby
Elizth, d, u, 8,scholar, Lancs West Derby
Emmeline, d, u, 5, scholar, Lancs Liverpool
Gertrude Blanch, d, u, 3, scholar, Lancs West Derby
Isabella, d, u, 1, scholar, Lancs West Derby
Louisa, neice, u, 21, seamstress boarder, scholar, Lancs West Derby