It was the 1871 census, not 1861.
Your 122 Kilshaw St. Williams family looks a good bet, despite the discrepancies in ages and plces of birth (1871 census has Sarah 28 and born in Scotland, and Susannah 10 and born in the Isle of Man).
Another possible lead:
there is a English born James Williams, lodger, 26, Coachman (cab driver) at 7 Belmont St. St Nicholas, Aberdeenshire in the 1861 census for Scotland (no note of whether married or not)