Huge thanks to everyone for their help - much appreciated. There are some definite oddities in the census returns, hence my confusion.
I will wade through everyone's comments and see what I get. I have just been in contact with someone called Walter who's grandad is the son of James(c1846) and brother to my gr grandad Thomas. Maybe together we can sort this out!