Len, I don't find him in 1851 census nor in 1841 index.
In 1861 index he is listed as being at And 1 44 at age 20; as it is an index, I don't know who else was there.
1871 Index he's likewise missing
1881 Andreas
03,02,009
STAFFORD, James,Village,Head,Married,40,Gen Lab,Andreas,Andreas
STAFFORD, Catherine A,Village,Wife,Married,39,,IOM,Andreas STAFFORD, Ellen J,Village,Daughter,,6,Scholar,IOM,Andreas

md Catherine Faragher 14 DEC 1872 Andreas

He doesn't seem to be in IGI nor is dau Ellen J