Have you seen on the IGI
James Stafford married Catherine Fargher 14 Dec 1872, Andreas. He is presumably the son.

Also on the 1881 census in Andreas
James Staford (1 f) 40 born Andreas, gen lab
Cathrin Staford wife 39
Ellen Staford dau 6
If he was illegitimate you could search the Andreas birth register for 1841 ish for a James ? with a mother Elizabeth ? Perhaps its an epic tale of true love!