David, have you tried looking for him in the 1871 census under "Edward" rather than James?
When they had two names they sometimes got swapped around [as with the James who Brian found in 1871, and who was entered as John J. in 1881].

Sue