Glenys,
Do you know what his occupation was (from marriage certificate, daughter's birth certificate and/or 1891 census)?
Possibly, he was a sailor and away at sea.
Also, can you track his wife in the 1881 census? Depending on the marriage date, this may be a clue as to where James was based.
Bruce