He is in the 1881 census as married, so could have just been visiting his mother in the IOM on that date.
Where were Elizabeth and their children in the 1881 census, whose ages you must have, because that should show you whether their husband and father was away?

I posted that 1881 census transcript again for you on 12 May when you were looking for the sisters, and wrote:
James and John were living with their mother in 1881:
Mary (Catharin) QUAYLE Head W Female 64 Maughold, Isle of Man, England Housekeeper
James QUAYLE Son M Male 42 Maughold, Isle of Man, England Labourer and Brick and Tile Works
John QUAYLE Son U Male 20 Maughold, Isle of Man, England Farm Labourer
Source Information:
Dwelling Booleey Velt Cottage
Census Place Maughold, Isle of Man, Isle of Man
Family History Library Film 1342345
Public Records Office Reference RG11
Piece / Folio 5600 / 34
Page Number 12
[Free on FamilySearch.]

I thought it was clear that this was Mary's son James Quayle, bapt Maughold 12 Jan.1840, son of Philip Quayle & Mary Corkill, and who appeared with them in censuses in 1841 and 1851 aged 1 and 11 [posted by Brian on the 13th and Donna on 14 April]. Not with the family in 1861.

Mother Mary dau of James Corkhill & Mary Callow bapt Maughold 20 Nov 1816.

No likely James Quayle marriage in Maughold with father Philip.

There was a James Quayle of the same age in the Maughold burial register:
James QUAYLE 57 yrs. Maughold 25 Oct 1897.
And a will on Brian's index which would identify which James Quayle:
QUAYLE, James 1898 Mau [no.] 121, LDS film no. 0106528.

Sue