In the census a person's relationship should be with the head of the house. Mary's grandmother should be shown as wife's grandmother.
It is more likely that Margaret Kelly was James's grandmother & Elizabeth's mother. If so, you should look for an Elizabeth Kelly born in Castletown (Malew parish) and born between 1815 and 1820.
As an example, there is one such Elizabeth on the IGI, who was christened April 25 1819. She is the daughter of Robert Kelly and Margaret Callow.
There is a marriage in Malew between Robert Kelly and Margaret Callow on November 11 1813. this suggests that this Margaret was not in or before 1793.
There are 16 Margaret Callows born between 1783 and 1793.
It is possible that Elizabeth married before it became necessary to register marriages.
The only suitable James age 11, that I can find in 1851 is a pauper in a workhouse with George age 8 and Henry age 4. A faint possibility is that their father died suddenly, leaving their mother penniless.
However, I still cannot find Elizabeth in 1851