There are only about 22 Barrys on the 1901 census in just a few families - one of which was, presumably, your grandfather's. Unfortunately, none of them are called James, so he would seem to have been born after the census.
Your best bet would be to start your research with birth and marriage certificates of your parents, then grand parents. Your aunts might know how old your grandparents were - I would try asking them. They might know other details too...
Best of luck - Roger