Maybe James never originated from the Isle of Man? Noticed a previous post and you advised a son was born in Scotland to James and Jane nee Martin.
Found a James Gilmour born c1840, Troon Ayrshire, a ships carpenter on 1891 Scotland Census living at 27 West Portland Street, Troon, Ayrshire living with wife Jessie, children David aged 20, Robert P 15, Elizabeth 13, John 8 and Allan 3. Cant find them together on any other Census but on 1861 Census there is a James Gilmour, ships carpenter, bc 1839, Troon, lodging at 186 Stobcross Street Glasgow. Theres a marriage on IGI for a James Gilmour to a Jessie Paton 25/11/1869 Dundonald, Ayr.
Couldnt find a James listed with a Father named Thomas on 1841/1851 Census but did find a Thomas Gilmour on 1841 Census born c1789 Renfrewshire Scotland, occupation Manf (short for Manufacturer?) address Prospeeshill, Paisley Low Renfrewshire - living with others but no James.
Sorry, nothing concrete to report, just a few observations mainly to let you know I did try and help!
Good luck.
Sandra