How did your family conclude that William Henry and George Herbert were the son's of James ?

All indications show them to be the son's of Arthur Wylde and Caroline Wadsworth.

1861 at Ladybarn Withington:-

Otho (Arthur) Wilde 36 Coachman Derbyshire Ashford
Caroline 30 wife Manchester
William H. 10
Emily 9
Edwin 7
Eleanor 3
Sydney 3mths

They are still in Withington in 1881 with George H. added.

In 1861 Mary, the wife of James, is 42, by the time George was born 1871 she would be to old.

There are no children in the census or IGI called William or George to James and Mary.

In her reply Carolyn states the marriage certificate for William shows his father as Arthur.

Len Faragher