Thanks Frances, that's really helpful. I did have most of this info but I hadn't considered the lone Swan family to be a clue.

You triggered the thought that maybe they had changed their name to Swann from Swan quite recently and that was why I was having such difficulty tracing them. I've now found Benjamin's son, my wife's grandfather - he's registered in the 1901 census as SWAN!