The argument for the word 'Manx' is that it comes from 'Manks', which was perhaps originally 'Mansk'. This makes sense if you speak a Scandinavian language I think. In Danish, someone from Denmark is 'Dansk', in Sweden a Swede is 'Svensk', and in Norway a Norwegian is 'Norsk'. So someone from Man is 'Mansk'.

As to why the isle of 'Man'... that's a very long story!