It's similar to another oft-quoted line, "Consistency is the hobgoblin of little minds." (This is actually a misquotation of a Ralph Waldo Emerson's "A foolish consistency is the hobgoblin of little minds.")
Emerson lived from 1803 to 1882, whereas Wilde lived from 1854 to 1900, so Wilde may have been cribbing.
In any case, you may find it as annoying as I do to hear someone hide behind that quote.
If so, the next time just reply, "And reciting a cliche like that is the hobgoblin of people who can't think for themselves."