Here in the U.S., it is my experience that over 75% of adults I meet socially will volunteer that phrase or a variation upon learning that I am a mathematician. I find this frustrating, since almost nobody would brag about being bad at history or English or really any other subject. Normally I change the subject, or perhaps say they must have had a bad classroom experience, but that feels unsatisfying and turns the conversation in an ugly direction.
Surely some of the bright minds here on Math.SE have found a good response. Note that I’m not looking to alienate a new acquaintance, so let’s keep it positive.
To put a slightly different spin on this (before @Zev votes to close 🙂 ). When folks ask me what I teach, I ask them to name their least favorite subject. Occasionally I do get English or history or science. But 90% of the time I get math … to which I reply “bingo.” 🙂 I then engage them in a discussion about how math (generically) and history (for me and many others) are ruined for the masses by secondary teachers who just don’t get it and who teach relying almost totally on rote memorization/manipulation. They concur 99% of the time on this. At some point, I reassure them that there are some really good teachers out there (and I claim to be one) 😛