Are you comparing me to some old lady?

Would you prefer "ne'er do well"?

Defined this way per Wikipedia --
"Ne'er-do-well" is a derogatory term for a good-for-nothing person; or a rogue, vagrant or vagabond without means of support. It is a contraction of the phrase never-do-well.

Or another old time favorite: an "unsavory" character.
True story: My Cub Scout den mother would warn us to watch out for unsavory characters. (Though by the time we became Webelos cubs, we figured we could take care of ourselves!) I remember her admonition to this day!