Use the word riffraff when you refer in a snide way to common or lower-class people as a group. A snob might refuse to stay at a party if he thinks it's full of riffraff.
You're only going to use riffraff in a joking way, unless you truly consider yourself part of a small, elite, superior segment of humanity. An obnoxious, arrogant character in a movie might tell her chauffeur, "Jeeves, pull around the corner. I don't want to walk through that crowd of riffraff." Before the fifteenth century, the term was rif and raf, which meant both "everybody," and "sweepings."