Being terribly mean or violent on purpose is viciousness. The viciousness of your friends' argument might have startled you — usually, they get along well with each other.
Someone with the trait of viciousness is guilty of acting in a deliberately cruel or callous way. Think of the viciousness of a brutal dictator, the viciousness of a bully, or the viciousness of nasty gossip that's meant to hurt someone deeply. Viciousness and vicious share a root with vice, or "wickedness," from the Latin vitium, "defect or offense" — which is fitting, as there's nothing more offensive than viciousness.