When something affects you adversely, it's harmful or negative. Holding a cinder block will adversely affect your running speed.
When you talk about things that are acting adversely, they are harming something or holding it back in some way. Not getting enough sleep will adversely affect your performance in school. Rain will adversely affect an outdoor wedding. Anything that's making things worse instead of better is adversely influencing the situation. Adversely means something close to badly or harmfully. So when you see this word, you know something is going wrong.