95% of people nowadays have a dirty and, since they don't tend to their souls (and often not even their bodies), they are prone to enemy manipulations and more susceptible to enemy attacks. It doesn't mean they are attacking you personally or doing anything bad to you.
Just clean yourself and use returning curses like other people have suggested.
It gets worse with people who have severe mental illnesses and people who have severe physical illnesses, as well as overweight/obese people. At least people who do any kind of physical activity that is not just walking, get their energies going, which pushes out a percentage of their negative energies, even if not as much as a proper aura cleaning. The cleanest people you'll find among NPCs are usually competitive/elite athletes, including Olympic-level ones.