Sounds like he is draining your energies, so you feel weak while with him.
People with many problems may unconsciously become a psychic vampire; if you are open to him (i.e. willing to help, feeling guilty for his problems, etc) this leads to the loop of giving out energy to an energy sucker.
Also he may have a very negative energy in his aura, so you feel down because of this; spending the day in a crowded city with people, leads to a lot of negativity most often.
Unless he is SS, it's very difficult to help him get rid of problems and negativity. Maybe a therapist would help to let him feel better and solve some problems, always for a "normal" person.
If he also has some jewish genes, this is focal point. Jews are natural vampires and are obviously very negative.
I'd consider moving out of this place, or use magick to crash his job so he won't came to your home. If he has jewish genes, HE IS a jew! Who wants a jew in his house???