If pick fights with people means taunts or being annoying, then you are right, the other person doesn't have any right to put your life into danger, no matter how annoying or puerile you are.
But if picking fights means throwing punches, pushing and shoving, or other behaviors classified as assault, then no.
Basically if you are within your rights to be where you are and do what you are doing, you are still entitled to defend yourself if attacked. Now, if you aren't in a stand your ground state, it's entirely possible that you might have a duty to retreat, but assuming that retreat is not practical or you've exhausted that possibility, your right to self defense is still extant.