A recurring theme in The Enchiridion, and Seneca's work too, is that we too easily give others influence over us. We should take as much care of our minds as we do our bodies.
If any person was intending to put your body in the power of any man whom you fell in with on the way, you would be vexed: but that you put your understanding in the power of any man whom you meet, so that if he should revile you, it is disturbed and troubled, are you not ashamed at this?
In this particular case, Epictetus is warning against being controlled by other people's opinions of us, which is useful advice. More broadly, the Stoics advise us to be careful of the company we keep, because it can be very hard to counteract the influence of peers. Modern research supports this idea - fitness, depression, divorce, smoking and host of other attributes are massively influenced by the people around us. If we want to persue personal improvement we should avoid people who will hamper us, and always be aware of the influence and power we give others over our thoughts and opinions.