Insulting brother getting me down?

My younger brother is always insulting me, saying I am hideous and fat and I have no friends because I don't see them as often as he sees his because they live quite far away, but it's not like I'm a loner or anything like that, I am sociable. But it just really bugs me that he always calls me a freak and that no one likes me because I am so ugly, my other family tell me to ignore him and they say I'm beautiful but I have to admit it has dented my confidence considerably and I find it hard to talk to my friends now as I'm paranoid they hate me. Is this normal for a brother to behave like this, oh, and he has mild to medium aspergers syndrome which makes him misbehave and lash out at me a lot, it's just really depressing me and I feel I have no one to talk to. Thankyou.