I am 21, he is 26,...we are crazy for each other..but he has kids?

I was 18 with a 24 year old bf that had two kids. We stayed together 8 years and had a child of our own. If you're crazy about him then go ahead, what's the worst that will happen? He's obviously a good dad if he has joint custody!