How do I not worry?

If you both have a fear of commitment, maybe he is scared of starting things with you also. The two of you seem to be able to talk and have in depth conversations. Maybe you should ask him out. Or if the conversations and hanging out keep happeneing eventually he will ask you out. Just keep yourself guarded around him. But odds are if you feel the chemistry so does he. Good luck.