Why are religious organizations tax exempt?

I've always felt that the good works that churches do can't make up for the billions lost in tax revenue that could do so much more. Imagine if churches paid taxes and still went on missions to rebuild homes and donate clothing and organize food drives. We'd all be so much better off.