I think if you played 5. cxe5 Bxe5 6. d4 you would have a stronger hold on the center, and would slow down his development drastically, castling on the King Gambit has the risk of the open king. Its common to see the king be force to move on the f1, but white still turns out ok.