Our son was a September birthday, and because the cutoff in NJ was October 1 he started when he was four. He was also small for his age and ended-up going through puberty a year or two after most boys; so he was absolutely tiny when he started high school and it kept him from playing football.
That said, the only reason we ever thought we should have held him back a year was because so many others parents did; starting mainly with June birthdays. If everyone simply started their children in school at the appropriate time we would have had no issues. So how much of people wishing they are started their children later is because they really weren't ready for school, and how much is because everybody else is holding their children back? Where does it end?
When we spoke to our school the summer before he started we asked if he was ready to start. We went through his academic abilities, and they said, "None of that matters. It's our job to teach them how to read and write. Our question to you is just whether or not he is able to leave you for the day." If they're okay to leave their parents for a few hours every day, they are read to start school.