My experience is that priests have little or no hands-on experience with kids of any age. I've known a couple that were naturals (the ones that people referred to as Father What-a-Waste), but 90% of them do not understand kids at all (including some of those teaching in Catholic schools, oy). This priest sounds like he is clueless and insensitive when it comes to wee ones in general and Wee Hob in particular; if I were you, I'd try to find a compromise that allows him to attend one service a week with his godfather (perhaps if you point out that you won't be able to sing in the choir otherwise) and try to find alternate activities for him during the other times you have to be at Mass to sing. If I'd had to attend more than one Mass a week as a kid, there would have been some serious rebellion on the horizon.

Alternatively, I'd go with what everyone else is saying and find another church, since the last few lines of your entry make me think this is part of a trend. Another place might let him train as a server as well.