Dang that surprises me. I'm not sure what the point of using a smaller bolt is, but a bushing of sorts like you mentioned should work.
If you do end up drilling it out, use several layers of electrical tape over the hole and it's edges. That has always helped me when I don't want to "fray" a finish around edges.