That seems like the best idea for cork, cardboard or similar such material. To improve on that Idea, how about this. Take an old 1 1/2" hole saw, chuck it up to a drill press, lay a grinding stone on the drill press table, turn on the press and grind the teeth off the hole saw (if that is possible).

Then take a conical shaped stone (the kind that can be attached to hand drills) and grind the inside edge of the 1 1/2" hole saw until the edge is sharp.

It would then be a snap using the DP to punch holes in the cork or cardboard (with the felt attached).

Of course this is all from my imagination. I haven't the slightest idea if it will work. However I like the idea of using a punch rather than a drill in thin materials.