Maybe it's just my drill bits, but when I drill a 1/4" hole for a 1/4" dowel, it never fits! I end up sanding the dowel until I can slide it in, and it is a big time-waster.
Recently, we used 8mm barrel hinges for a box, and found out that a 8.2mm drill bit was recommended by the manufacturer. I decided to apply the same principle to my dowel dilemma, using one of the metric drill bits that was part of an inexpensive set I bought on eBay.
I looked at my conversion table, and found that 1/4" is 6.35mm, so I drilled a test hole with a 6.5mm bit. As I had hoped, the dowel slid in snugly, but easily.
Metric bits can be hard to come by, but those cheap sets from China are perfectly adequate for occasional tasks, and nice to have on hand. They certainly make what should be a simple task exactly that!