I am not a metalworker but am particularly interested in your project. The idea of broaching the holes seems sound. However, I wonder if the "tube" would try to turn inside-out when you retract the tool since it will be an interference fit. My suggestion would be to have plenty of blanks to begin with - just in case.