You can permanently delete files (without them going into your recycle bin) by right clicking the file, then holding down the 'shift' key while clicking 'delete'. It will ask for conformation to permanently delete the file. Click 'yes', and its GONE!
Hater of the recycle bin,
~Mike :)