unicorn is technically very correct. Although I use a automatic defragmentor just for the sake of it and psychological satisfaction, I too join with unicorn to tell you the performance difference I see is: Close to NONE.
I do DVD Backups and create/move/delete 20 Gigs of files almost everyday. So the Hard Disk is fragmenting very frequently.
Although I previously suggested you a defragmentor, (just because, "Not using a defragmentor" was not one of your requirements)
not using a defragment is also one of your options. 
Cheers,
McoreD
P.S: Attached a Poll.