I had IT check the CVS server for any error event in the event log and we found that there were disk I/O errors that often occurred right around the same time as some (all?) of my mysterious CVS errors. IT dropped in some new hardware, but that still hasn't reduced the errors.
Looking in the event viewer, the errors are event (type) 15, category of none, and source is "disk".
The errors happen in 3 second groups...like from 11:21:08 through 11:21:10 there will be about 9 of these errors, a few each second.
Our current theory is that it's some sort of throttling issue where the sheer bulk of activity involved in doing the CVS update and CVS Labelling is just too much for our system. No real idea of a solution to fix it yet, but... there ya go.
We also modified our build schedule so that builds are not occurring at the same time as the daily backups. Again this has improved things to some degree. Another line of attack on that problem is doing some Spring Cleaning on our development servers to reduce the amount of items to be backed up. It's easy to consider disk space as 'cheap' -- and it is -- but there is still overhead to having a lot of disk space in use. So.. it's wise to go ahead and spend some time cleaning up the servers and doing the basics: delete the obsolete, archive the historical but no longer active data off to off-line storage, trash the 'let me just store this here temporarily' stuff, etc.
The roles of IT and Build engineers sometimes overlap, so do whatever you need to to maintain a good relationship with the hardcord IT guys. Me, I make it a point to find out what their favorite food is and bring them a treat every couple months. 