In the past 8-10 weeks I've had a nightmare trying to get a reliable build process. Here's the numbers of what's causing the build to fail *other* than normal coding errors:
-
30 failures due to CVS temp directory problems
-
2 failures due to CVS file corruption
-
2 failures due to unknown CVS error
-
13 failures due to timeout when doing CVS tagging
-
4 failures due to email error
-
2 failures due to internal network problems
-
11 failures on the build machine due to not enough disk space
To solve the CVS temp directory problems we've resorted to having a script run on the CVS server that periodically deletes any CVS temp folder that is older than 6 hours old. Even so, there are times when the script can't delete the folder because it is locked open. Something is happening in CVS to cause processes on the server to hang around even though the client action is long gone.
The CVS timeouts -- I have no clue on that one yet.
Not enough disk space on the build machine -- that one was deceptive because the errors never actually said "disk space". They were all just unexplained failures or file not found or something of the sort. Now that I realize that's an issue I've set up an alert to warn me when disk space goes below 10% free. Just something to keep in mind!