TFS 2015: Intermittent Database Backup Failures (Error 53)

DevOps Azure DevOps
1

We’ve got some weird stuff going on with our TFS 2015 instance. Every now and then, the TFS database backup fails. It runs at 2 AM. It failed today with this error:

“Operating system error 53(The network path was not found.). BACKUP DATABASE is terminating abnormally.”

A system admin and I have been looking at the server. There’s plenty of disk space on the destination server where the database backups are stored, so that appears to not be the problem. The system admin reviewed the logs on the server, but she couldn’t find any issues there, either. As I said this error isn’t consistent. It doesn’t even happen once a week. Its been about 3 weeks since it happened last. But it does happen. The system admin asked me if there was a way we could test it. Is there a way we can test this?

2

You could trigger the backup jb maually…

3

Really? OK, I’ll try to figure out where that is. I’m sure its on the TFS Administration server.

4

If setup correctly, there is a job created by the backup wizards in the SQL server console.

5

But in 2015, that was something you needed to configure using the TFS power tools. So chances are the backup was never setup according to the correct practices, since they differ from SQL server standard practices.

6

I hope you’re wrong, but that could be. All that configuration happened before I was hired.

7

And FWIW, this job has been running fine for years, so what’s changed? Why does it now occasionally not work?

8

maybe network dysfunctionalities? a scheduled switch reboot?

9

You might have something there, Daniel. It does make sense to me that it has to be something outside of the TFS system that’s causing this interruption.

10

I’m in the TFS Admin console. I see a “Take Full Backup Now”. This server isn’t used much, so I guess I can do that.

11

You may try to change the backup location to a “closer” one

12

I’ll have to ask the sys admin where that might be

13

BTW, FWIW, it takes me more than a minute and less than 2 minutes, to remote into the TFS Admin server.

14

Or maybe the destination server acts as backup location for other systems that are doing the backups at the same time with TFS

15

I’ve asked the sys admin about it. Both the TFS Administration server and the TFS database server are on VMs in the same building.

16

I manually launched the full backup job from the TFS Administrator server. After more than 30 minutes I got the following error:

TF401393: The full backup job has not completed after 30 minutes.
Googling that error I got several things, but one of the was the job “… fails to complete within the expected timeframe”. I suspect that the original TFS admin configured it to run for 30 minutes, but now it might take longer. I’m going to leave the the job running because at the top of the popup it says, “The full backup job has not completed yet.” So, I conclude that it is still running. At this point its getting close to the time I leave, so I’m tempted to just leave it all running and lock my machine.

But, if I’m correct about the original TFS admin only configuring it to run for 30 minutes, I presume I can click on the “Reconfigure Scheduled Backups”. I’d increase it to 60 minutes. However, having never done this before, I’m hesitant because I don’t know what all the original developer configured and if that reconfiguration would give me what the original guy entered, allowing me to accept what he’s put in, or if it will give me blanks for all the options to configured. Does anyone here have any idea about that?

17

It’s been too long since I opened that screen.