We are getting frequent backup failures, when backing to URL below is the message from event log

An error occurred during data transfer operations with SqlServer, HRESULT:  0x80770003

SQLVDI: Loc=CVDS::Close. Desc=Open devices!. ErrorCode=(0). Process=7396. Thread=7764. Client. Instance=MSSQLSERVER. VD=Global\https://storage/dbname.bak_SQLVDIMemoryName_0.

BACKUP failed to complete the command BACKUP DATABASE dbname. Check the backup application log for detailed messages.

Also below is the detailed message after turning on Traceflag 3051

1/27/2015 9:10:28 AM:    ======== BackupToUrl Initiated =========
1/27/2015 9:10:28 AM:    Inputs: Backup = True, PageBlob= True, URI = https://storage/dbname.bak, Acct= xxxxxx, Key= xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx, FORMAT= True, Instance Name = MSSQLSERVER, DBName = dbname LogPath = G:\MSSQL\logs
1/27/2015 9:10:28 AM:    Process Id: 7396
1/27/2015 9:10:28 AM:    Time for Initialization = 31.2446 ms
1/27/2015 9:10:28 AM:    BackupToUrl Client is getting configuration from SqlServr
1/27/2015 9:10:28 AM:    Time for Handshake and VDI config = 15.6255 ms
1/27/2015 9:10:28 AM:    Time for Get BlobRef = 32.0122 ms
1/27/2015 9:10:29 AM:    lease id is 258757f9-8f8b-4110-bf56-7ab1a5caf67f
1/27/2015 9:10:29 AM:    Time for blob create = 343.0298 ms
1/27/2015 9:10:49 AM:    A timeout occurred on GetCommand, timeout length of 20000, will retry
1/27/2015 9:11:09 AM:    A timeout occurred on GetCommand, timeout length of 20000, will retry
1/27/2015 9:11:29 AM:    A timeout occurred on GetCommand, timeout length of 20000, will retry
1/27/2015 9:11:49 AM:    A timeout occurred on GetCommand, timeout length of 20000, will retry
1/27/2015 9:12:09 AM:    A timeout occurred on GetCommand, timeout length of 20000, will retry
1/27/2015 9:12:29 AM:    A timeout occurred on GetCommand, timeout length of 20000, will retry
1/27/2015 9:12:49 AM:    A timeout occurred on GetCommand, timeout length of 20000, will retry
1/27/2015 9:13:09 AM:    A timeout occurred on GetCommand, timeout length of 20000, will retry
1/27/2015 9:13:29 AM:    A timeout occurred on GetCommand, timeout length of 20000, will retry
1/27/2015 9:13:29 AM:    Backup communication with SqlServr failed, hr = 0x80770003
1/27/2015 9:13:29 AM:    A fatal error occurred during Engine Communication, exception information follows
1/27/2015 9:13:29 AM:     Exception Info: An error occurred during data transfer operations with SqlServer, HRESULT:  0x80770003
1/27/2015 9:13:29 AM:     Stack:    at Microsoft.SqlServer.VdiInterface.VDI.PerformPageDataTransfer(CloudPageBlob pageBlob, AccessCondition leaseCondition, Boolean forBackup)
   at BackupToUrl.Program.MainInternal(String[] args)

• Edited by kannu123 Tuesday, January 27, 2015 6:03 PM

Hello,

Since you are receiving timeout errors when trying to backup to a storage container on Azure, could you please confirm that your network meets the minimum bandwidth requirement of 1 MB?

What is  the size of the database you are trying to backup to Azure? Could you try to perform compress backups if the database has a big size?

The backup to URL is a multi-threaded process; please confirm a proxy is not in the middle limiting the number of connections.

The following resources may be useful.

https://technet.microsoft.com/en-us/library/dn449495.aspx

http://www.sqlcoffee.com/SQLServer2014_0006.htm

Hope this helps.

Regards,

Alberto Morillo
SQLCoffee.com

• Edited by Alberto MorilloMVP, Moderator Tuesday, January 27, 2015 6:25 PM

Thanks Alberto, the db size is around 750 GB and I am using compression. How can I check if proxy is limiting the number of connections?

Hello,

It is indeed a big database. You need bandwidth for that. Test the throughtput of backups to Azure using smaller databases first.

Is it possible for you to contact your system administrators and verify the proxy configuration or other devices in the middle?

By the way, has your network been upgraded to Ethernet fabrics?

https://social.msdn.microsoft.com/Forums/en-US/251e862a-59d8-4cb4-a354-1195b6356a96/moving-onpremise-sql-server-to-azure-vm-makes-sql-jobs-execution-take-longer-time?forum=sqlservermigration

Hope this helps.

Regards,

Alberto Morillo
SQLCoffee.com

• Edited by Alberto MorilloMVP, Moderator Tuesday, January 27, 2015 7:03 PM

You backup750 GB via WAN to Azure? Respect, a sporty challenge.

On the machine where SQL Server is running open Internet Explorer, go to "Internet Options" => tab "Connections" => button "LAN settings", there you find proxy settings.

The failures are intermittent, I have Automatically Detect Settings and no proxy defined on LAN Settings. So seems to be an issue with size? Also I get intermittent failure for backups of smaller size.

Are you using T-SQL for the backup? If so you could use stripped backup = backup to several BAK files instead of one; may this helps?

I was looking at the Total DBSize and not whats used, the backup size is around 115GB with compression, I will try to strip the backup across multiple files.

We are getting frequent backup failures, when backing to URL below is the message from event log

An error occurred during data transfer operations with SqlServer, HRESULT:  0x80770003

SQLVDI: Loc=CVDS::Close. Desc=Open devices!. ErrorCode=(0). Process=7396. Thread=7764. Client. Instance=MSSQLSERVER. VD=Global\https://storage/dbname.bak_SQLVDIMemoryName_0.

BACKUP failed to complete the command BACKUP DATABASE dbname. Check the backup application log for detailed messages.

Also below is the detailed message after turning on Traceflag 3051

1/27/2015 9:10:28 AM:    ======== BackupToUrl Initiated =========
1/27/2015 9:10:28 AM:    Inputs: Backup = True, PageBlob= True, URI = https://storage/dbname.bak, Acct= xxxxxx, Key= xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx, FORMAT= True, Instance Name = MSSQLSERVER, DBName = dbname LogPath = G:\MSSQL\logs
1/27/2015 9:10:28 AM:    Process Id: 7396
1/27/2015 9:10:28 AM:    Time for Initialization = 31.2446 ms
1/27/2015 9:10:28 AM:    BackupToUrl Client is getting configuration from SqlServr
1/27/2015 9:10:28 AM:    Time for Handshake and VDI config = 15.6255 ms
1/27/2015 9:10:28 AM:    Time for Get BlobRef = 32.0122 ms
1/27/2015 9:10:29 AM:    lease id is 258757f9-8f8b-4110-bf56-7ab1a5caf67f
1/27/2015 9:10:29 AM:    Time for blob create = 343.0298 ms
1/27/2015 9:10:49 AM:    A timeout occurred on GetCommand, timeout length of 20000, will retry
1/27/2015 9:11:09 AM:    A timeout occurred on GetCommand, timeout length of 20000, will retry
1/27/2015 9:11:29 AM:    A timeout occurred on GetCommand, timeout length of 20000, will retry
1/27/2015 9:11:49 AM:    A timeout occurred on GetCommand, timeout length of 20000, will retry
1/27/2015 9:12:09 AM:    A timeout occurred on GetCommand, timeout length of 20000, will retry
1/27/2015 9:12:29 AM:    A timeout occurred on GetCommand, timeout length of 20000, will retry
1/27/2015 9:12:49 AM:    A timeout occurred on GetCommand, timeout length of 20000, will retry
1/27/2015 9:13:09 AM:    A timeout occurred on GetCommand, timeout length of 20000, will retry
1/27/2015 9:13:29 AM:    A timeout occurred on GetCommand, timeout length of 20000, will retry
1/27/2015 9:13:29 AM:    Backup communication with SqlServr failed, hr = 0x80770003
1/27/2015 9:13:29 AM:    A fatal error occurred during Engine Communication, exception information follows
1/27/2015 9:13:29 AM:     Exception Info: An error occurred during data transfer operations with SqlServer, HRESULT:  0x80770003
1/27/2015 9:13:29 AM:     Stack:    at Microsoft.SqlServer.VdiInterface.VDI.PerformPageDataTransfer(CloudPageBlob pageBlob, AccessCondition leaseCondition, Boolean forBackup)
   at BackupToUrl.Program.MainInternal(String[] args)

The size of the backup file is 115GB and I have verified there is no Proxy settings on the server, also its happening on multiple servers whereas the Tlog backup to the same container are successful all the time.

• Moved by Manu RekharMicrosoft contingent staff Wednesday, January 28, 2015 9:31 AM Moving to correct category
• Merged by Katherine XiongMicrosoft contingent staff, Moderator Friday, January 30, 2015 9:18 AM duplicate

Hi,

The issue seems to be more related to SQL Server than Azure Storage. Moving the thread to concerned queue.

Regards,

Manu rekhar

Double post: https://social.msdn.microsoft.com/Forums/en-US/d7ebecc3-b5e2-42cf-913a-135e9df078a7/backup-failures-to-url?forum=sqldatabaseengine

since it is timeout check the size of db.

And try to compress if it is huge.

Mark as answer if it is useful

So we are using compression and we are running the backup job as SQL Server job. The first few dbs would be really fast and then at random db the throughput would get crazy low and then finally timeout.

I don't believe it's an issue with the size as we are seeing the same behavior on the smaller dbs.

Backup to multiple blobs in single backup operation is not supported.

Sorry I am having the same issue , where are the trace logs stored  ?

Hey David, am still encountering issue with this no matter what size of the db is and is intermittent and not related to Netowrk

To generate the detailed verbose logging for the backup  you can turn on TraceFlag 3051

I have turned on the trace flag , where are the trace logs stored , I checked the SQL server logs but dont see them.

It will not be stored in the SQL Error log but generates another log file in the errorlog directory and will be named BackupToUrl-SQLInstance-dbname.log

2015-02-16 09:36:18 AM: ======== BackupToUrl Initiated =========
2015-02-16 09:36:18 AM: Inputs: Backup = True, PageBlob= True, URI =  1000000D08C9DDF0115D1118C7A00C04FC297EB010000007B43096AA3F55D4E80B6BBCA1980504C000000001200000061007A007500720065006B00650079000000106600000001000020000000F05BA72EC5226B3EB91EB36A1BBDBD97BD603E1D8E902DF24196668A2FC2540F000000000E8000000002000020000000313AA7B0DE7B56D9806B51C03CCB964B30F13E640A33C012C26FE6F7A5B0178C5000000059C182C059366D70A121188E6844A5011126B5EDC62CF2611C146DD5B18900643BA008981BE917F47179050190515A39BCE22517A077EA16B33C6059F71BB8700C4680512213F56EAC6D96106E9E8EB440000000809F4FF77ADC598C9895FB4CF12E8AD31C3AA0DD905E6C7A1B71F39E8A3ADC190C2C94C7B8368B200B0D767B7958B4B9C0C9A750C24FBC2922130F718D9E85DA, FORMAT= False, Instance Name = MSSQLSERVER, DBName = Test File5 LogPath = C:\Program Files\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\Log
2015-02-16 09:36:18 AM: Process Id: 5176
2015-02-16 09:36:18 AM: Time for Initialization = 15,6032 ms
2015-02-16 09:36:18 AM: BackupToUrl Client is getting configuration from SqlServr
2015-02-16 09:36:18 AM: Time for Handshake and VDI config = 0 ms
2015-02-16 09:36:18 AM: Time for Get BlobRef = 0 ms
2015-02-16 09:36:21 AM: Time for Get Blob Exists = 2593,9113 ms
2015-02-16 09:36:22 AM: lease id is 3bf13dd1-83bf-45e8-938f-650d404902ee
2015-02-16 09:36:22 AM: Time for blob create = 1046,9962 ms
2015-02-16 09:36:22 AM: Read payload complete, final read Bytes Transferred = 0, Bytes Transferred from multiple read 0, Size requested 65536, current offset = 0, Duration = 500,0327 ms 
2015-02-16 09:36:23 AM: IOCompletion is processing asynchronous I/O completions for FLUSH
2015-02-16 09:39:23 AM: Payload: Start 0, Cmd Size 65536, Payload Size 65536, starttime 34583,3959658, endtime 34585,8863924, duration 2490,4266 ms,  attempts 1, Callback executed? True
2015-02-16 09:39:23 AM: HTTP status code 201, HTTP Status Message Created
2015-02-16 09:39:23 AM: Payload: Start 65536, Cmd Size 65536, Payload Size 65536, starttime 34583,4115267, endtime 34591,7618317, duration 8350,305 ms,  attempts 1, Callback executed? True
2015-02-16 09:39:23 AM: HTTP status code 201, HTTP Status Message Created
2015-02-16 09:39:23 AM: Payload: Start 4325376, Cmd Size 65536, Payload Size 65536, starttime 34583,4583975, endtime 34599,1116808, duration 15653,2833 ms,  attempts 1, Callback executed? True
2015-02-16 09:39:23 AM: HTTP status code 201, HTTP Status Message Created
2015-02-16 09:39:23 AM: Payload: Start 4390912, Cmd Size 65536, Payload Size 65536, starttime 34583,4583975, endtime 34607,5512258, duration 24092,8283 ms,  attempts 1, Callback executed? True
2015-02-16 09:39:23 AM: HTTP status code 201, HTTP Status Message Created
2015-02-16 09:39:23 AM: Payload: Start 4456448, Cmd Size 65536, Payload Size 65536, starttime 34583,4740247, endtime 34598,7366925, duration 15262,6678 ms,  attempts 1, Callback executed? True
2015-02-16 09:39:23 AM: HTTP status code 201, HTTP Status Message Created
2015-02-16 09:39:23 AM: An unexpected exception occurred during communication on VDI Channel.
2015-02-16 09:39:23 AM: Exception Info: Unrecoverable error occurred during Flush operation
2015-02-16 09:39:23 AM: Stack:    at Microsoft.SqlServer.VdiInterface.VDI.AsyncIOCompletion(BlobRequestOptions options, List`1 asyncResults, CloudPageBlob pageBlob, Boolean onFlush)
   at Microsoft.SqlServer.VdiInterface.VDI.PerformPageDataTransfer(CloudPageBlob pageBlob, AccessCondition leaseCondition, Boolean forBackup)
2015-02-16 09:39:23 AM: The Active queue had 0 requests until we got a clearerror
2015-02-16 09:39:23 AM: A fatal error occurred during Engine Communication, exception information follows
2015-02-16 09:39:23 AM: Exception Info: Unrecoverable error occurred during Flush operation
2015-02-16 09:39:23 AM: Stack:    at Microsoft.SqlServer.VdiInterface.VDI.PerformPageDataTransfer(CloudPageBlob pageBlob, AccessCondition leaseCondition, Boolean forBackup)
   at BackupToUrl.Program.MainInternal(String[] args)