Hi Guys, we have a Windows server 2008 DC with exchange server 2007 and backup exec 12 installed, when we try to backup the information stores we get an error saying the vss writers failed etc. when we run the command vssadmin list writers it confirms that the exchange vss writer is at a failed state. We have installed all the latest updates for both the server and exchange 2007 including sp1and update rollup 4 and this problem still comes up, we have been in contact with symantec and they suggest it is something to do with excange and not backup exec iself. any ideas on why the writers keep failing like this? Any help on this would be gratefully appreciated. Thanks Chris

Hello Chris, Can you post the error which you get with failed status? Post the output of VSSadmin list writers and any error in event log related to backup also. Additionally you can check below article for troubleshooting. Troubleshooting Exchange 2007 VSS Backups http://msexchangeteam.com/archive/2008/08/25/449684.aspx

Thanks for the reply, here is the error we get in backup exec: Backup- \\FAIRFIELD001\Microsoft Information Store\First Storage Group Writer Name: Microsoft Exchange Writer, Writer ID: {76FE1AC4-15F7-4BCD-987E-8E1ACB462FB7}, Last error: The VSS Writer failed, but the operation can be retried (0x800423f3), State: Failed during backup complete operation (12).V-79-57344-34110 - AOFO: Initialization failure on: "\\FAIRFIELD001\Microsoft Information Store\Second Storage Group". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).Snapshot provider error (0xE000FED1): A failure occurred querying the Writer status.Check the Windows Event Viewer for details.Writer Name: Exchange Server, Writer ID: {76FE1AC4-15F7-4BCD-987E-8E1ACB462FB7}, Last error: The VSS Writer failed, but the operation can be retried (0x800423f3), State: Stable (1).We also had ese errors about vss writers and backup exec errors about the vss writers in the system and app logs but we had to clear the files and do not have these anymore. thanks again Chris

Hello Chris, Are you getting same errors if you disable AOFO in BackupExec backup job? Check by disabling it.

Hi Thanks for the reply, we have never had the AOFO enabled on the backup jobs the options is always untickedThanks

Dear customer: In order to better troubleshoot the issue, please help collect the following information: 1. On Exchange server 2007, open Event Viewer, navigate to application, right click it and save it as .evtx file and send it to v-rocwan@microsoft.com. 2. On Exchange server 2007, open EMS, run the following command and post the result into the forum for further analyze: VSSadmin list writers vssadmin list providers If the Exchange writer is not in stable state (in Retryable error state as shown below) then try to restart the information store service on the Exchange server, and check Exchange writer is in stable state. 3. Did you enable CCR or LCR for the mailbox server? 4. Copy the commands below into notepad, and save the file as vsstracing.bat. Run the same to make the registry changes required to configure VSS tracing. net stop swprv reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceFile /t REG_SZ /d C:\trace.txt /f reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceLevel /t REG_DWORD /d 0xffffffff /f reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceEnterExit /t REG_DWORD /d 1 /f reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceToFile /t REG_DWORD /d 1 /f reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceToDebugger /t REG_DWORD /d 1 /f reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceFileLineInfo /t REG_DWORD /d 1 /f reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceForceFlush /t REG_DWORD /d 0 /f After running the batch file that enables tracing, we need to reproduce the issue. Once we have received the errors, stop the VSS services. You can do this by running the following commands on the command prompt. net stop VSS net stop SWPRV Then, disable VSS tracing by running the following command: reg delete HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /f The trace file is created on the root of the C drive. Send the trace file to me for analyze. For more information refer to http://support.microsoft.com/kb/887013 When you send e-mail to me, please let me know the subject of the post. Thanks for your cooperation. If anything is unclear, please feel free to let me know. Rock Wang - MSFT

I am having the same issue. Has a fix or workaround been found yet?Thanks,gary

Rock Wang Im having thesame problem with exchange server 2007installed on windows server 2003 R2 standard x64 sp2and backup exec 12.5 installed,I had installed all the latest updates for both the server and exchange 2007 including sp1and update rollup5 and this problem still comes up... any help would beappreciated. "vssadmin list writers"vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool(C) Copyright 2001 Microsoft Corp. Writer name: 'System Writer' Writer Id: {e8132975-6f93-4464-a53e-1050253ae220} Writer Instance Id: {902fb2cc-ab59-4999-a8c8-b085a6e71b4a} State: [1] Stable Last error: No error Writer name: 'IIS Metabase Writer' Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366} Writer Instance Id: {c5a4101a-a471-4b89-a311-416be743d093} State: [1] Stable Last error: No error Writer name: 'WMI Writer' Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0} Writer Instance Id: {000d9b56-a299-4ad7-8d7b-df5ff7e4ac16} State: [1] Stable Last error: No error Writer name: 'Microsoft Exchange Writer' Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7} Writer Instance Id: {4f600a43-f671-4e2e-a2c6-4b6cf68da421} State: [9] Failed Last error: No error Writer name: 'MSDEWriter' Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277} Writer Instance Id: {53044a55-b21f-461c-a1e1-f2f3ab818dd1} State: [1] Stable Last error: No error Writer name: 'Registry Writer' Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485} Writer Instance Id: {cb7d4534-f8d9-4550-a784-54c0fa344a7a} State: [1] Stable Last error: No error Writer name: 'Event Log Writer' Writer Id: {eee8c692-67ed-4250-8d86-390603070d00} Writer Instance Id: {58e3c06c-cd45-4af8-92d2-f8a35cf91d6c} State: [1] Stable Last error: No error Writer name: 'COM+ REGDB Writer' Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f} Writer Instance Id: {3f242b16-7267-44bb-ae8c-95965886f049} State: [1] Stable Last error: No error "vssadmin list providers"vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool(C) Copyright 2001 Microsoft Corp. Provider name: 'Microsoft Software Shadow Copy provider 1.0' Provider type: System Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5} Version: 1.0.0.7 " Thanks, Moe

I'm having the same issue with my Vss 'Microsoft Exchange Writer' failing.I'm currently running Exchange 2007 with Server 2008 SBS x64bit.Symantec say Microsoft need to create a hotfix as they did for Server 2003.Thanks,

G'day MusicalChris,First, I'd be looking at what type of HDD subsystem you have. Do you have a RAID-1, RAID-10 or RAID-5 array, and how many drives in the array? Also, which RAID controller are you running this off, how much RAM does it have and do you have this RAM battery-backed?Depending on the HDD subsystem in place, you may find that the Microsoft Exchange Writer (the VSS Writer) isn't able to complete its operation before the RAID array has been completely snapshotted.- Hilton TravisSBSC PAL (Australia) FY 08-09

I am facing the same exact problem.The environment :-=============We have a domain with a SBS 2008 server and a additional windows 2008 DC.SBS 2008 server is not updated deliberately with Service pack 2 for 2008 server systems but it is upto date with all the updates till date.We have Backup Exec 12.5 for windows servers installed on windows 2008 64-bit DC from where i try to take a backup of the SBS server.The output of vssadmin list writers shows the exchnage writer failed.All the other writers are stable. Only the exchnage writer plays up.Writer name: 'Microsoft Exchange Writer' Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7} Writer Instance Id: {202166d5-7fbe-4816-86a7-64a2d82b6e91} State: [9] Failed Last error: Retryable errorOutput for vssadmin list providersvssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool(C) Copyright 2001-2005 Microsoft Corp.Provider name: 'Microsoft Software Shadow Copy provider 1.0' Provider type: System Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5} Version: 1.0.0.7Once i reboot the SBS 2008 server this error goes away.But this stable state of the Exchange writer stays only till the backup job is tried again. I have checked the symantec forums and they say that this had been an issue on the MS side with the VSS . This existed in windows 2003 as well for which we have the hotfix KB940349. http://support.microsoft.com/kb/940349.But i dont think this could still be causing the issue as i expect this hotfix to be included in the windows 2008. So i did not bother downloading and installing it.Is there any new hotfix for this. Please let me know a way to fix this VSS error. permanently.Itried the steps above and the trace file has the following section. 0039607187,0x002a7c:0x18f4:0x0a3a578b] modules\tracing\vs_trace.cxx(0412): ****************************************************************[0039607203,0x002a7c:0x18f4:0x0a3a578b] modules\tracing\vs_trace.cxx(0414): ** TRACING STARTED - ProcessId: 0x2a7c, ContextId: 0xa3a578b [0039607218,0x002a7c:0x18f4:0x0a3a578b] modules\tracing\vs_trace.cxx(0425): ** Current time: 09/09/09 17:45:47[0039607249,0x002a7c:0x18f4:0x0a3a578b] modules\tracing\vs_trace.cxx(0426): ** Command-line: C:\Windows\system32\vssvc.exe[0039607265,0x002a7c:0x18f4:0x0a3a578b] modules\tracing\vs_trace.cxx(0427): ** VSS SKU ID: 0x02[0039607281,0x002a7c:0x18f4:0x0a3a578b] modules\tracing\vs_trace.cxx(0428): **************************************************************** [0039607312,0x002a7c:0x18f4:0x0a3a578b] ENTER {CVsServiceModule::CVsServiceModule}: [0039607327,0x002a7c:0x18f4:0x0a3a578b] EXIT {CVsServiceModule::CVsServiceModule}: hr: 0x00000000[0039607359,0x002a7c:0x18f4:0x0a3a578b] ENTER {CVssCriticalSection::CVssCriticalSection}: [0039607374,0x002a7c:0x18f4:0x0a3a578b] EXIT {CVssCriticalSection::CVssCriticalSection}: hr: 0x00000000[0039607390,0x002a7c:0x18f4:0x0a3a578b] ENTER {CVssCriticalSection::CVssCriticalSection}: [0039607405,0x002a7c:0x18f4:0x0a3a578b] EXIT {CVssCriticalSection::CVssCriticalSection}: hr: 0x00000000[0039607421,0x002a7c:0x18f4:0x0a3a578b] ENTER {CVssCriticalSection::CVssCriticalSection}: [0039607437,0x002a7c:0x18f4:0x0a3a578b] EXIT {CVssCriticalSection::CVssCriticalSection}: hr: 0x00000000[0039607468,0x002a7c:0x18f4:0x0a3a578b] ENTER {CVsServiceModule::_WinMain}: [0039607483,0x002a7c:0x18f4:0x0a3a578b] coord\src\svc.cxx(1036): CVsServiceModule::_WinMain: Trace: VSS command-line: 'C:\Windows\system32\vssvc.exe' [0039607499,0x002a7c:0x18f4:0x0a3a578b] ENTER {CVsServiceModule::StartDispatcher}: [0039607530,0x002a7c:0x1834:0x0a3a578b] ENTER {CVsServiceModule::ServiceMain}: [0039607546,0x002a7c:0x1834:0x0a3a578b] ENTER {CVsServiceModule::SetServiceStatus}: [0039607577,0x002a7c:0x1834:0x0a3a578b] coord\src\svc.cxx(0494): CVsServiceModule::SetServiceStatus: Attempt to change the service status to 2 [0039607593,0x002a7c:0x1834:0x0a3a578b] EXIT {CVsServiceModule::SetServiceStatus}: hr: 0x00000000[0039607608,0x002a7c:0x1834:0x0a3a578b] ENTER {CVsServiceModule::OnInitializing}: [0039607624,0x002a7c:0x1834:0x0a3a578b] ENTER {CVssSidCollection::Initialize}: [0039607655,0x002a7c:0x1834:0x0a3a578b] ENTER {CVssSidCollection::AddWellKnownSid}: [0039607671,0x002a7c:0x1834:0x0a3a578b] ENTER {CVssSidCollection::AddSid}: [0039607686,0x002a7c:0x1834:0x0a3a578b] modules\sec\security.cxx(2199): CVssSidCollection::AddSid: Adding SID S-1-5-32-544 ...[0039607717,0x002a7c:0x1834:0x0a3a578b] ENTER {CVssSidCollection::VerifyIsLocal}: [0039607733,0x002a7c:0x1834:0x0a3a578b] EXIT {CVssSidCollection::VerifyIsLocal}: hr: 0x00000000[0039607749,0x002a7c:0x1834:0x0a3a578b] modules\sec\security.cxx(2258): CVssSidCollection::AddSid: Adding SID: BUILTIN\Administrators [0039607764,0x002a7c:0x1834:0x0a3a578b] EXIT {CVssSidCollection::AddSid}: hr: 0x00000000[0039607780,0x002a7c:0x1834:0x0a3a578b] EXIT {CVssSidCollection::AddWellKnownSid}: hr: 0x00000000[0039607795,0x002a7c:0x1834:0x0a3a578b] ENTER {CVssSidCollection::AddWellKnownSid}: [0039607827,0x002a7c:0x1834:0x0a3a578b] ENTER {CVssSidCollection::AddSid}: [0039607842,0x002a7c:0x1834:0x0a3a578b] modules\sec\security.cxx(2199): CVssSidCollection::AddSid: Adding SID S-1-5-18 ...[0039607858,0x002a7c:0x1834:0x0a3a578b] ENTER {CVssSidCollection::VerifyIsLocal}: [0039607873,0x002a7c:0x1834:0x0a3a578b] EXIT {CVssSidCollection::VerifyIsLocal}: hr: 0x00000000[0039607905,0x002a7c:0x1834:0x0a3a578b] modules\sec\security.cxx(2258): CVssSidCollection::AddSid: Adding SID: NT AUTHORITY\SYSTEM [0039607920,0x002a7c:0x1834:0x0a3a578b] EXIT {CVssSidCollection::AddSid}: hr: 0x00000000[0039607936,0x002a7c:0x1834:0x0a3a578b] EXIT {CVssSidCollection::AddWellKnownSid}: hr: 0x00000000[0039607951,0x002a7c:0x1834:0x0a3a578b] ENTER {CVssSidCollection::AddWellKnownSid}: [0039607967,0x002a7c:0x1834:0x0a3a578b] ENTER {CVssSidCollection::AddSid}: [0039607983,0x002a7c:0x1834:0x0a3a578b] modules\sec\security.cxx(2199): CVssSidCollection::AddSid: Adding SID S-1-5-32-551 ...[0039608014,0x002a7c:0x1834:0x0a3a578b] ENTER {CVssSidCollection::VerifyIsLocal}: [0039608029,0x002a7c:0x1834:0x0a3a578b] EXIT {CVssSidCollection::VerifyIsLocal}: hr: 0x00000000[0039608045,0x002a7c:0x1834:0x0a3a578b] modules\sec\security.cxx(2258): CVssSidCollection::AddSid: Adding SID: BUILTIN\Backup Operators [0039876569,0x002b40:0x1130:0xb14b3ee4] EXIT {CXMLDocument::Next}: hr: 0x00000000[0039876585,0x001834:0x2ac4:0xac4ba69c] ENTER {LoadResourceString}: [0039876601,0x002b40:0x1130:0xb14b3ee4] ENTER {CXMLDocument::IsNodeMatch}: [0039876616,0x001834:0x2ac4:0xac4ba69c] ENTER {CVssAutoString::LoadString(HMODULE)}: [0039876632,0x002b40:0x1130:0xb14b3ee4] EXIT {CXMLDocument::IsNodeMatch}: hr: 0x00000000[0039876663,0x001834:0x1570:0xac4ba69c] backupext\vsxml\vs_cmxml.cxx(8869): CVssBackupComponents::ExposeWriterMetadata: Adding context: 'Writer Class Id' = '{76FE1AC4-15F7-4BCD-987E-8E1ACB462FB7}' (2) [0039876663,0x002b40:0x1130:0xb14b3ee4] EXIT {CXMLDocument::FindElement}: hr: 0x00000000[0039876694,0x001834:0x1570:0xac4ba69c] EXIT {CVssFunctionTracer::AddContextInternal}: hr: 0x00000000[0039876725,0x002b40:0x1130:0xb14b3ee4] ENTER {CXMLDocument::FindAttribute}: [0039876741,0x001834:0x1570:0xac4ba69c] ENTER {LoadResourceString}: [0039876757,0x002b40:0x27f0:0xb14b3ee4] EXIT {CAutoSid::operator==}: hr: 0x00000000[0039876772,0x001834:0x1570:0xac4ba69c] ENTER {CVssAutoString::LoadString(HMODULE)}: [0039876788,0x002b40:0x27f0:0xb14b3ee4] (0000): CVssDeletingWriter::IsSystemAccountSID: Returning bool: TRUE[0039876803,0x001834:0x1570:0xac4ba69c] EXIT {CVssAutoString::LoadString(HMODULE)}: hr: 0x00000000[0039876803,0x002b40:0x298c:0xb14b3ee4] EXIT {CXMLDocument::IsNodeMatch}: hr: 0x00000000[0039876835,0x001834:0x1570:0xac4ba69c] EXIT {LoadResourceString}: hr: 0x00000000[0039876850,0x002b40:0x298c:0xb14b3ee4] ENTER {CXMLDocument::Next}: [0039876866,0x001834:0x1570:0xac4ba69c] ENTER {CVssFunctionTracer::AddContextInternal}: [0039876881,0x002b40:0x298c:0xb14b3ee4] EXIT {CXMLDocument::Next}: hr: 0x00000000 [0039876913,0x001834:0x1570:0xac4ba69c] backupext\vsxml\vs_cmxml.cxx(8870): CVssBackupComponents::ExposeWriterMetadata: Adding context: 'Writer Name' = 'Microsoft Exchange Writer' (2) [0039876913,0x002b40:0x298c:0xb14b3ee4] ENTER {CXMLDocument::IsNodeMatch}: [0039876944,0x001834:0x1570:0xac4ba69c] EXIT {CVssFunctionTracer::AddContextInternal}: hr: 0x00000000[0039876959,0x002b40:0x1130:0xb14b3ee4] EXIT {CXMLDocument::FindAttribute}: hr: 0x00000000[0039876975,0x001834:0x1570:0xac4ba69c] backupext\vsxml\vs_cmxml.cxx(8872): CVssBackupComponents::ExposeWriterMetadata: WriterId = [0039876991,0x002b40:0x1130:0xb14b3ee4] ENTER {CXMLDocument::FindAttribute}: [0039877022,0x001834:0x1570:0xac4ba69c] backupext\vsxml\vs_cmxml.cxx(8873): CVssBackupComponents::ExposeWriterMetadata: WriterName = Microsoft Exchange Writer [0039877022,0x002b40:0x1130:0xb14b3ee4] EXIT {CXMLDocument::FindAttribute}: hr: 0x00000000[0039877037,0x001834:0x1570:0xac4ba69c] backupext\vsxml\vs_cmxml.cxx(8874): CVssBackupComponents::ExposeWriterMetadata: InstanceId for Writer {202166D5-7FBE-4816-86A7-64A2D82B6E91} is [0039877053,0x002b40:0x1130:0xb14b3ee4] ENTER {CXMLDocument::FindElement}: [0039877084,0x001834:0x25e4:0xac4ba69c] backupext\vsxml\vs_cmxml.cxx(8874): CVssBackupComponents::ExposeWriterMetadata: InstanceId for Writer {02763B2C-7753-4D20-8DD5-5C157F728EF8} is [0039877084,0x002b40:0x1130:0xb14b3ee4] ENTER {CXMLDocument::Next}: [0039877115,0x001834:0x048c:0xac4ba69c] ENTER {CVssAutoString::LoadString(HMODULE)}: [0039877115,0x002b40:0x27f0:0xb14b3ee4] EXIT {CVssDeletingWriter::IsSystemAccountSID}: hr: 0x00000000[0039877147,0x001834:0x048c:0xac4ba69c] EXIT {CVssAutoString::LoadString(HMODULE)}: hr: 0x00000000[0039877162,0x002b40:0x27f0:0xb14b3ee4] ENTER {CVssRegistryKey::Close}: [0039877178,0x001834:0x048c:0xac4ba69c] EXIT {LoadResourceString}: hr: 0x00000000[0039877193,0x002b40:0x27f0:0xb14b3ee4] EXIT {CVssRegistryKey::Close}: hr: 0x00000000 The HDD on which SBS 2008 is installed is not a RAID array. Its on a simple SATA disk.Any replies would be appreciated.Thanks Shashi

Hi, Just to let you know I had exactly same configuration and problem. I disabled Local Continuous protection on the storage groups and my backup now works fine! With LCP running I just could not get it working. And backup is more important than LCP in my view. I hope this helps Vladimir

This worked for me... Also my Data rates went from 3-500 to over 2000 my backup times went from about 2hrs to about 45mins..Thanks for the info! Do we know if this is fixed in newer versions of Backupexec? Currently running 11D

Another user with the same problems ... Windows Server 2008, Exchange 2007. I upgraded Backup Exec (running on a different server) to BE2010, and have been getting VSS errors every night. Exchange 2007 was on SP1, rollup4, I upgraded it to rollup 11 and that didn't fix it, so I upgraded to SP2, rollup 1. Same problems. Symantec says it's all Microsoft's fault, naturally. Installed Windows Server Backup and ran a backup of the Exchange server, got the same errors: "The log file ...\first storage group\E00000015868.log is missing (error -528) and cannot be used." When you click on the "for more information link" you get the old familiar 'page not found' error. I ran 'vssadmin list writers' and they all reported no error. I went through the rest of the steps listed above (http://msexchangeteam.com/archive/2008/08/25/449684.aspx)and running the BETESTtool failed, with event ID 32 and 8193. An hour crawling the wwweb found zero useful information on either of those errors. Any recourse?tom

Have you tried to increase the default timeout period for the VSS writers? I increased mine to 20 from the default of 10 and it helped me. I was using netbackup 7 to backup exchange 2010. installed server backup and started a job which failed with the same scenario, VSS exchange writer in retryable state. increased timeout period and backup using MS server backup worked correctly. Still working on Netbackup now, it may work now as well just havent tested. hope this helps.

Hi, i have the same Problem with Windows Small Business Server 2011 and Backup Exec 2010 R3. How can i increase the VSS Timeout Period? regards Oliver

Hi All, we have found a solution for the problem. Backup Exec System Recovery is aparently using another Snapshot Driver and generates this error. We deinstalled the Backup Exec System Recovery and the Backup works. regards Oliver

Oliver What version of the Backup Exec System Recovery do you have that you uninstall. Thank you.

Hi C7oi, Symantec Backup Exec System Recovery 2010