CAS server and OWA gradually become slow
Hello,We have installed 2 CAS servers in our production exchange 2003 coexistance enviornment. The servers are runningServer 2008, Exchange 2007 SP1 and Rollup 6 has been applied.NLB has been setup and a SAN cert has been installed. For troublshooting purposes I have stopped the NLB for one of the servers so that we are only connecting to a single server. We have upgraded all of the firmware and NIC drivers. We have also setup all exclusions and even shut off Forefront while testing this issue to rule out AV.The problem is thatOWA works veryfast at first but thenit gradually slows down. If we reboot theCAS server it works very fast(almost as fast as our 2003 front end servers). However later in the day it willstart taking up to 20 seconds to populate the page after authentication. Recycling theapplication pools seems to speed things up again as well but a reboot definetely speeds it back up.Since we only have CAS servers at the moment we are only connecting to our 2003 mailboxes. We have tested accessing owa on the backend server directly and always get a fast connecton.Thanks,SteveJoe
February 17th, 2009 6:28pm
Hi Steve,
Did you run the best practice analyzer tool to check your system ? Besides this have a look with perfmon what happens when it works OK and when it slows down.
Regards,
Johanblog: www.johanveldhuis.nl
Free Windows Admin Tool Kit Click here and download it now
February 17th, 2009 8:29pm
Thanks for the reply,We have run the ExBPA and do not get any errors or additional info. I will look into running perfmon. It looks like perfmon has changed quite a bit in 2008. Any particular settings that we should run for troublshooting this?Thanks,SteveJoe
February 17th, 2009 9:50pm
Hi,Check for memory usage and virtual memory usage. Do you use RPC over HTTP if so please check this white paper from Microsoft:
http://technet.microsoft.com/en-us/library/cc540453.aspxRegards,Johanblog: www.johanveldhuis.nl
Free Windows Admin Tool Kit Click here and download it now
February 17th, 2009 10:36pm
Though this system is part of the production enviornment we are only using it for testing before replacing our 2003 OWA servers with them. We do not have anyone connecting with Outlook Anywhere. We have only gotten as far as testing OWA access and it is slow enough that we cant move forward until this is resolved. Only 2 of us are connecting to the server at any one time and I am not seeing any errors or warnings running perfmon. The first couple of times we logon are fast. After that it gets slower and slower. If we recycle the MSExchangeOWAAppPool we will get fast access again temporarily.Thanks,SteveJoe
February 17th, 2009 11:13pm
Hi,
I think we can compare netmon log from below three scenario to get four netmon logs.
1. Please enable another CAS server. And then type https://NLBvirtualname/OWA
2. Please type https://CASservername/OWA. (Both CAS server1 and CAS server 2).
3. Please access OWA from BE.
Besides, we need to narrow down the root cause of the issue to see if it is a IIS server related one or Network related one.
1. Please re-start IIS and then access OWA.If everytime you re-start IIS server,speed to access OWA get fast,then we assume it could be a IIS server issue.
2. Please follow steps in article to improve the performance of IIS.
3. Please check NLB settings via document How to load-balance Exchange 2007 Client Access Servers (CAS) using Windows Network Load Balancing (WNLB) technology.
4. Please Monitor the Memory | Pool Nonpaged Bytes performance counter on CAS server.( Nonpaged Pool is over the warning threshold How to use Memory Pool Monitor (Poolmon.exe) to troubleshoot kernel mode memory leaks)
5. Please follow the steps in article below to check network settings.
You may experience network-related problems after you install Windows Server 2003 SP2 or the Scalable Networking Pack on a Windows Small Business Server 2003-based computer that has an advanced network adapterhttp://support.microsoft.com/default.aspx?scid=kb;EN-US;936594
More information to share with you:
No Microsoft Windows Server Multipath I/O (MPIO) disks or logical unit numbers (LUNs) are available when you use MPIO for disks that are presented by an iScsi Host Bus Adapter (HBA) in Microsoft Windows Server 2003 SP1
http://support.microsoft.com/kb/923801
Regards,
Xiu
Free Windows Admin Tool Kit Click here and download it now
February 19th, 2009 10:38am
Thanks for the reply xiu,We are running netmon now. We have removed NLB so that we can rule out any slowness from NLB.We are testing accessing the https://CASservername/Exchange since we only have 2003 mailbox server at this time.Accessing OWA from BE is very fast all of the time from all servers and workstations.We do believe that this is more an issue with IIS7. Everytime we restart IIS or the OWA app pool on the CAS server it is fast for the first attempt.We did notice today that it has been slow every other attempt or so. Watching the memory for w3wp.exe does not go up or move much when access is fast. It will jump up about 5MB to 10MB when access is slow.We have investigated the Scalable Networking Pack issue and do not think it is an issue for us since we haveTOE disabled.I cannot find any information for running poolmon on Server 2008.Any ideas?Thanks,Steve
Joe
February 19th, 2009 11:46pm
Is it possible that another piece of software is running on the CAS server and possibly eating up the memory? Use performance counters to check memory and page file usage. Dont forget to look at disk and processor usage which I believe shows up as default when you open perfmon anyway. Second, is there plenty of space left on the harddrives? If not, the OS may have a hard time keeping up after being up for so long, specifically with pagefile usage.Also, if you find the the server is starting to be slow, keep the session open andtry using another client computer at the same time. If you use another client computer and it is fast again, there may be something wrong with your client computer such as the browser not cleaning up resources properly. Also, even though you are only using it as a test with not many users, can we assume you have the recommended amount of hardware resources in the server?
Free Windows Admin Tool Kit Click here and download it now
February 19th, 2009 11:58pm
These are new servers that only have Exchange 2007 requirements running. The hard drives have over 100GB available. We can reboot this server and get the same results without the server being up for very long. We have been trying from multiple computers.This is production equipment.Dell 2950 Dual 3GHZ and 16GB of memory.We have just not pointed our URL to it yet because of the slowness. These 2 servers will replace the existing 2003 OWA servers.Joe
February 20th, 2009 12:13am
Hi,
First Id like to explain that davex contains logic for handling DAV requests, redirecting Exchange 2007 mailbox users to /owa, and serving up the Exchange 2003/2000 OWA experience. So please try to reload davex.dll to solve the problem.
1. Stopped the default web site2. Renamed davex.dll to davex.dll.old(c:\Program Files\Microsoft\Exchange Server\bin)3. Started the default web site4. IIS restarted(iisreset /noforce)
Note: Please check Anti-Virus Software to see whether it has killed w3wp.exe.
If it is not the issue, then I think we may try to backup IIS7 configuration and then re-install IIS 7 and then re-install CAS.
Related article to share with you:
How to create and manage configuration backups in Internet Information Services 7.0
http://support.microsoft.com/kb/954872
How to Install Exchange 2007 SP1 Prerequisites on Windows Server 2008 or Windows Vista
http://technet.microsoft.com/en-us/library/bb691354.aspx
Reset OWA related Virtual Directory
http://support.microsoft.com/kb/941201/en-us
Regards,
Xiu
Free Windows Admin Tool Kit Click here and download it now
February 20th, 2009 9:34am
We do not have the davex.dll on the CAS server since this is CAS only.We will rebuild this first CAS server and see if the problem exists prior to rollups and such.
Steve
February 20th, 2009 7:09pm
FYI, we did find our problem . A firewall rule was blocking ICMP between the domain controllers and the new CAS servers. We never saw any drops because we were only using netmon on the CAS servers and seeing no issues. When we asked the firewall admins to monitor during a logon.... the only drops theynoticedpings from DC's to CAS. Opening ICMP resolved the issue.Though we were always able toconnectit just took an additional 15 seconds during authentication and the memory kept growing fast on the OWA app pool with the increased requests we are guessing.-SteveJoe
Free Windows Admin Tool Kit Click here and download it now
March 4th, 2009 5:42am
Thanks for your feedback. :)
March 4th, 2009 9:55am