High RPC Average Latency on Exchange Mailbox server
I've run Perfmon on my Exchange 2007 Mailbox server for 24 hours and the results show that the RPC Average Latency is about 80ms on average. I understand this should be 50ms at most.
How can I find out what's causing this?
January 21st, 2011 4:03pm
On Fri, 21 Jan 2011 20:56:08 +0000, Chuck_P2101 wrote:
>I've run Perfmon on my Exchange 2007 Mailbox server for 24 hours and the results show that the RPC Average Latency is about 80ms on average. I understand this should be 50ms at most. How can I find out what's causing this?
What other counters hae you collected?
A guess would be your storage system.
---
Rich Matheisen
MCSE+I, Exchange MVP
--- Rich Matheisen MCSE+I, Exchange MVP
Free Windows Admin Tool Kit Click here and download it now
January 21st, 2011 9:41pm
Greetings Rich,
I haven't collected any other counters yet, what sort of counters should I be looking at in your opinion, especially to find out whether it's storage related?
January 22nd, 2011 7:23am
On Sat, 22 Jan 2011 12:16:12 +0000, Chuck_P2101 wrote:
>
>
>Greetings Rich,
>
>I haven't collected any other counters yet, what sort of counters should I be looking at in your opinion, especially to find out whether it's storage related?
Use this to analyze the perfmon log files (you'll need a couple of
prerequisites, too):
http://pal.codeplex.com/
Use this to set up the counters and gather the information:
http://blogs.technet.com/b/mikelag/archive/2010/07/09/exchange-2007-2010-performance-data-collection-script.aspx
---
Rich Matheisen
MCSE+I, Exchange MVP
--- Rich Matheisen MCSE+I, Exchange MVP
Free Windows Admin Tool Kit Click here and download it now
January 22nd, 2011 11:39am
Chuck_P2101:I've run Perfmon on my Exchange 2007 Mailbox server for 24 hours and the results show that the RPC Average Latency is about 80ms on average. I understand this should be 50ms at most. How can I find out what's causing this?
MSExchangeIS\RPC Averaged Latency should not be higher than 25 ms on average.
Monitoring Without System Center Operations Manager
http://technet.microsoft.com/en-us/exchange/bb201720(EXCHG.80).aspx
This document details the most important counters to monitor and their threshold values. I've used it quite a lot in one environment where we don't have SCOM 2007 and monitoring of Exchange with HP OpenView isn't properly implemented yet. It should guide
you to track down other potential issues and the root cause.
Personally I prefer using SCOM 2007, which I've been so lucky to implement at a customer and has developed to become one of my late obsessions. Btw, should you happen to use BlackBerry devices?MCTS: Messaging | MCSE: S+M
January 22nd, 2011 6:00pm
> MSExchangeIS\RPC Averaged Latency should not be higher than 25 ms on average.
Thanks Jon-Alfred. I was reading from
http://technet.microsoft.com/en-us/library/aa998266(EXCHG.80).aspx where it stated that 50ms was the general number. The counter I was using was "MSExchangeIS\RPC Averaged
Latency". Yet in the link you provided, it does indeed say that the counter should be 25ms or less, and it's the "MSExchangeIS Client (*)\RPC Average Latency" (which shows the breakdown by client type) that can be 50ms. Which begs the questions
- Which counter should I be using?
- Surely the two counters mean the same thing, but the latter one shows by client? I don't understand how the two values can be different if that *is* the case?
Many thanks,
Free Windows Admin Tool Kit Click here and download it now
January 23rd, 2011 5:53am
On Sun, 23 Jan 2011 10:46:22 +0000, Chuck_P2101 wrote:
>
>
>> MSExchangeIS\RPC Averaged Latency should not be higher than 25 ms on average.
>
>Thanks Jon-Alfred. I was reading from http://technet.microsoft.com/en-us/library/aa998266(EXCHG.80).aspx where it stated that 50ms was the general number. The counter I was using was "MSExchangeIS\RPC Averaged Latency". Yet in the link you provided, it
does indeed say that the counter should be 25ms or less, and it's the "MSExchangeIS Client (*)\RPC Average Latency" (which shows the breakdown by client type) that can be 50ms. Which begs the questions
>
>- Which counter should I be using?
That depends on what you're measuring. If "MSExchangeIS\RPC Averaged
Latency" is under 25ms then the server's doing okay -- but you're
measuring ALL RPC operations and reporting the *average*. If there a
gazillions of operations that run on the server that aren't associated
with *users* they may complete very quickly and the average value will
be low. That doesn't mean the individual protocols aren't a problem,
or that there's not a problem with really large mailboxes and users
working in non-cached mode casuing problems.
>- Surely the two counters mean the same thing, but the latter one shows by client?
By client *protocol*, not be individual user.
>I don't understand how the two values can be different if that *is* the case?
Does this help?
http://technet.microsoft.com/en-us/library/bb201689(EXCHG.80).aspx
---
Rich Matheisen
MCSE+I, Exchange MVP
--- Rich Matheisen MCSE+I, Exchange MVP
January 23rd, 2011 12:13pm
So the initial Perfmon I ran was using the "MSExchangeIS\RPC Averaged Latency" counter and this showed an average value of 80ms.
Bearing in mind your comment that this is only an average for ALL RPC operations, would you actually recommend we use this counter at all? Or should we be using the ISClient one instead? We don't run ActiveSync, IMAP, POP etc, only OWA and MAPI. Should I
be running MSExchangeIS Client(*)\RPC Operations/sec as well for example to show which client is hitting the mailbox server the hardest?
Basically, I'm not sure if I am using the correct Perfmon counter and if I actually have a problem or not!
Free Windows Admin Tool Kit Click here and download it now
January 23rd, 2011 12:58pm
On Sun, 23 Jan 2011 17:51:50 +0000, Chuck_P2101 wrote:
>So the initial Perfmon I ran was using the "MSExchangeIS\RPC Averaged Latency" counter and this showed an average value of 80ms.
If it's that high you have a problem. You just don't know where it is.
That's why I gave you references to PAL and PerfWiz. You can't
troubleshoot performance problems in a vacuum.
>Bearing in mind your comment that this is only an average for ALL RPC operations, would you actually recommend we use this counter at all? Or should we be using the ISClient one instead? We don't run ActiveSync, IMAP, POP etc, only OWA and MAPI. Should
I be running MSExchangeIS Client(*)\RPC Operations/sec as well for example to show which client is hitting the mailbox server the hardest?
>
>Basically, I'm not sure if I am using the correct Perfmon counter and if I actually have a problem or not!
You have a problem. Use PerfWiz to set up the necessary trace logging
and PAL to generate the reports. Fix the stuff that PAL flags.
FYI, using ExPBA (it's in the EMC "Toolbox") is something you should
do, too. Fix the problems it points out.
---
Rich Matheisen
MCSE+I, Exchange MVP
--- Rich Matheisen MCSE+I, Exchange MVP
January 23rd, 2011 1:52pm
Will do.
Going forward, when using Perfmon to analyse Exch performance, is that a valid counter to use, or should we concentrate on the "MSExchangeIS Client (*)\RPC Average Latency' one?
Free Windows Admin Tool Kit Click here and download it now
January 23rd, 2011 3:41pm
On Sun, 23 Jan 2011 20:34:27 +0000, Chuck_P2101 wrote:
>Going forward, when using Perfmon to analyse Exch performance, is that a valid counter to use, or should we concentrate on the "MSExchangeIS Client (*)\RPC Average Latency' one?
"MSExchangeIS\RPC Averaged Latency" if you're interested in the
overall health of the server. But looking at both isn't a bad idea.
---
Rich Matheisen
MCSE+I, Exchange MVP
--- Rich Matheisen MCSE+I, Exchange MVP
January 23rd, 2011 3:49pm