Hi From what I understand, the Exchange checkpoint depth in Exchange 2003 is 20MB. The definition of checkpoint depth, as I understand it, is "The maximum amount of data that the Extensible Storage Engine can write to logs before it writes to the database is known as the log checkpoint depth." At the same time, Exchange will only allow 1008 log files to be generated that aren't written to the database, before dismounting the store. Sources: http://theessentialexchange.com/blogs/michael/archive/2008/04/28/ESE-Checkpoint-Depth.aspx http://support.microsoft.com/kb/886298 But the two figures don't match? Surely a checkpoint depth of 20MB means 4 log files in Exchange 2003? Does this also mean that in any given SG, the max number of log files that haven't had their data written to the store is 4MB?? *confused* [We are running Exchange 2003 SP2

It's explained a bit here too http://msexchangeteam.com/archive/2006/09/08/428860.aspx Checkpoint depth of 20MB (per storage group) means it will wait for 20MB worth of log files before writing to the database file on the disk. If you try to manually dismount the database(s) in the storage group, it would take a bit longer because it needs to write those logs to the database. Remember, the log files are still safe on disk (well - you unless your storage controller is playing tricks on you) so it's not like you are relying only on RAM for this 20MB in a power failure situation. The 1008 number is checkpoint depth *exhaustion* which is the maximum that Exchange will allow. Most commonly you hit this number if you have a backup running for a really long time (or if a backup gets stuck) - because log files stop being committed during a backup. You can monitor the number of uncommitted logs in Performance Monitor (Perfmon) - Create a counter for Database ==> Instances\Log Generation Checkpoint Depth . You may see that high IO (for example, moving mailboxes) pushes you above the 20MB (4 logs in Exchange 2003) mark.

On Tue, 11 Jan 2011 22:39:54 +0000, Pancamo wrote: >Hi From what I understand, the Exchange checkpoint depth in Exchange 2003 is 20MB. The definition of checkpoint depth, as I understand it, is "The maximum amount of data that the Extensible Storage Engine can write to logs before it writes to the database is known as the log checkpoint depth." At the same time, Exchange will only allow 1008 log files to be generated that aren't written to the database, before dismounting the store. Sources: http://theessentialexchange.com/blogs/michael/archive/2008/04/28/ESE-Checkpoint-Depth.aspx http://support.microsoft.com/kb/886298 But the two figures don't match? Surely a checkpoint depth of 20MB means 4 log files in Exchange 2003? Does this also mean that in any given SG, the max number of log files that haven't had their data written to the store is 4MB?? *confused* [We are running Exchange 2003 SP2 The KB article applies to Exchange 2000. The checkpoint depth of 1008 log files (in any one storage group) is correct for Exchange 2003. --- Rich Matheisen MCSE+I, Exchange MVP --- Rich Matheisen MCSE+I, Exchange MVP

There are two different numbers here. -The 20MB (which Microsoft recommends dropping to 5MB for a 2000 or 2003 *clustered* environment ... might even be the default for 2003... as per http://technet.microsoft.com/en-us/library/aa996874(EXCHG.80).aspx) * Number that controls when ESE will start writing to the database. This is a performance tuning option and not a hard limit. -The 1008 *maximum number of uncommitted log files before it shuts down (hard coded)

Interesting topic... I ran into this problem recently. We have an Exchange 2003 SP2 server, had dismounted its storage twice in recent months. Got error MSExchangeIS 1159. Typically indicate we have too many uncommitted transaction logs. Checked the database, we have over 10 15GB mailboxes and 30 4GB mailboxes and the database size is over 200GB and log folder was like, WOW, got 800 log files in the middle of the day already. So I concluded we have too many big mailboxes which generates too many transaction logs... So we did a cleanup to remove a few those big ones. Everything was fine after that, recently I jumped on the server and checked the log folder, WOW! There are over 2000 transaction log files there! And yet the Exchange 2003 server is running fine... So what's the deal? I guess the way I check transaction log files are wrong. The number of uncommitted logs cannot be decided by simply count the transaction log files in the log folder! Then my question is: What is the correct method to check how many transaction log files have not been committed? Anyone? Cant seems find a document for that...

Thanks all, so basically with Exchange 2003, there will *always* be *at least* a 4 log delay between a transaction being written to a log file and being written to the database? Does anyone know what this figure is for Exchange 2007 out of interest?

On Wed, 12 Jan 2011 05:10:29 +0000, aha_tom wrote: >Interesting topic... I ran into this problem recently. We have an Exchange 2003 SP2 server, had dismounted its storage twice in recent months. Got error MSExchangeIS 1159. Typically indicate we have too many uncommitted transaction logs. Checked the database, we have over 10 15GB mailboxes and 30 4GB mailboxes and the database size is over 200GB and log folder was like, WOW, got 800 log files in the middle of the day already. So I concluded we have too many big mailboxes which generates too many transaction logs... So we did a cleanup to remove a few those big ones. It isn't the number of log filess in the directory, it's the number of log files that contain uncommitted transactions. >Everything was fine after that, recently I jumped on the server and checked the log folder, WOW! There are over 2000 transaction log files there! And yet the Exchange 2003 server is running fine... So what's the deal? I guess the way I check transaction log files are wrong. The number of uncommitted logs cannot be decided by simply count the transaction log files in the log folder! That's correct. >Then my question is: What is the correct method to check how many transaction log files have not been committed? > >Anyone? Cant seems find a document for that... Use this PerfMon counter: Database ==> Instances \ Log Generation Checkpoint Depth --- Rich Matheisen MCSE+I, Exchange MVP --- Rich Matheisen MCSE+I, Exchange MVP

On Wed, 12 Jan 2011 19:55:41 +0000, Pancamo wrote: >Thanks all, so basically with Exchange 2003, there will *always* be *at least* a 4 log delay between a transaction being written to a log file and being written to the database? Does anyone know what this figure is for Exchange 2007 out of interest? No. The 20MB value is the maximum amount of data that the ESE can write to logs before it writes it to the database. But transactions can span more than 20MB and on a busy server it doesn't take long to accumulate 20MB of data. So you may have many log files with uncommitted transactions, but you won't have more then four log files worth of data that hasn't been written to the database. But writing to the database does NOT mean the transactions in that 20MB have been completed. The modifications to the database may still be rolled back if a transaction fails. --- Rich Matheisen MCSE+I, Exchange MVP --- Rich Matheisen MCSE+I, Exchange MVP

OK, find this myself. Check transaction log depth by use this command: eseutil /mk E00.chk (your checkpoint file name) It will then return: Initiating FILE DUMP mode... Checkpoint file: E01.chk LastFullBackupCheckpoint: (0x0,0,0) Checkpoint: (0x30148,2480,0) FullBackup: (0x300FD,8,16) FullBackup time: 01/12/2011 23:09:03 IncBackup: (0x0,0,0) IncBackup time: 00/00/1900 00:00:00 Signature: Create time:03/24/2004 17:21:17 Rand:442894356 Computer: Env (CircLog,Session,Opentbl,VerPage,Cursors,LogBufs,LogFile,Buffers) ( off, 252, 12600, 1741, 12600, 500, 10240, 147456) Operation completed successfully in 1.16 seconds. Now, I still need a bit explaining for this output. "Checkpoint: (0x30148,2480,0)" is where the transaction log been committed? "FullBackup: (0x300FD,8,16)" is the point where the latest Full Backup completed?

On Thu, 13 Jan 2011 03:33:30 +0000, aha_tom wrote: >Now, I still need a bit explaining for this output. "Checkpoint: (0x30148,2480,0)" is where the transaction log been committed? Yes. >"FullBackup: (0x300FD,8,16)" is the point where the latest Full Backup completed? Yes. --- Rich Matheisen MCSE+I, Exchange MVP --- Rich Matheisen MCSE+I, Exchange MVP

>>>>No. The 20MB value is the maximum amount of data that the ESE can write to logs before it writes it to the database. But transactions can span more than 20MB and on a busy server it doesn't take long to accumulate 20MB of data. So you may have many log files with uncommitted transactions, but you won't have more then four log files worth of data that hasn't been written to the database. But writing to the database does NOT mean the transactions in that 20MB have been completed. The modifications to the database may still be rolled backif a transaction fails OK, so 20MB is the maximum amount of data it will write to the log files before it starts writing to the database. But that 'writing to the database' doesn't mean that those transaction logs are necessarily commited? So, you could say that the Checkpoint Depth has nothing to do with commited / uncomitted log files, just the amount of data before Exchange writes *something* to the database? Is my understanding correct?

On Thu, 13 Jan 2011 07:07:00 +0000, Pancamo wrote: >>>>>No. The 20MB value is the maximum amount of data that the ESE can write to logs before it writes it to the database. But transactions can span more than 20MB and on a busy server it doesn't take long to accumulate 20MB of data. So you may have many log files with uncommitted transactions, but you won't have more then four log files worth of data that hasn't been written to the database. But writing to the database does NOT mean the transactions in that 20MB have been completed. The modifications to the database may still be rolled backif a transaction fails >OK, so 20MB is the maximum amount of data it will write to the log files before it starts writing to the database. But that 'writing to the database' doesn't mean that those transaction logs are necessarily commited? Log *files* aren't committed. *Transactions* are committed. A log file may contain many transactions, or pieces of many transactions, or just one transaction, or maybe just a piece of one transaction. A transaction isn't committed until the entire set of modifications is completed (ref. ACID -- "atomicity, consistency, isolation, durability"). The log file is a way to record the activity in a way that would allow the transaction to be "replayed" into a database (e.g. after a restore). The data isn't written to the log file and then read from the log file into the database under normal circumstances. >So, you could say that the Checkpoint Depth has nothing to do with commited / uncomitted log files, just the amount of data before Exchange writes *something* to the database? Is my understanding correct? Yes. --- Rich Matheisen MCSE+I, Exchange MVP --- Rich Matheisen MCSE+I, Exchange MVP

Got it (I think)....in which case, why does MS recommend dropping the Depth to 5MB for Windows 2003 clusters? If my thinking is correct: i. When failiing over a EVS in Exchange 2003, the stores are dismounted (and then mounted again when the passive node takes ownership) ii. In order for the stores to dismount cleanly, all the data in the log files for the SG's need to be commited to the DB iii. By decreasing the Depth to 5 MB, there is only 5 MB of data per SG that needs to be commited before the store can mount, as opposed to 20 without this change Am I thinking along the correct lines? Secondly, when backing up Exchange 2003, I know that transactions are not written to the DB since the Lazy Writer is disabled. I assume in this situation, the Depth can exceed 20 MB? Thanks again for the excellent assistance with this.

On Sat, 15 Jan 2011 23:38:31 +0000, Pancamo wrote: >Got it (I think)....in which case, why does MS recommend dropping the Depth to 5MB for Windows 2003 clusters? So there are fewer unwritten changes (not committed transactions) to be processed if a failover must take place. >If my thinking is correct: > >i. When failiing over a EVS in Exchange 2003, the stores are dismounted (and then mounted again when the passive node takes ownership) >ii. In order for the stores to dismount cleanly, all the data in the log files for the SG's need to be commited to the DB Close enough. >iii. By decreasing the Depth to 5 MB, there is only 5 MB of data per SG that needs to be commited before the store can mount, as opposed to 20 without this change "Committed" implies that all transactions have completed and their alterations are safely and permanently part of the database. Not all the data in the log files represent complete, committed, transactions. >Am I thinking along the correct lines? If you can assume that only complete trnasactions are written to the log files, yes. :-) >Secondly, when backing up Exchange 2003, I know that transactions are not written to the DB since the Lazy Writer is disabled. I assume in this situation, the Depth can exceed 20 MB? There's still a limit on the amount of data that can be maintained in memory, and that means that those changes are written to log files long before they can be written to the database. That's where the problems with the 1,008 log files comes into play. If the backup runs too long and there are many changes being made then the number of log files with uncommitted transactions will rise and . . . {{ poof }}. FYI, this is another reason why on-line defrags are stopped when backups are running. The number of potential changes to the database during a defrag is enormous. >Thanks again for the excellent assistance with this. --- Rich Matheisen MCSE+I, Exchange MVP --- Rich Matheisen MCSE+I, Exchange MVP

Thanks! Before I close this out >>i. When failiing over a EVS in Exchange 2003, the stores are dismounted (and then mounted again when the passive node takes ownership) >>ii. In order for the stores to dismount cleanly, all the data in the log files for the SG's need to be commited to the DB >Close enough. What would be the exact correct statement here? :) I just want to make sure my understanding of Exchange is 100% correct. Would it be "In order for the stores to dismount cleanly, all the [COMPLETED] data in the log files for the SG's need to be commited to the DB ? Thanks again. Also: >Log *files* aren't committed. *Transactions* are committed. A log file >may contain many transactions, or pieces of many transactions, or just >one transaction, or maybe just a piece of one transaction. So a commited log file is really one where all the transactions within it are commited...once all the transactions within a log file are commited, is it valid to say that a log file has been "commited", or is there another term for this? Secondly, I know log files are named sequentially. From what I understand, each log file can contain 5 MB of data in Ex2003, then another log file is created. Let's say we have three log files E0000000026.log E0000000027.log E0000000028.log There is a long running transaction, which is still going on, that spans all three log files. There is a also a single transaction in #27 that has ended. This means that #27 contains a transaction that can be commited, however am I correct in thinking that #27 cannot be commited until the long running transaction that spans the three files has ended? Therefore, one long running transaction can prevent a bunch of log files being 'commited'? #26

On Sun, 16 Jan 2011 14:11:22 +0000, Pancamo wrote: >Thanks! Before I close this out >>i. When failiing over a EVS in Exchange 2003, the stores are dismounted (and then mounted again when the passive node takes ownership) >>ii. In order for the stores to dismount cleanly, all the data in the log files for the SG's need to be commited to the DB >Close enough. What would be the exact correct statement here? :) It isn't necessary to commit all the changes before the database is dismounted (it's certainly *desired*, though). That's why the changes are written to the transaction log files before they're written to the database. If not all the transactions are committed then the database is in a "dirty" state and the log files, when replayed on it, will be used to recover it from that state. >I just want to make sure my understanding of Exchange is 100% correct. Would it be "In order for the stores to dismount cleanly, all the [COMPLETED] data in the log files for the SG's need to be commited to the DB ? Thanks again. You have to think in terms of how to bring a database back to a consistent state. You can't assume that just because a database dismounted that it's in a "clean shutdown" state. That's why I said "close enough" -- you were limiting your questioning to only databases in a "clean" state. :-) >Also: >Log *files* aren't committed. *Transactions* are committed. A log file >may contain many transactions, or pieces of many transactions, or just >one transaction, or maybe just a piece of one transaction. >So a commited log file is really one where all the transactions within it are commited...once all the transactions within a log file are commited, is it valid to say that a log file has been "commited", or is there another term for this? A "committed" log file has always been used as a convenient shorthand for "a log file no longer needed to bring the current database to a consistent state". There may be some other term, but I don't know what it is. The important thing to remember is that it's transactions that are important, not log files. >Secondly, I know log files are named sequentially. From what I understand, each log file can contain 5 MB of data in Ex2003, then another log file is created. Let's say we have three log files E0000000026.log E0000000027.log E0000000028.log There is a long running transaction, which is still going on, that spans all three log files. There is a also a single transaction in #27 that has ended. This means that #27 contains a transaction that can be committed, The log file doesn't know whether a transaction has been comitted or not. It's the Information Store and ESE that keep track of that and keep the internal (see "eseutil /mh") information about log files and the data they contain updated -- which also updates the Exx*.chk file. So, the transaction is just "in the log file". >however am I correct in thinking that #27 cannot be commited until the long running >transaction that spans the three files has ended? That log file, if it disappears and IS service crashes, would leave the database in an dirty shutdown state. Assuming that didn't happen, the checkpoint would not be advanced until all the information in #27 was comitted to the database. >Therefore, one long running transaction can prevent a bunch of log files being 'commited'? That's correct. The checkpoint would not be advanced. --- Rich Matheisen MCSE+I, Exchange MVP --- Rich Matheisen MCSE+I, Exchange MVP

Great thread, am finding it really interesting! This will be one of the final questions, I promise :) >>Therefore, one long running transaction can prevent a bunch of log files being 'commited'? >That's correct. The checkpoint would not be advanced. ...and what if this long running transaction caused the 20 MB Checkpoint Depth to be exceed? Would the stores dismount? I have only seen this rarely when backup software locks the DB and we break the 1008 log files....yet, I guess it's also possible for a transaction to become 'locked' somehow (although have never seen this before)? And, finally, do you know why MS recommended the Checkpoint Depth to be reduced to 5 MB for Exchange 2003 clusters? Is this so there was less data to 'commit' before the fail over process could complete?

On Sun, 16 Jan 2011 21:36:48 +0000, Pancamo wrote: >Great thread, am finding it really interesting! This will be one of the final questions, I promise :) >>Therefore, one long running transaction can prevent a bunch of log files being 'commited'? >>>That's correct. The checkpoint would not be advanced. >...and what if this long running transaction caused the 20 MB Checkpoint Depth to be exceed? Would the stores dismount? No. The data would (supposedly) be written to the log files only. There are a few "corner cases" where the normal way data and log files are treated that violate the rules. It was always amusing to watch the ESE folks at MEC (before they killed it) squirm when they got questions about them from the audience. I think it was Loren Burchall at the MEC in Boston that admitted that there are circumstances when data are actually written to the database without first being written to log files. He was really uncomfortable explaining it, but he didn't try to hide it. >I have only seen this rarely when backup software locks the DB and we break the 1008 log files.... But that isn't the same as the 20MB limit. :-) >yet, I guess it's also possible for a transaction to become 'locked' somehow (although have never seen this before)? I have, but not in years. --- Rich Matheisen MCSE+I, Exchange MVP --- Rich Matheisen MCSE+I, Exchange MVP

>>...and what if this long running transaction caused the 20 MB Checkpoint Depth to be exceed? Would the stores dismount? >No. The data would (supposedly) be written to the log files only. Do you mean written to the database only?

On Thu, 20 Jan 2011 20:05:41 +0000, Pancamo wrote: >>>...and what if this long running transaction caused the 20 MB Checkpoint Depth to be exceed? Would the stores dismount? >No. The data would (supposedly) be written to the log files only. Do you mean written to the database only? No. The information I have is pretty dated, and lots of thing have changed over the last decade. But a long-running transaction can modify the database as long as the data are written to the log files. Changes made in the database are made to a temporary area (so they can be rolled back if the trnasaction doesn't complete successfully) and are comitted only at the end of the transaction. Think of the transaction logs as a "after log", designed to be replayed on an older database copy to bring it back to the "point-in-time" of failure. The "before log" is kept within the database itself and is used to undo a failed transaction's changes. --- Rich Matheisen MCSE+I, Exchange MVP --- Rich Matheisen MCSE+I, Exchange MVP

Thanks for your patience in answering all my questions! You must be a saint :-)