Transaction Logs

If a backup is completing successfully, Exchange will flush all logs that have been committed to the database. So normally, all committed transaction logs will be flushed if:
1. All databases in the Storage Group are backed up
2. All databases in the Storage Group are mounted during the backup

The store determines what logs will be deleted by looking at the first log that has not yet been committed and deleting all log files previous to that. You can view the first uncommitted transaction log by running eseutil /mk on the checkpoint file.

After the backup completes, ESE Event ID 224 will be logged telling you what series of transaction logs will be deleted: If eseutil /mk E01.chk outputs E010000G then E0100005 - E010000F will be deleted. The purging process is sequential and will purge all log files in the series with one caveat - the purge process will stop if it goes to delete a log file that is missing. So in the above example, if log E010000A is missing, then only logs E0100005 - E0100009 will be deleted. In this scenario, after the next backup, Exchange will again try and purge all log files that have been committed. This time eseutil /mk E01.chk outputs E01000016 as the uncommitted log file and therefore E010000B - E0100015 will be purged.

If transaction logs are not purging, sooner or later you'll run out of disk space. If you have to create space in a hurry do not move the log files, compress them (in my lab I've seen 3 GB worth of log files compress to 1.5 GB). For recovery scenarios and for the purge process to complete successfully, do not move the transaction logs.

Transaction logging in Exchange server 2003:
http://www.microsoft.com/technet/prodtechnol/exchange/guides/UseE2k3RecStorGrps/d42ef860-170b-44fe-94c3-ec68e3b0e0ff.mspx

Using ESEUTIL to determine which transaction logs have been committed:
http://support.microsoft.com/kb/182961

How to remove Exchange server transaction logs:
http://support.microsoft.com/kb/240145

Message Restrictions and Size Limits

Interesting read about how Exchange applies message restrictions / size limits.
Exchange Insider
It's important to note that for internal messages, the restriction setting on individual accounts trumps global settings.
For Internet email, global settings are applied: 322679

Teo

Free IPod...??

Hey [friend's name], Check out this site ipods.freepay.com
Click here: http://ipods.freepay.com/?r=27443572

Designing Storage for Exchange 2003

Exchange Storage Design

The point of proper storage design is to ensure that each users gets enough IOPS. Consider the IOPS that the storage can provide:
* RAID 10 ( 2R 1W) RAID 5 (4R 1W)
* 15K rpm disks - 180 IO/second Before Controller
-Maximum Throughput = 180 X 80% (buffer) X .75 (RAID Factor) = 108 IO/second
-RAID 10 Factor = (R + W)/(R + 2W)
-RAID 5 Factor = (R + W)/(R + 4W)
* 10 Disks at RAID 10 - 1080 IO/second (Max IO/second that the disk will give you)
Using the example above, 1000 users would be able to receive 1.08 IOPS each.

JetStress can also be used to verify that 10 (15K RPM) disks at RAID 10 successfully sustain 1080 IOPS. Increasing the threadcount of JetStress will determine the most IOPS that those 10 disks can give you. As the thread count is increased, it's expected that the IOPS will not go above 1080.

Also, don't forget to consider IOs per database (since databases will be on seperate disks). If you pace 500 users on one database and 1500 on another, you will not achieve the required IOPS/user.

How many disks are needed?
To work this backwards we would need to following information:
* Mailboxes per server 1000
* Users I/O profile - 1.5 * IOPS = 1000 * 1.5 = 1500
*Read ratio(4 / (2+1) = .8
*Write Ratio (1 - .8) = .2
*RAID PENALTY for RAID 10 - 2 - Each write requires 2 disk I/O

(IOPS X Read Ratio) + [RAID Penalty](IOPS X Write Ratio)
-------------------------------------------
Spindle Speed Behind Controller (180)

*Result = 1800 / 180 = 10

Teo
Links:
Exchange Team Blog - Disk Sizing
Exchange Team Blog - Disk IO
Petri - Exchange Sizing

Exchange 2003 bandwidth requirements

I did some research over the weekend into the bandwidth requirements of
Exchange 2003 and Outlook 2003. Microsoft published an Operation paper titled "10 things to think about". Here they recommend using the following for a ballpark estimate:
Regular MAPI clients
# of concurrent users * 2.5 Kbps

Heavy MAPI clients
# of concurrent users * 3 Kbps

I noted some technology considerations below. I was also able to find
two white papers on traffic analysis; one by Citrix and the other by
Microsoft. Finally, I jotted down my initial thoughts.

-- TECHNOLOGY CONSIDERATIONS --

Exchange 2003 Considerations
* LDAP searches directed at Global Catalog servers. By default, Exchange will refer Outlook clients to LDAP servers in the Exchange servers local site for directory access.
* Public Folder replication (unable to predict)
* OWA usage - Based on readings a single Outlook client, based on a
heavy profile, could require as much as 15 Kb/s per client (see white
paper below)

Outlook 2003 Considerations
* Every environment is different depending on the users profile (think of
IOPS when considering storage)
* Improved MAPI communication - Data is compressed and more data per
packet sent that in previous versions of Outlook. (only when working with
Exchange 2003).
* A single Outlook client, based on a heavy profile,
could require as much as 10 Kb/s per client (see white paper below)
* Cached mode has been designed to work over WAN links and tolerate
latency. However, it does require synchronization of an OST and OAB
(offline address book). This could result in high network utilization
if many users connect to Exchange at the same time and/or synchronize
their mail at the same time.

* OAB (Offline Address Book) - In a large organization, the OAB can be
several MB's in size (38 MB in my current environment). This can grow
significantly large when PKI is implemented. In addition there are multiple scenarios where Exchange forces all users to execute a full download:
KB839826

Desktop search engines
* We've had some problems with the Google and MSN desktop
search engine. When these applications try and index a user's
mailbox, they'll execute as many connections as possible. Eventually one
desktop can put the load of 100 clients (just while indexing a single mailbox).
KB905184


-- WHITE PAPERS --
Citrix Traffic Analysis
Citrix Link

Microsoft Traffic Analysis
M$ Link

-- LINKS --
Top 10 things to Consider



-- MY THOUGHTS --
One approach could be to work with the current Exchange adminstrators to
try and establish 3 basic user profiles. Then
hopefully make some basic bandwidth estimates and work with the network
team on creating a flexible network plan. As the migration
moves forward that plan could be adjusted based on trends.

Another approach could be to look at WAN accelerators
Link

Teo

LegacyExchangeDN

-- Mail Routing --
The routing of mail to Exchange recipients is done with the LegacyExchangeDN attribute. Messages sent are stamped with the senders LegacyExchangeDN.

-- Migrations --
With any version of Exchange, when the mailbox is moved, it's LegacyExchangeDN is overwritten to represent the new location in the directory. Because of routing explanation above, this breaks replyability. Normally, this is fixed by adding an X.500 proxy address that represents the LegacyExchangeDN. Exchange 2003 SP1 does this to all mailboxes moved between administrative groups.

-- Disaster Recovery --
A great article on the role that the attribute LegacyExchangeDN plays in a disaster recovery scenario.
MSExchange.org

Finding where a user was deleted

A blog I review posted 3 steps that show who deleted a user account.

1. adfind -default -showdel -f (isdeleted=TRUE) -gc


2. repadmin /showobjmeta dcname deletedobjectDN find /i "isdeleted"


3. Eventcomb to find event id in security logs.


a. psloglist \\dcname security -i 630 -a date.

Link

Cached Mode, Security Groups, and Paged Pool Memory

The Exchange team's blog has an interesting article which goes into detail about security tokens, and how much paged pool memory they consume. If you have users in more than 80 security groups, it can impact performance. Read more by following the link below...

Link

Teo

Log Parser

I came across a great article about Log parser from Exchange & Outlook Administrator (Windows IT Pro). It described the log parser tool and how to use it on a variety of logs.
The first thing I thought about was gathering statistics about OWA useage. I used the following query that I got from Technet:
logparser "SELECT TO_STRING(time, 'HH') AS Hour, COUNT(*) AS Hits INTO MyChart.jpg FROM ex*.log GROUP BY Hour ORDER BY Hour ASC" -i:IISW3C -o:CHART -chartType:ColumnClustered -chartTitle:"Hourly Hits" -groupSize:420x280
This outputs a great JPEG that shows you hits per hour.

I went to the Logpaerser forum to try and get some help in modifying the query so that I could get unique logons per hour. Here is what Daniel Einspanjer helped me develop Forum Link:

First, create the two sql files:
QAuthenticationv2(1).sql
SELECT
TO_STRING(time, 'HH') AS Hour,
COUNT(*) AS Hits,
cs-username
INTO STDOUT
FROM c:\owa_logs\*
WHERE
(cs-username IS NOT NULL)
AND
(sc-status = 200)
GROUP BY Hour, cs-username

QAuthenticationv2(2).sql
SELECT
Hour,
SUM(Hits) AS Hits
INTO Authenticationv2.gif
FROM STDIN
GROUP BY Hour

Then execute them as follows:
c:\logparser>Logparser.exe file:QAuthenticationv2(1).sql -i:IISW3C -o:CSV | LogParser.exe -i:CSV file:QAuthenticationv2(2).sql -o:CHART -charttype:Columnclustered

Links:
TechNet
Logparser Download
http://www.logparser.com/
Log Parser Book

Exchange 2000 & 2003 - Message Tracking Logs

Two KB articles that help interpret Message Tracking Logs:
KB821905
KB246965
A website that Explains what the ID's in the Message Tracking Logs mean:
Swinc
Finally, a script that will report on all the Message Tracking Logs
Outlook Exchange

Gmail Signature

Just wanted to post a link to a site that will create graphic signatures.
Link

Here's the code for mine:

Forum:
[EMAIL=teoheras@gmail.com][IMG]http://gsig.brightdev.com/2/teoheras.png[/IMG][/EMAIL]

Webpage:
<a href="mailto:teoheras@gmail.com"><img alt="GSig" src="http://gsig.brightdev.com/2/teoheras.png" /></a>

Unable to move mailbox - access denied

I've found that I cannot move mailboxes using ESM or ADUC from my laptop. I get access denied. I know I have access because I can successfully move mailboxes when logged on to the Exchange server. I found this article online that explains and resolves this issue.

http://support.microsoft.com/?kbid=886700

Teo

Urlscan Exchange 2003 Windows 2003 SP1

I finally fixed the problem I encountered with OWA and urlscan after installling Windows 2003 SP1. Basically I was getting page cannot be found errors. I found that owalogon.asp was being denied by the urlscan template that I copied from an MS KB article. Here are the KB articles that I would recommend to anyone installing URLScan:

How to configure the urlscan tool
http://support.microsoft.com/?kbid=326444

URLScan tool may cause problems in Outlook Web Access
http://support.microsoft.com/?kbid=325965

DO use the urlscan template listed here
Fine-tunning and known issues when you use the Urlscan utility in an Exchange 2003 environment
http://support.microsoft.com/?kbid=823175


**Do NOT** use the urlscan template listed in this KB - This is what broke OWA with Windows 2003 SP1
IIS lockdown and URLScan configurations in an Exchange environment
http://support.microsoft.com/?kbid=309508
In the authors defense, the template is for Exchange 2000. It worked fine with Exchange 2003 until I installed Windows 2003 SP1.

Exchange Administrator w/o local administrator rights

Someone posted a question asking how you could delegate the Exchange Administrator role but not make them a member of the local administrators group on the server. Exchange 2003 SP1 requires this other wise you get an error with ID no: c10308a2. Microsoft has released the following KB article that allows you to circumvent this.

http://support.microsoft.com/default.aspx?scid=kb;en-us;905809

Exporting and Importing Proxy addresses

We've been doing alot of modifications to the proxy addresses of our user accounts via scripts. We also had an ADC issue which caused roughly 1/3 of our mailboxes to become disconnected. This was especiallly problematic because when you reconnect a mailbox it's proxy addresses are re-generated and any custom/friendly addresses are lost. So, I think it's a good idea to document a simple way to export and import proxy addresses. This will also work for any other attribute.
Here's how:

Ldifde -d "DC=domain,DC=com" -s DC_Name -r "(&(mailnickname=*))" -l proxyAddresses -f proxies.txt

The query above will generate the following output to proxies.txt
----- Begin File: proxies.txt-----
dn: CN=Teo\, Heras,OU=Users,DC=lab,DC=microsoft,DC=com
changetype: add
proxyAddresses: X400:c=US;a= ;p=Microsoft;o=Lab;s=Heras;g=Teo;
proxyAddresses: SMTP:Teo_Heras@microsoft.com
----- End File-----

Additional attributes besides proxy addresses should be added next to "-l" (comma seperated).

Reimporting will require the manipulation of the output file as follows:
----- Begin File: proxies.txt -----
dn: CN=Teo Heras,OU=Users,OU=West Chester,OU=Corporate,DC=cablelab,DC=comcastlab,DC=com
changetype: modify <---- change from add to modify
replace: proxyAddresses <---- This was added
proxyAddresses: SMTP:Teo_Heras@Comcast.com
proxyAddresses: X500:/O=Comcastlaborg/OU=Lab-CDC/cn=Recipients/cn=theras0000
proxyAddresses: X400:c=US;a= ;p=Comcastlaborg;o=Lab-CDC;s=Heras;g=Teo;
- <---This is critical and the log file will tell you
----- End File -----

Finally, we'll import the file by doing the following:
c:\ldifde -i -f proxies.txt -s my_dc -j c:-i means import, -j c:\ is the path to log file


A large file may be hard to modiy, so I wrote a script that parses through the log file and writes the attributes back to AD. There are definately better alternatives (such as restoring AD to a lab and using VB to synchronize attributes), but it's useful to see how to parse through the output and use VBScript functions to pull the values you need.

----------SCRIPT----------
'This script will parse through the ldif export:
'and write back the proxy addresses

Option Explicit
Const ForReading = 1
Const ADS_PROPERTY_UPDATE = 2
'Define Proxy address Array
Dim arrProxyAddresses
Dim arrToWrite()
Dim objFSO, objDictionary, objTextFile, strTextfile, arrTextFile, strTextLine, objUser, strPriMail
Dim strUsrDN,strProxyAddresses, colKeys, strKey, intsize, strProxyAddress, intPriMailCount
Dim strUserDNLen, intFirstPipeLoc, intSecPipeLoc, intProxyLength, strProxyAddressArr, intFullProxyLen

set objFSO = CreateObject("Scripting.FileSystemObject")
Set objDictionary = CreateObject("Scripting.Dictionary")
set objTextFile = objFSO.OpenTextFile("c:\proxies.txt", ForReading)
strTextfile = objTextFile.ReadAll
'Fills each array entry with a line from the LDIF export.
arrTextfile = Split(strTextfile, VbCrLf)

'Loop through array and fill dictionary object
For Each strTextLine In arrTextfile
'The logic below leaves strUsrDN populated until a blank line is detected
'A blank line means the next entry is being read.
If InStr(strTextLine, "dn:") Then
strUsrDN = strTextLine
Elseif InStr(strTextLine, "proxyAddresses:") Then

'Ensure that the line doesn't just contain proxyaddress: I've seen notepad break this line placing the value on the line below.
If Len(strTextLine) = 16 Then
MsgBox "Error on " & strUsrDN
WScript.Quit
End If

'Check for a primay smtp address
If instr(strTextLine, "SMTP:") Then
intPriMailCount = 1
End If

'As long as a proxy address is detected, append all proxy addresses found
'The loop begins by checking whether or not the strProxyAddresses field is blank
If strProxyAddresses = "" Then
'Write the first proxy address without the delimeter. Otherwise when we call the split
'Function we will have a null value for the firs one.
strProxyAddresses = strTextLine
Else
strProxyAddresses = strProxyAddresses & "|" & strTextLine
End If
Elseif strTextLine = "" Then
'Check that the user object has a primary smtp address to apply
If intPriMailCount = 0 Then
MsgBox strUsrDN & " does not have a primary SMTP address."
WScript.Quit
End If
'When a blank line is detected it means the first LDIF entry has been read.
'write to the dictionary object and clear all variables
objDictionary.add strUsrDN, strProxyAddresses
'Clear out variables. When empty they are used for validation and they should be empty when the
'loop begins.
strProxyAddresses = ""
strUsrDN = ""
intPriMailCount = 0
End If

Next

'Loop through dictinary object, parse content, and write to user account.
colKeys = objDictionary.keys
For Each strKey In colKeys
'Parse through the userDN value
strUserDNLen = Len(strKey)
strUsrDN = Mid(strKey, 5, strUserDNLen)
strProxyAddresses = objDictionary.Item(strKey)
arrProxyAddresses = Split(strProxyAddresses, "|")
intsize = 0
For Each strProxyAddress In arrProxyAddresses
'Strip "proxyaddress:" - the length of proxyaddress: is 17
strProxyAddress = Mid(strProxyAddress, 17, Len(strProxyAddress))
'Proxy addresses have to be written as an array, so after stripping out the
'proxyaddress: string we'll create a new array with the values needed
ReDim Preserve arrToWrite(intsize)
arrToWrite(intsize) = strProxyAddress
'Keep track of the primary proxy address so that it can be written to the mail attribute later
If instr(strProxyAddress, "SMTP:") Then
'Use Mid to strip out SMTP:
strPriMail = Mid(strProxyAddress,6,Len(strProxyaddress))
End If
intsize = intsize + 1
Next

'Here's what I'll be writing to the user object
Set objUser = GetObject("LDAP://" & strUsrDN)
MsgBox "Writing to " & objUser.DistinguishedName
objUser.putex ADS_PROPERTY_UPDATE,"proxyAddresses", arrToWrite
objUser.SetInfo
objuser.put "mail", strPrimail
objUser.SetInfo
Next

----- End Script -----

KB Articles:
How to Modify a User's E-mail addresses by Using Ldifde
http://support.microsoft.com/?kbid=313823

How to import and Export Directory Objects to Active Directory
http://support.microsoft.com/kb/q237677/

Running a Hard Recovery on a database

We recently had a scenario where an administrator had tried to restore a full and an incremental backup. He escalated the problem to us when he couldn't see the data from his incremental backup. It turned out that a hard recovery was ran after his full backup restored. Oddly enough, most vendors have the hard recovery process run by default after every recovery. On Backup Exec, for example, this process is identified as a check next to the box labled 'Commit Logs.'

A hard recovery is the process that brings a restored database back to a consistent state (administrator intervention is required). A typical hard recovery of a database is the restoration of a full backup or a full and differential backup of the information store. During the recovery process the administrator manually begins log file replay either through the ESEUTIL /cc command or the backup program interface (‘Last Backup Set’ in NTBackup). ESEUTIL /cc must be run from within the folder where Restore.env resides (eseutil /cc {restore.env path}). ESEUTIL /cc looks for instructions in the Restore.env file. Prior to beginning a hard recovery of a database, make sure that all database files and transaction logs have been backed up. If backups are not completing successfully, then it may be necessary to shut down the information store and copy the database and transaction log files to an alternate location. This way, if the database is damaged in any way, it can be restored to the same state it was before.

Once a hard recovery is performed, the database header is changed and all other log files will be useless. The hard recovery process changes the header information on the database and only the log files from this point forward can be re-played. For this reason, it is critical that the stores not be mounted until you are sure that there are no other restores that need to take place. If, after mounting the database, you find you need to restore other log files, you will have to restore the backups to an alternate location and EXMerge the data into the production database. Otherwise, you risk further downtime by re-running your restore process and loosing the data that was written to the database(s) once the stores were mounted. Once a satisfactory restore has been performed on a database, a full backup must be run.

CIS Benchmark for Exchange 2003

I just read on http://www.e2ksecurity.com/ (Paul Robichaux blog) that the CIS Benchmark for Exchange 2003 document has been released. It covers how to harden an Exchange 2003 server environment.

http://www.cisecurity.org/bench_exchange.html

Overcomming Recovery Storage Group Limitation

The Recovery Storage Group (RSG) works great for recovering deleted items to a production database. The limitation is that the mailbox has not been moved or deleted (purged) from the original database. The Recovery Storage Group process compares two attributes before allowing a restore of mailbox items from the restored database to the production database: The msExchMailboxGUID (read only) one the mailbox and msExchOrigMDB on the database in the RSG.

Dealing with a deleted mailbox:
A mailbox's GUID (msExchMailboxGUID) is the same for the life of the mailbox. Restoring a deleted (purged) mailbox by recreating it will not not allow the RSG to connect the new maibolx to the mailbox that exists in the restored database. The new recreated mailbox has a new GUID and it cannot be changed to match the old one (the msExchMailboxGUID is a read only attribute). Microsoft recommends the following steps:
1. Add the database the mailbox was in before it was purged to the RSG
2. Restore the databse to the Recovery Storage Group
3. Mount and then dismount the database in the RSG (this will ensure that the database is in a clean shutdown state eseutil /mh databasename.edb
4. Create a new Storage Group and Database ensuring that the file names for the new database are identicle to those of the database in the RSG. Then dismount the database.
5. Copy the .stm and .edb files from the RSG location to the path of the new database.
6. In the properties of the new database (through ESM) place a check mark next to the box (this database can be over written by a restore).
7. Mount the database, connect mailbox to an AD account, Exmerge the data out of the recovered mailbox and into the new mailbox.

Dealing with a mailbox that was moved:
A database in the RSG will have an attribute called msExchOrigMDB set to the distinguished name of the original database. If a mailbox has been moved to another database the only backup available may be of the database before the mailbox was moved. To restore items Microsoft recommends the following steps:
1. Move the mailbox back to the original database
2. Modify the msExchOrgiMDB attribute so that it lists the DN of the database that now holds the mailbox in question.

Related KB Articles:
http://www.microsoft.com/technet/prodtechnol/exchange/guides/UseE2k3RecStorGrps/71dd4ae1-2a64-4411-804b-33b5972c8493.mspx
http://www.microsoft.com/technet/prodtechnol/exchange/guides/UseE2k3RecStorGrps/71dd4ae1-2a64-4411-804b-33b5972c8493.mspx
http://support.microsoft.com/?id=824126#XSLTH4144121122120121120120

Messages appear in folder titled 'Top of Information Store'

We had an issue where serveral users had reached their mailbox limits but where reporting that they had manually cleaned their mailboxes. What we eventually found was that users had somehow dragged messages into the root of the Outlook hierarchy 'Mailbox - User Name.' We found this by searching for all messages larger than 1 KB (Advanced Search) and seeing that they resided in a folder named 'Top of Information Store.' Besides searching and deleting the messages (from within the search box) there is another way to see what messages exist in the 'Top of Information Store.'
1.Find mbdvu32.exe in the Tools\ExAllTools\MDBVU32 directory on the Exchange CD
2. Exedute mbdvu32.exe
3. Click OK to clear the first window that pops up.
4. Make sure that the correct profile is slected in the 'Choose Profile' windows
5. Click on the MDB menu option.
6. Click on the OpenMessageStore option.
7. Make sure that "Mailbox- [user's full name]" is selected and click on Open.
8. Click on the MDB menu option again.
9. Click Open Root Folder.
10. In the Child Folders box - double click on "Top of Information Store".
11. In the window to the right, titled 'Messages in Folder' you will be see all the messages that the user has dragged into the top of the Outlook hierarchy.

Thanks goes out to the people at MS Exchange Blog and their article below. It actually shows how to fix OOF problems using the same utility.
http://hellomate.typepad.com/exchange/2003/10/when_oof_doesnt.html

Places I've been

I found a cools site today that allowed me to color in a world map detailing the places I'be been.



create your own visited country map
or check our Venice travel guide