Why Friends don't Let Friends use the Office 365 Migration tool (MMT)

Overview

This blog post details the issues encountered recently in migrating a customer to the Microsoft Office365 Solution using the built-in E-mail migration tool (aka Move Mailbox tool or MMT).  It should be noted that many of these issues are well known by Microsoft and they are working on fixes or workarounds for some.

The following are a high-level list of the areas the MMT has issues that were experienced during the migration of this customer to the Cloud.

1. Limiting Choices
2. Speed
3. Reporting
4. Data inconsistencies

Limiting choices

Up until recently the Microsoft's MMT was just one of many options for migration. But recent changes to the Directory Synchronization tool (Dirsync) now misleads customers and partners to think they can only use the MMT instead of 3^rd party migration tools.  The recent change was that once you use the Office 365 Dirsync the only way to get a mailbox provisioned in 365 is to run it through the migration process using the MMT.  The Office 365 Portal UI actually will point out that once you add a license to a mailbox you MUST migrate that mailbox using MMT to have the new mailbox provisioned.

Now this would not be so bad If there was an option to only provision the mailboxes with the MMT  and migrate with another 3^rd party tool but that feature is not available or is not exposed at this time.   Some of the 3rd party tools have worked around this issue, but I don't have details on that just yet.  It baffles me that the dirsync tool was changed force customers to use the MMT without any clear communications to the partner or customer channel. 

Workaround:

To work around this issue you would have to change the MIIS attribute mapping in Dirsync to leave out some of the attributes that trigger this situation, or don’t use Dirsync at all. Editing the attributes mappings for Dirsync is technically not supported and can cause other issues. Not using Dirsync limits customer options for Single-Sign-On (SSO) and integration with AD, which is not a great story for the cloud. Some 3^rd parties have created their own dirsyc options to work around this, but they lack the integration with AD and may not be compatible with SSO or other features that require dirsync.

Speed

Acording to their own support staff Microsoft is constantly experiencing “congestion” on their migration servers for MMT that is causing the following error to happen to random mailboxes regardless of size, complexity or status.

E-Mail Migration failed for this user because no e-mail could be downloaded for 1 hour, 20 minutes.

The main solution that Microsoft is looking into to remedy this is to change the timeout to 4 hours.  So considering that a new batch cannot start until another batch completes, now we’re going to waste 4 hours of migration time instead of 80 minutes. When the MMT is having “congestion” issues, normally only between the hours of 5pm EST and 6am EST, you will see the following when you open Exchange Control Panel (ECP) and select any information from the MMT:

So the best recommendation by Microsoft support is to “Run your migrations during the day, things are not as slow then.  Truthfully the MMT migration tool is faster during the day, a PowerShell command to retrieve MMT data that may take 15 minutes at night runs in a mere 5 minutes during the day.  But every day it seems to be getting slower as more customers and partners realize it’s slow at night and move to the day.  Eventually it will be slow all the time, but if Microsoft does not address these issues soon everyone will stop using the tool and the performance problems will go away.

And don’t get me started about the actual migration speed.  In the time it took to move 50 mailboxes with the MMT we migrated in parallel over 500 similar sized mailboxes with another 3^rd party tool.  The MMT tool only runs about 5-20MB per minute, when we’re getting over 5 times that speed with other tools.

Lastly, once you start a batch you are stuck with it until it’s done or you cancel it. The MMT tool does not support running multiple batches at the same time, although you can run up to 50 mailboxes at the same time in the same batch.  So say you have one 25GB mailbox in a batch and then you have another batch of users that need to start the next night.  When you come in to start the next batch you notice that all but one of the mailboxes in the first batch are done, but the large 25GB mailbox is only 50% complete. Your choice now is pain or more pain……  i.e. you can choose to stop the current batch with one user or wait until it’s done next Tuesday. There is a pause button in the MMT tool, but it won't acctualy pause since it actually cancels the migration and stops the batch.  So you stop that batch, start the next batch. Well now you need to remember to come back around and create another batch to finish up that large mailbox at a later time, which means it’s incomplete until that is done.

Workaround:

The only workaround at this time is to re-submit failed timeouts over and over again until they are successful.  Not only is this workaround inadequate, but it costs customers and partners more money because they have to have resources babysitting the migration process all night in addition to support resources during the day. The migrations tend to go well into the working day which unfortunately costs customers downtime as its eating precious network bandwidth during a time when they are trying to get their business done.  As you can see from the following screen-shot we had to run 31 separate batches for a migration that should have only been 4-5 tops.

 

  Reporting

The reporting capabilities of the Microsoft tool at first seem to be adequate, but once many migration batches are run it becomes apparent that reporting the success and errors in downloadable reports does not always work.  The screen shots below show two batches that completed, but only one has the downloadable reports available.

 Workaround:

The only way to get any good data from MMT is through PowerShell, but there is no module for directly downloading the statistics.  You could try and download them from each batch, but that proves to be unreliable because as you can see above not each batch gets a migration report.

So, you essentially need to run some PowerShell commands based upon a CSV that has all the e-mail address of the users that you migrated, luckily we had one of those.  The only remaining issue with this approach is that the throttling policy can make these take hours to complete.

$mailboxes = import-csv mailboxes.csv

$migrationStats = foreach ($address in $mailboxes) {Get-MigrationUserStatistics $mailbox.UserPrincipalName}

$migrationStats | export-csv migrationstats.csv

Data Inconsistencies

The one thing you can say about the MMT is that it’s consistently inconsistent

The following user had different data every time we pulled statistics for them.  Within a 15 minute period the percentage complete went from 87% to 45% and then back to 95% while the numbers reported in the tool were WAY off.

 “Completed” does not really mean completed

 As we were trying to move the last two mailboxes out of nearly 600 moved, we got another inconsistency that would bring doubt on all reporting.

Riddle me this: How can a mailbox be completed, but change from 14% to 17% to 22% and then eventually 95%?

 

The following screen shots show the MMT GUI is incorrectly indicating that one # items were migrated, even though PowerShell shows different numbers for # of items were migrated.  When we re-ran these users all their mail was duplicated because we got bad data from the MMT.   These are just two examples of the 20 mailboxes that had this issue. 

 

Another Mailbox for Keith:

 

 

Workaround:

Only trust half of what you can see and none of what the GUI says.  If the migration statistics for one mailbox say the MMT moved 2500 items, run the “get-mailboxstatistics” command against that mailbox and verify it thinks there are at least that many or more in the mailbox.  And if the MMT says zero, then you better not trust it at all and verify within the mailbox the actual amounts by opening the mailbox in OWA using the migration account with permissions.

Summary:

 The Microsoft Move Tool (MMT) is included in Exchange 2010 and generally can be a very useful tool to move mailboxes for migration.  Unfortunately Microsoft has deployed it in their Office 365 offering in a way that limits it's usefulness and increases the overall time to migrate. It would not be so bad if you had other choices, but you use the MMT the moment you install the Office365 Dirsync tool.  It should be noted that the four issues above do not seem to impact 3rd party tools as much which makes them faster and much more reliable.

 

Dynamic scalability

Paying for what you use is a side effect of dynamic scalability. There are a number of scenarios that come to mind which require this capability, and cloud platforms make it easier than ever. Of course, it is in the provider’s best interest to give you an easy way to add more capacity, be it storage space or computing power. The more you use, the more money they make. In Windows Azure, changing the computing power available to an application is very easy.
Amazon’s EC2 offers a feature they refer to as ‘Auto Scaling.’ Auto Scaling lets you specify conditions according to which the computing capacity available to your application is increased or decreased. For applications that experience spikes or dramatic variability in usage, it may be your best option.

Benefits of Cloud Computing: Pay for what you use

One of the most distinguishing features of cloud based application platforms is the ability to dynamically adjust the computing resources available to an application, and pay for those resources accordingly. Amazon calls this capability elasticity: “Amazon EC2 enables you to increase or decrease capacity within minutes, not hours or days. You can commission one, hundreds or even thousands of server instances simultaneously.”

Others are drawing parallels between cloud computing and electric utilities, which provide resources on demand based on consumption. Like your electric bill, your cloud computing bill can be unexpectedly large if you don’t keep a close watch on the resources your solutions are utilizing.

Paying for what you use, and being able to adjust what you are using dynamically at the touch of a button, is an enormous benefit, but it also brings with it a new consumption model that you will need to understand. Your organization may need to make adjustments to its procedures and policies to ensure that the utilization of cloud computing resources is monitored and controlled. Otherwise, the cost could be great.