Posts filed under ‘Exchange and O365’
How to migrate a Exchange DAG using Nutanix MOVE.
We recently moved 2 Exchange servers part of a DAG using the Nutanix MOVE. The source environment was VMware ESXi (VMware ESXi, 7.0.3, 23794027) and the target environment was AHV 10.3.0.2. I am sharing the steps we followed below to help fellow techs who may come across a similar requirement.
Started the process as below:
- RE-directed all the client and SMTP traffic to another node.
- Put the 1st Exchange node in the maintenance mode.
- Created a Nutanix MOVE job to migrate the Exchange server on the maintenance mode.
- After verifying all the Exchange services are up and running , and the client & SMTP traffic is working fine (including the DAG replication ) , We moved forward with the remaining node with the same steps.
I can confirm that the above approach helped us to migrate our Exchange DAG environment from VMware to Nutanix AHV successfully.
How to offboard Exchange Online Mailboxes to Exchange On-Prem using Exchange Hybrid Environment
Recently , we had a project to offboard Exchange Online Mailboxes to On-Prem Exchange. In this environment customer already have Entra Connect Server. I would like to share the steps we followed during the project so it could be referred by anybody who is having a similar requirements.
However , I am not going to list out any commands here as it is freely available in many web sites.
- First we deployed Exchange On-Prem Server and completed all the configuration like DAG , Connector etc.
- Then we deployed the Exchange hybrid Configurations with Centralized Mail Transport.
- All the autodiscover,MX and other Exchange related DNS records were re-pointed to Exchange On-Prem Public IP’s.
- Created a TestMailbox in Exchange On-Prem and tested the email flow.
– Email sent to Exchange On-Prem Mailbox from Outside delivered successfully.
– Email sent to ExchangeOnline Mailbox from outside bounced back with an error user not found in the organization.
– Email tests sent from Exchange On-Prem to Exchange Online and vice versa failed. - After troubleshooting we found out the Exchange Online mailboxes were not visible on Exchange-On-Prem ECP. Further troubleshooting revealed that the AD user attribute for the AD synced users does not have Remote Routing Address.(e.g testdomain.mail.onmicrosoft.com)
- So we decided to do a bulk update of all the users to populate their Remote Routing Address.(You must do a Delta AD Sync after this process). Thereafter , we were able to see all the Exchange Online Mailboxes in the Exchange On-Prem ECP.
- Tested all the email scenarios and confirmed everything is working fine.
- The other obstacle we faced is that , when an email sent from Exchange Online Mailboxes to outside world the relaying was denied by the Exchange On-Prem server.Even though , we believed the default FrontEnd receive connector in Exchange On-Prem had all the requirement for this, we ended up creating a new Receive Connector and added all the Exchange Online IP’s to allow the email relay and resume the e-mail flow.
Hope it helps some one who is struggling to find a proper article for similar scenario. In case , if you want to reach out to me for detailed steps like commands used for bulk import/export/AD properties update , please feel free to reach out to me.
Move Exchange mailbox FailedOther stops at 95%
When migrating mailboxes to the newer server , the process may stop or fails at 95% with the message FailedOther.
In these cases ensure you have not set any quota on the target database or other settings. If every thing looks ok and still the migration is failing , you could use the below commands to resolve the failure
Firstly , remove the failed move request
Get-MoveRequest -Identity “Failed Mailbox ID” | Remove-MoveRequest
Then run
New-MoveRequest -Identity “Failed Mailbox ID” -TargetDatabase “DB” -BadItemLimit “200” -AcceptLargeDataLoss
That’s it you could see now the mailbox migration has completed successfully.
Unable to move mailboxes via ECP/EAC
When you are migrating from Exchange 2013 to Exchange 2019 you may notice that you are not able to migrate the mailboxes from the Exchange 2013 to Exchange 2019 via the EAC.
The workaround in this situation is to use the New-MoveRequest or New-MigrationBatch Exchnage Powershell commands. Even though , when you these commands the move request may fail in some cases and may not appear in the EAC. You could refer MS articles on how to create the CSV for migration batches and use it with these command line tools.
In breif:
- Create a CSV file using the EAC
Then modify the CSV file as below:
Run the below commands to initiate the migration and get the statistics
New-MigrationBatch –Name –CSVData ([System.IO.File]::ReadAllBytes(“”)) –Local –TargetDatabase
Get-MigrationUser | Get-MigrationUserStatistics
Let’s go back to the original problem and let’s see how can we fix it. You need to find out the System Mailboxes (especially the Migration mailbox) from the older version of Exchange server and transfer it to a database in the newer version of exchange.
Get-Mailbox -Server “OLD Exchange Server ” -Arbitration | New-MoveRequest -TargetDatabase “”Database in the Newer Exchange”
That’s it you can happily use the EAC to migrate the mailboxes.
The public folder database ‘Public Folder Database 0206806109″ cannot be deleted
If you are migrating from an earlier version of Exchange to a newer version , you may come across a situation where you are not able to decommission the old exchange server due to the inability to delete the Public Folder databases with the below error message.
Public Folder Database 0206806109
Failed
Error:
The public folder database “Public Folder Database 0206806109” contains folder replicas. Before deleting the public folder database, remove the folders or move the replicas to another public folder database. For detailed instructions about how to remove a public folder database, see http://go.microsoft.com/fwlink/?linkid=81409&clcid=0x409.
You may opt to use the ADSI edit to forcibly delete the Public folder. But , I would not recommend this approach as the ADSI Tool there is no error check or validation is performed.
You can use the below powershell commands to safely delete the replicas of the remaining Public folders and System public folders after ensuring all the needed Public Folders are migrated to the new exchange servers.
Get-PublicFolder-ServerExch-2010 "\NON_IPM_SUBTREE"-Recurse-ResultSize:Unlimited | Remove-PublicFolder-ServerExch-2010-Recurse-ErrorAction:SilentlyContinue
Now you could remove the Publice folder dataabsae
Remove-PublicFolderDatabase “Public Folder Database 0206806109”
Source:https://blog.rmilne.ca/2020/09/30/unable-to-remove-exchange-2010-public-folder-database/
A server-side database availability group administrative operation failed. Error The operation failed. CreateCluster errors may result from incorrectly configured static addresses
Recently , we were working with one of our customer to build the Exchange 2019 DAG. When we add the 2 Exchange Nodes to the DAG , the process got failed with the error “A server-side database availability group administrative operation failed. Error The operation failed. CreateCluster errors may result from incorrectly configured static addresses – Error: Windows Failover Clustering isn’t installed on ‘xxxxxx’.. [Server: xxxxxxxxx] – Restart the server to complete the failover cluster installation also check for the duplicate mac , incorrect IP address, if it is a VM make sure you have done sysprep.”
However , we were pretty sure that the servers were created using a fresh installation instead of syprepped images. Also , we noticed that the Server Manager on both the servers were displaying a message ” Restart is Pending”.
Therefore , we simply restarted the server , and thereafter we were able to add the 2 exchange servers to the DAG.
How to migrate the AAD Connect to a new Server
For those who running AAD Connect to Synchronize their On-Premise AD , may come across a requirement to upgrade the OS on the AAD connect server. In this situation , the safest approach would be to install a new server with latest OS version and then migrate the AAD connect with the below mentioned steps safely without facing lengthy outages.
- Prepare a new server with the Operating System.
- Install the AAD Connect (Preferably the same version as the existing or the latest version available).
- Now , Prior to proceed with the migration you need to ensure two things.
– Take a backup of existing configuration using the AAD Connect –> View or Export configuration
–>Click on Export Settings. This will create a json file in X:\Program Data\AADConnect. (Copy it to the new server)
– Secondly use the Azure AD Sync Configuration Documenter to collect the existing configuration in HTML format.(https://github.com/Microsoft/AADConnectConfigDocumenter/releases) . The Installation instruction can be found in https://github.com/Microsoft/AADConnectConfigDocumenter/wiki.
Thereafter , proceed with the installation of AADC on the new server , select the Customize option (instead of Express) –> Import Synchronization settings ,and import the configuration using the json filed copied in the above step and press Next.
In the last screen of the AADC Installation wizard , select the below options.
– Start the synchronization process when the configuration completes.
– Enable staging mode.
Now , you using the AADC configuration documenter capture the settings on the new AADC server. Then run the below command to compare both the configuration.(Refer the wiki for instructions) after copying both the files in the same location.
AzureADConnectSyncDocumenterCmd.exe “AADC-SERVER-OLD” “AADC-SERVER-NEW”
After reviewing the output and confirming the configuration are identical proceed with the final steps in the migration.
– Enable the staging mode on the Old AADC server.(by running the AADC – Tasks- Configure Staging mode).
– Disable the staging mode on the New AADC server.
– Perform a test and confirm the synchronization is working as expected.
– Uninstall the AADC from the old server and proceed with the decommission.
How to Remove office 365 mailbox without deleting user account
We were planning to deploy an Exchange Hybrid Configuration with On-Premise Exchange 2019 servers. Thereafter , we will be moving some of the mailboxes to On-Premise Exchange
Servers and disable only the Exchange Online mailbox to keep the user account intact with the license so he could use other Applications (like SharePoint Online , OneDrive and Desktop Apps ,etc).
The environment is already synced via Azure AD Connect.
Since the Exchange Online license is assigned to the user , the user could have one mailbox in Office 365 and one in an on-premises Exchange ,causing mail delivery issues . Thus , to solve the issue
we need to
1) Remove the Exchange Online License for the user.
2) Clear the mailbox info:
Set-User user@tga.gov.sa -PermanentlyClearPreviousMailboxInfo
3) Resync and Ensure the MSExchangeGuid is Synced to point that now the Mailbox is in On-Premise Server.
4) Re-assign the Exchange Online License in O365.
5) Now the user will not be having a mailbox , But, he will be able to access the SharepointOnline , OneDrive , the DesktopApps , etc.
OR , we could use the below steps as well (Recommended by the MS Support)
– Migrate the O365 mailbox to On-Premise Exchange Server.
– Go to the License & Apps under the User property in O365 portal.
– Unselect the Exchange Online (Plan 2) from the Apps list.
– This will ensure that the user mailbox is only in On-Premise Exchange Server and still he will be able to access the SharepointOnline, OneDrive and DesktopApp ,etc)
How to verify SPF/DKIM/DMARC/DomainKey/RBL tests parsed on the email.
Hi All
In some situations , when an email is blocked by the antispam device , you need to analyze the headers to findout the actual root cause. However , sometimes the blocked reasons shown by the antispam device are not descriptive especially when you are troubleshooting DKIM related failures.
This is crucial as we need notify the sender’s domain owner to rectify the issues. In these cases you could use the below tool to analyze the tests performed in the email and findout the actual reason for the email blockage in a much descriptive manner.
https://www.appmaildev.com/en/dkimfile
I found it very useful , and thought of sharing it with the community.
Good luck guys.
The Microsoft Exchange Replication service couldn’t find a valid configuration for database ‘mydb’ on server ‘server1’. Error: An Active Manager operation failed. Error: The active copy for database could not be determined:
Recently one of our customer had an Exchange Server outage. Whereas one of the node from 2 node DAG was not able to communicate with other server. We tried to rebooting the server but no luck . The symptoms were
– Cluster service was not starting.
– On the other node FCM was not able to connect to the DAG cluster.
– Unable to open the ECP/OWA.
– Outlook / Mobile Users cannot access their mailboxes.
The environment consist of 2 Exchange Server 2016 & DAG.
So we started with removing the failed node by typing the below commands in the healthy exchange server:
# Open Exchange PowerShell
#Get-ClusterNode -Name node1 | Remove-Clusternode
# net stop clussvc
# net start clussvc
# Remove-DatabaseAvailabilityGroupServer -Identity “DAG Name” -MailboxServer “Failed Exchange Server Name” -ConfigurationOnly
# Get-ClusterNode “Failed Exchange Server Name” | Remove-ClusterNode
However , still no luck. Then we tried to remove the mailbox database copies form the failed node(I believe the copies were active on the second server) we got a new error stating that “Mailbox databse copies cannot be disabled on the Databases with circular logging enabled. So we had to remove the circular logging and then delete the database copies.
#Get-MailboxDatabse | Set-MailboxDatabase -CircularLoggingEnabled $False
# Get-MailboxDatabaseCopy -Identity “DatabaseName” |Remove-MailboxDatabaseCopy -Identity “DatabaseName\ServerName”
Now it is the time for mounting the databses: When we ran the below command
#Get-MailboxDatabase | Mount-Database
We received the below error”
Failed to mount database “———-“. Error: An Active Manager operation failed. Error: An Active Manager operation
encountered an error. To perform this operation, the server must be a member of a database availability group, and the
database availability group must have quorum. Error: Automount consensus not reached (Reason: FSW boot time did not
match (FSW-Remote: 2020-10-01T11:37:09.4930830Z FSW-Reg: 0001-01-01T00:00:00.0000000)). [Server:———-]
+ CategoryInfo : InvalidOperation: (—-:ADObjectId) [Mount-Database], InvalidOperationException
+ FullyQualifiedErrorId : [Server=————,RequestId=5022acdd-0c48-4584-b2eb-1d0a2c692f0d,TimeStamp=10/7/2020
11:09:14 AM] [FailureCategory=Cmdlet-InvalidOperationException] BCF4F25E,Microsoft.Exchange.Management.SystemConfi
gurationTasks.MountDatabase
Here , we wanted to check the DAG status
#Get-DatabaseAvailabilityGroup -status
Output :
WARNING: Unable to get Primary Active Manager information due to an Active Manager call failure. Error: An Active
Manager operation failed. Error: An Active Manager operation encountered an error. To perform this operation, the
server must be a member of a database availability group, and the database availability group must have quorum. Error:
Automount consensus not reached (Reason: FSW boot time did not match (FSW-Remote: 2020-10-01T11:37:09.4930830Z FSW-Reg:
0001-01-01T00:00:00.0000000)). [Server: ———-]
Name : DAGName
Memberservers: {Server1}
Operational Servers {}
The output confirmed that there a no operatinal servers. So we had to restart the DAG with the surviving node.
#Start-DatabaseAvailabilityGroup -Identity “DAGNAME” -MailboxServer ServerName”
Now the Get-DatabaseAvailabilityGroup -status cmd will gave the below output:
Name : DAGName
Memberservers: {Server1}
Operational Servers {Server1}
Now it is the time for mounting all the databases:
Get-MailboxDatabase | Mount-Database
Voila , The MailboxDatabases were started mounting and the ECP access is restored.
Hope you like it.. 