Posts filed under ‘Exchange and O365’
The Microsoft Exchange Replication service couldn’t find a valid configuration for database ‘mydb’ on server ‘server1’. Error: An Active Manager operation failed. Error: The active copy for database could not be determined:
Recently one of our customer had an Exchange Server outage. Whereas one of the node from 2 node DAG was not able to communicate with other server. We tried to rebooting the server but no luck . The symptoms were
– Cluster service was not starting.
– On the other node FCM was not able to connect to the DAG cluster.
– Unable to open the ECP/OWA.
– Outlook / Mobile Users cannot access their mailboxes.
The environment consist of 2 Exchange Server 2016 & DAG.
So we started with removing the failed node by typing the below commands in the healthy exchange server:
# Open Exchange PowerShell
#Get-ClusterNode -Name node1 | Remove-Clusternode
# net stop clussvc
# net start clussvc
# Remove-DatabaseAvailabilityGroupServer -Identity “DAG Name” -MailboxServer “Failed Exchange Server Name” -ConfigurationOnly
# Get-ClusterNode “Failed Exchange Server Name” | Remove-ClusterNode
However , still no luck. Then we tried to remove the mailbox database copies form the failed node(I believe the copies were active on the second server) we got a new error stating that “Mailbox databse copies cannot be disabled on the Databases with circular logging enabled. So we had to remove the circular logging and then delete the database copies.
#Get-MailboxDatabse | Set-MailboxDatabase -CircularLoggingEnabled $False
# Get-MailboxDatabaseCopy -Identity “DatabaseName” |Remove-MailboxDatabaseCopy -Identity “DatabaseName\ServerName”
Now it is the time for mounting the databses: When we ran the below command
#Get-MailboxDatabase | Mount-Database
We received the below error”
Failed to mount database “———-“. Error: An Active Manager operation failed. Error: An Active Manager operation
encountered an error. To perform this operation, the server must be a member of a database availability group, and the
database availability group must have quorum. Error: Automount consensus not reached (Reason: FSW boot time did not
match (FSW-Remote: 2020-10-01T11:37:09.4930830Z FSW-Reg: 0001-01-01T00:00:00.0000000)). [Server:———-]
+ CategoryInfo : InvalidOperation: (—-:ADObjectId) [Mount-Database], InvalidOperationException
+ FullyQualifiedErrorId : [Server=————,RequestId=5022acdd-0c48-4584-b2eb-1d0a2c692f0d,TimeStamp=10/7/2020
11:09:14 AM] [FailureCategory=Cmdlet-InvalidOperationException] BCF4F25E,Microsoft.Exchange.Management.SystemConfi
gurationTasks.MountDatabase
Here , we wanted to check the DAG status
#Get-DatabaseAvailabilityGroup -status
Output :
WARNING: Unable to get Primary Active Manager information due to an Active Manager call failure. Error: An Active
Manager operation failed. Error: An Active Manager operation encountered an error. To perform this operation, the
server must be a member of a database availability group, and the database availability group must have quorum. Error:
Automount consensus not reached (Reason: FSW boot time did not match (FSW-Remote: 2020-10-01T11:37:09.4930830Z FSW-Reg:
0001-01-01T00:00:00.0000000)). [Server: ———-]
Name : DAGName
Memberservers: {Server1}
Operational Servers {}
The output confirmed that there a no operatinal servers. So we had to restart the DAG with the surviving node.
#Start-DatabaseAvailabilityGroup -Identity “DAGNAME” -MailboxServer ServerName”
Now the Get-DatabaseAvailabilityGroup -status cmd will gave the below output:
Name : DAGName
Memberservers: {Server1}
Operational Servers {Server1}
Now it is the time for mounting all the databases:
Get-MailboxDatabase | Mount-Database
Voila , The MailboxDatabases were started mounting and the ECP access is restored.
“ Error 432 4.3.2 STOREDRV.Deliver; recipient thread limit exceeded in Exchange 2016
One of our customer complained that the Internal E-mails are not being delivered. When we checked the Queue Viewer we observed the emails are stuck at the queue with the error “Error- 432 4.3.2 STOREDRV.Deliver; recipient thread limit exceeded”
This could be due to large number of emails being sent internally(either to one mailbox / several mailboxes).In our case , the customer’s processing server was down for some days, and when it was restored it started sending all the backlog emails.
To overcome this issue we had to temporarily disable the throttling , by adding the below text in the EdgeTransport.exe.config file(located in the Exchange Bin Folder)
add key=”MailboxDeliveryThrottlingEnabled” value=”False”
(Some articled recommends to add the value in the MSExchangedelivery.exe.config file instead).
Thereafter , you need to restart the MS Exchange Transport & Exchange Mailbox Transport Delivery services.
In case if you don’t completely disable the throttling you could add the below keys.
add key=”RecipientThreadLimit” value=”2″
add key=”MaxMailboxDeliveryPerMdbConnections” value=”3″
Good Luck.
Cannot open mailbox features in ECP
Sometimes you may receive the below error , when you try to access the Mailbox features for a Mailbox via ECP.
In this situation the Exchange server version is 2016 and it happened only on a single mailbox.
After several hours of troubleshooting we came to know that , this could happen if the mailbox is in quarantined state. During this problem user will not be able to access his mailbox neither by Outlook nor any clients)
(You could refer https://www.nucleustechnologies.com/blog/how-to-fix-exchange-mailbox-quarantine-issue/ to have an idea about possible causes for a mailbox to go in to quarantined mode).
To confirm this we ran the below command via the Exchange Powershell
Get-Mailboxstatistics -identity “Test User” | fl Isquarantined
The output was True , now we could confirm his mailbox is in quarantined state.
To disable the quarantine state we ran the below command:
Disable-MailboxQuarantine “Test User”
But , for us unfortunately , the user mailbox goes again to quarantine mode after few minutes. Thus we tried to repair the mailbox using the below command.:
New-MailboxRepairRequest -Mailbox test@test.local -CorruptionType ProvisionedFolder,SearchFolder,AggregateCounts,Folderview
In our situation even the repair was not solving the issue and we had to recreate the mailbox. The reason , I shared the above steps as it could help somebody , without the need for recreating the mailbox.
Good Luck.
How to create an O365 Mailbox when there is no On-Prem Exchange Servers.
In some cases the IT department decomission the On-Prem Exchange Server after migrating the mailboxes to O365. (For detailed steps for uninstallation of Exchange Please refer my previous article : https://vands.pro/2018/11/06/how-to-decomission-on-prem-exchange-server-after-migrating-the-mailboxes-to-o365/.
OK, Coming back to mailbox creation ; All these steps need to be done in the ADUC
+ Create an AD User.
+ Type the email address in the email field.
+ Go to the Account Tab and select the correct domain name.
+ In the Attributes Editor modify the two parameter’s as below
proxyAddresses: SMTP: myemail@email.com
targetAddress: SMTP: myemail@companyname.onmicrosoft.com
+ Either perform a manual sync or wait for the next schedule.
+ After the Sync is completed you will be able to see the user in O365 Portal and need to assign the Exchange License to complete
the mailbox creation.
Credits: https://c7solutions.com/2014/07/creating-mailboxes-in-office-365-when-using-dirsync
How to Decomission On-Prem Exchange Server after migrating the Mailboxes to O365
Few weeks ago, we had project to migrate the Exchange 2010 Mailboxes to O365. After the migration completed we had to remove the On-Prem Exchange Servers(Even though it is not a supported scenario from Microsoft , still you could do this.) This will lead you to to adopt non standard methods when creating new mailboxes(Refer my post https://vands.pro/2018/11/06/how-to-create-an-o365-mailbox-when-there-is-no-on-prem-exchange-servers/ for detailed steps). Also if you still needed an Exchange Server in your premises to avoid complication you could contact MS O365 Support to obtain a free Exchange Hybrid License and install an Exchange Server.
Coming back to the original goal of this article you need to follow the below steps if you have decided to remove the Exchange Servers; Also note that uninstallation of Exchange will not impact the O365 mailboxes in any manner.
+ By this time you should have changed the MX and related DNS records on
the on-prem DNS and Public DNS Servers.
+ Set the AD Internal URL to null
Get-ClientAccessServer | Set-ClientAccessServer
-AutoDiscoverServiceInternalUri $Null
+ Then disable the AD Sync(Temporarily)
Set-MsolDirSyncEnabled -EnableDirSync $false
(Get-MsolCompanyInformation).DirectorySynchronizationEnabled – To
view the current status.
+ Open Exchange Shell in one of the Exchange Server(uninstallation should
be started in the order of MBOX,CAS,HUB & Edge)
+ Remove default Public folders
Get-PublicFolder “\” -Recurse -ResultSize:Unlimited |
Remove-PublicFolder -Recurse -ErrorAction:SilentlyContinue
+ Remove system Public folders
Get-PublicFolder “\Non_Ipm_Subtree” -Recurse -ResultSize:Unlimited |
Remove-PublicFolder -Recurse -ErrorAction:SilentlyContinue
+ Remove Offline Address Book
Get-OfflineAddressBook | Remove-OfflineAddressBook
+ Remove send connectors
Get-SendConnector | Remove-SendConnector
+ Remove Public Folder Database
Get-PublicFolderDatabase | Remove-PublicFolderDatabase
+ + Remove Arbitration Mailbox
Get-Mailbox -Arbitration | Disable-Mailbox -Arbitration -DisableLastArbitrationMailboxAllowed
+ Disable / Delete All non migrated Mailboxes
Get-Mailbox | Disable-Mailbox
+ Open CMD as Administrator and Go to the bin folder in the Exchange Installation Folder in your C: drive(Depend on your environment)
setup.com /m:unistall.
+ Once all the Exchange Roles are uninstalled disjoin the servers from the AD.
+ Re-run the AD Sync Tool and remove the Tick on Exchange Hybrid Configuration.
+ Re-enable the AD Sync
Set-MsolDirSyncEnabled -EnableDirSync $false
Credits: https://www.itpromentor.com/remove-hybrid-keep-sync/
Good Luck Guys.
Azure AD Password Sync Error with Event ID 611
Recently we faced an issue with the Password Sync from our On-Prem AD server . The strange issue was when we create/delete an AD object the synchronization completes successfully except for the Password changes. In the event viewer on the AAD connect server we could see event ID 611.
Password synchronization failed for domain: test.com
Details:
System.DirectoryServices.Protocols.LdapException: The operation was aborted because the client side timeout limit was exceeded.
If you are facing the same issue , you need to modify the registy entry on the AAD connect server as below;
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Ldap
Modify the parameter LdapClientIntegrity to 0.
But I am still not sure , is it a Bug or not, If I come across any information about the cause of the error , I will update this post accordingly.
Good Luck.
How to login to Exchange Online Powershell
In most of the cases you may need to login O365 via PowerShell to manage the environments , Then you could use the below cmdlets to initiate the sessions.
+ Set-ExecutionPolicy RemoteSigned
+ $UserCredential = Get-Credential
+ $Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $UserCredential -Authentication Basic -AllowRedirection
+ Import-PSSession $Session -DisableNameChecking
+ Remove-PSSession $Session ( Need to run this to avoid the waiting time as Microsoft has a limit on the session)
That’s it now , you could start using the Powershell cmd’s. Please refer the below MS KB Article for the explanation of these commands, and prerequisites.
https://docs.microsoft.com/en-us/powershell/exchange/exchange-online/connect-to-exchange-online-powershell/connect-to-exchange-online-powershell?view=exchange-ps
In some cases when you try to run Import-Module MSOnline you will be see an error message no cmdlets found. In order to solve this issue you need to install Windows Management Framework 5.1. Once you install this update need to restart the computer. Thereafter you could use the below commands to Manage your environment.
+ $UserCredential = Get-Credential
+ Import-Module MSOnline
+ Connect-MsolService -Credential $UserCredential
+To Disable the Sync
Set-MsolDirSyncEnabled -EnableDirSync $false
+To view the Current status
(Get-MsolCompanyInformation).DirectorySynchronizationEnabled
+ To force a manual Sync after a Change in On-Prem Object.
Import-Module ADSync
Get-ADSyncScheduler
Start-ADSyncCycle -PolicyType Delta
How to descommission the On-Prem Exchange server after the successful migration to O365.
When you performed a cutover or Hybrid migration to O365 , You need to uninstall the On-Prem Exchange Server. Even though MS recommends to keep at least one Exchange On-Prem Server(Does not require any license and special license available for this use cases), Some environments require them to be uninstall in that case we could follow the below steps.(Please note I have not included any screenshots because if you are reading this article you should have well familiarized with O365 Admin / EAC consoles.
- Change the DNS records internally & externally to point it to O365.
- Open EAC on Office365
- Click on Mailflow -> Open Connectors
- Disable or Delete the 2 Connectors ( Both Inbound & Outbound)
- Click on Organization and remove the O365 to Onpremises …config.
- Stop the AD Sync (Set-MsolDirSyncEnabled –EnableDirSync $false)
- Remove all the unwanted or non migrated mailboxes from the On-Prem Server
- Remove the Public Folders
- Remove / Disable Arbitration Mailbox ( Get-Mailbox –Arbitration )
- Optional(Remove OAB)
- Uninstall Exchange
- Re-enable ADSync (Set-MsolDirSyncEnabled –EnableDirSync $true)
How to recover from FWS and DAG Member failure in 2 Node DAG
Hi Folks
Recently we had a situation where one of our customer was affected with a malware and most of his servers became unusable. The impact caused the File Witness Servers( a Domain Controller) and one of the Exchange Node from the 2 Node DAG environment to become instable.
So after studying the impact we decided to do the below ;
- Remove the Failed Node from the DAG and rebuild it from scratch and attach it to the DAG again.
- Change the FWS to another server.
But unfortunately we were not able to proceed as we expected because the cluster service on the remaining node was not able to reach any cluster defined. When I opened the Failover Cluster Manager I was not able to reach or connect it to the DAG Cluster (As it was not able to reach any the Quorum in our case it is the FWS. The same was confirmed by the below command:
- cluster node
This will show the failed node as down and the survived DAG node in Joining state
To overcome the problem you have to restart the cluster without quorum to do that type the below command on the Exchange server
net stop clussvc
net start clussvc /fq
Boom .. everything returned normal with Windows Clustering on the remaining node ( you could verify it with the same command ; cluster node) . I was able to connect it to the DAG cluster via the Windows Clustering Manager.
Now the cluster is restored and I had to move the FWS to another server so I ran the command below which set the new FWS ( Source: https://practical365.com/exchange-server/recovering-a-failed-exchange-2016-database-availability-group-member/)
Set-DatabaseAvailabilityGroup -Identity “DAG-Name” -WitnessDirectory c:\FWS -WitnessServer “New Server Name”
Now we were able to proceed with the remaining steps that is to
– remove the Mailbox Copies from the Failed Server
– Move the Active Mailboxes from the Failed Server to the active Server
The commands I used are
- Get-MailboxDatabaseCopyStatus -Server “Failed Exchange Server Name” | Remove-MailboxDatabaseCopy -Confirm:$false
- Move-ActiveMailboxDatabase “Mailbox Database Name” -ActivateOnServer “Exchange Server Name” -SkipHealthChecks -SkipActiveCopyChecks -SkipClientExperienceChecks -SkipLagChecks -MountDialOverride:BESTEFFORT
Thereafter you could proceed with the remaining steps as mentioned below;
To remove the failed server from the DAG (-ConfigurationOnly switch will execute the command without trying to contact the failed server)
- Remove-DatabaseAvailabilityGroupServer -Identity “DAG Name” -MailboxServer “Failed Exchange Server Name” -ConfigurationOnly
Thereafter you need to remove the failed server from the Cluster Group to do that;
- Get-ClusterNode “Failed Exchange Server Name” | Remove-ClusterNode
Once you are able to pass through all the steps , the only thing left is to rejoin the Failed Exchange Server to the same DAG. (Refer Article:https://practical365.com/exchange-server/recovering-a-failed-exchange-2016-database-availability-group-member/)
Hope this will help someone in a similar situation.
Good Luck
Muralee
