Posts filed under ‘Exchange and O365’

The Microsoft Exchange Replication service couldn’t find a valid configuration for database ‘mydb’ on server ‘server1’. Error: An Active Manager operation failed. Error: The active copy for database could not be determined:

Recently one of our customer had an Exchange Server outage. Whereas one of the node from 2 node DAG was not able to communicate with other server. We tried to rebooting the server but no luck . The symptoms were

– Cluster service was not starting.
– On the other node FCM was not able to connect to the DAG cluster.
– Unable to open the ECP/OWA.
– Outlook / Mobile Users cannot access their mailboxes.

The environment consist of 2 Exchange Server 2016 & DAG.

So we started with removing the failed node by typing the below commands in the healthy exchange server:
# Open Exchange PowerShell
#Get-ClusterNode -Name node1 | Remove-Clusternode
# net stop clussvc
# net start clussvc 
# Remove-DatabaseAvailabilityGroupServer -Identity “DAG Name” -MailboxServer “Failed Exchange Server Name” -ConfigurationOnly
# Get-ClusterNode “Failed Exchange Server Name” | Remove-ClusterNode

However , still no luck. Then we tried to remove the mailbox database copies form the failed node(I believe the copies were active on the second server) we got a new error stating that “Mailbox databse copies cannot be disabled on the Databases with circular logging enabled. So we had to remove the circular logging and then delete the database copies.

#Get-MailboxDatabse | Set-MailboxDatabase -CircularLoggingEnabled $False
# Get-MailboxDatabaseCopy -Identity “DatabaseName” |Remove-MailboxDatabaseCopy -Identity “DatabaseName\ServerName”

Now it is the time for mounting the databses: When we ran the below command

#Get-MailboxDatabase | Mount-Database

We received the below error”

Failed to mount database “———-“. Error: An Active Manager operation failed. Error: An Active Manager operation
encountered an error. To perform this operation, the server must be a member of a database availability group, and the
database availability group must have quorum. Error: Automount consensus not reached (Reason: FSW boot time did not
match (FSW-Remote: 2020-10-01T11:37:09.4930830Z FSW-Reg: 0001-01-01T00:00:00.0000000)). [Server:———-]
+ CategoryInfo : InvalidOperation: (—-:ADObjectId) [Mount-Database], InvalidOperationException
+ FullyQualifiedErrorId : [Server=————,RequestId=5022acdd-0c48-4584-b2eb-1d0a2c692f0d,TimeStamp=10/7/2020
11:09:14 AM] [FailureCategory=Cmdlet-InvalidOperationException] BCF4F25E,Microsoft.Exchange.Management.SystemConfi
gurationTasks.MountDatabase

Here , we wanted to check the DAG status

#Get-DatabaseAvailabilityGroup -status

Output :

WARNING: Unable to get Primary Active Manager information due to an Active Manager call failure. Error: An Active
Manager operation failed. Error: An Active Manager operation encountered an error. To perform this operation, the
server must be a member of a database availability group, and the database availability group must have quorum. Error:
Automount consensus not reached (Reason: FSW boot time did not match (FSW-Remote: 2020-10-01T11:37:09.4930830Z FSW-Reg:
0001-01-01T00:00:00.0000000)). [Server: ———-]

Name : DAGName
Memberservers: {Server1}
Operational Servers {}

The output confirmed that there a no operatinal servers. So we had to restart the DAG with the surviving node.

#Start-DatabaseAvailabilityGroup -Identity “DAGNAME” -MailboxServer ServerName”

Now the Get-DatabaseAvailabilityGroup -status cmd will gave the below output:

Name : DAGName
Memberservers: {Server1}
Operational Servers {Server1}

Now it is the time for mounting all the databases:

Get-MailboxDatabase | Mount-Database

Voila , The MailboxDatabases were started mounting and the ECP access is restored.

October 7, 2020 at 3:39 pm Leave a comment

“ Error 432 4.3.2 STOREDRV.Deliver; recipient thread limit exceeded in Exchange 2016

One of our customer complained that the Internal E-mails are not being delivered. When we checked the Queue Viewer we observed the emails are stuck at the queue with the error “Error- 432 4.3.2 STOREDRV.Deliver; recipient thread limit exceeded”

This could be due to large number of emails being sent internally(either to one mailbox / several mailboxes).In our case , the customer’s processing server was down for some days, and when it was restored it started sending all the backlog emails.

To overcome this issue we had to temporarily disable the throttling , by adding the below text in the EdgeTransport.exe.config file(located in the Exchange Bin Folder)

add key=”MailboxDeliveryThrottlingEnabled” value=”False”

(Some articled recommends to add the value in the MSExchangedelivery.exe.config file instead).

Thereafter , you need to restart the MS Exchange Transport & Exchange Mailbox Transport Delivery services.

In case if you don’t completely disable the throttling you could add the below keys.

add key=”RecipientThreadLimit” value=”2″
add key=”MaxMailboxDeliveryPerMdbConnections” value=”3″

Good Luck.

September 17, 2020 at 3:33 pm Leave a comment

Cannot open mailbox features in ECP

Sometimes you may receive the below error , when you try to access the Mailbox features for a Mailbox via ECP.

In this situation the Exchange server version is 2016 and it happened only on a single mailbox.
After several hours of troubleshooting we came to know that , this could happen if the mailbox is in quarantined state. During this problem user will not be able to access his mailbox neither by Outlook nor any clients)
(You could refer https://www.nucleustechnologies.com/blog/how-to-fix-exchange-mailbox-quarantine-issue/ to have an idea about possible causes for a mailbox to go in to quarantined mode).

To confirm this we ran the below command via the Exchange Powershell

Get-Mailboxstatistics -identity “Test User” | fl Isquarantined

The output was True , now we could confirm his mailbox is in quarantined state.

To disable the quarantine state we ran the below command:

Disable-MailboxQuarantine “Test User”

But , for us unfortunately , the user mailbox goes again to quarantine mode after few minutes. Thus we tried to repair the mailbox using the below command.:

New-MailboxRepairRequest -Mailbox test@test.local -CorruptionType ProvisionedFolder,SearchFolder,AggregateCounts,Folderview

In our situation even the repair was not solving the issue and we had to recreate the mailbox. The reason , I shared the above steps as it could help somebody , without the need for recreating the mailbox.

Good Luck.


September 1, 2019 at 12:07 pm Leave a comment

Choose the right authentication method for your Azure Active Directory hybrid identity solution

Choosing the right authentication method , during the new implementation of Hybrid Cloud Infrastructure or Migrating an existing On-Premise towards Cloud with the intention to have a  Hybrid Solution is really mandatory as it will help to avoid any disruptive situations. It has to be a well informed plan and execution process. The decision should be made only on valid compelling reasons. However, Microsoft have published an article in this regard and it can be accessed via https://aka.ms/auth-options.

I would recommend to refer this article as it could be an handy guide to guide the customers and other stakeholders during a project.

June 24, 2019 at 12:09 pm Leave a comment

How to create an O365 Mailbox when there is no On-Prem Exchange Servers.

In some cases the IT department decomission the On-Prem Exchange Server after migrating the mailboxes to O365. (For detailed steps for uninstallation of Exchange Please refer my previous article : https://vands.pro/2018/11/06/how-to-decomission-on-prem-exchange-server-after-migrating-the-mailboxes-to-o365/.

OK, Coming back to mailbox creation ; All these steps need to be done in the ADUC

 

+ Create an AD User.

+ Type the email address in the email field.

+ Go to the Account Tab and select the correct domain name.

+ In the Attributes Editor  modify the two parameter’s as below

proxyAddresses: SMTP: myemail@email.com

targetAddress: SMTP: myemail@companyname.onmicrosoft.com

+ Either perform a manual sync or wait for the next schedule.

+ After the Sync is completed you will be able to see the user in O365 Portal and need to assign the Exchange License to complete
the mailbox creation.

Credits: https://c7solutions.com/2014/07/creating-mailboxes-in-office-365-when-using-dirsync

 

November 6, 2018 at 11:45 am 1 comment

How to Decomission On-Prem Exchange Server after migrating the Mailboxes to O365

Few weeks ago, we had project to migrate the Exchange 2010 Mailboxes to O365. After the  migration completed  we had to remove the On-Prem Exchange Servers(Even though it is not a supported scenario from Microsoft , still you could do this.) This will lead you to to adopt non standard methods when creating new mailboxes(Refer my post https://vands.pro/2018/11/06/how-to-create-an-o365-mailbox-when-there-is-no-on-prem-exchange-servers/ for detailed steps). Also if you still needed an Exchange Server in your premises to avoid complication you could contact MS O365 Support to obtain a free Exchange Hybrid License and install an Exchange Server.

Coming back to the original goal of this article you need to follow the below steps if you have decided to remove the Exchange Servers; Also note that uninstallation of Exchange will not impact the O365 mailboxes in any manner.

+ By this time you should have changed the MX and related DNS records on 
    the on-prem DNS and Public DNS Servers.

+ Set the AD Internal URL to null
  Get-ClientAccessServer | Set-ClientAccessServer 
  -AutoDiscoverServiceInternalUri $Null

+  Then disable the AD Sync(Temporarily)
    Set-MsolDirSyncEnabled -EnableDirSync $false
    (Get-MsolCompanyInformation).DirectorySynchronizationEnabled – To
    view the current status.

+ Open Exchange Shell in one of the Exchange Server(uninstallation should
  be started in the order of MBOX,CAS,HUB & Edge)

+ Remove default Public folders

Get-PublicFolder “\” -Recurse -ResultSize:Unlimited |
Remove-PublicFolder -Recurse -ErrorAction:SilentlyContinue

+ Remove system Public folders

Get-PublicFolder “\Non_Ipm_Subtree” -Recurse -ResultSize:Unlimited |
Remove-PublicFolder -Recurse -ErrorAction:SilentlyContinue

+ Remove Offline Address Book

Get-OfflineAddressBook | Remove-OfflineAddressBook

+ Remove send connectors

Get-SendConnector | Remove-SendConnector

+ Remove Public Folder Database

Get-PublicFolderDatabase | Remove-PublicFolderDatabase

+ + Remove Arbitration Mailbox

Get-Mailbox -Arbitration | Disable-Mailbox -Arbitration -DisableLastArbitrationMailboxAllowed

+ Disable / Delete All non migrated Mailboxes

Get-Mailbox | Disable-Mailbox

+ Open CMD as Administrator and Go to the bin folder in the Exchange Installation Folder in your C: drive(Depend on your environment)
   
  setup.com /m:unistall.

+ Once all the Exchange Roles are uninstalled disjoin the servers from the AD.

+ Re-run the AD Sync Tool and remove the Tick on Exchange Hybrid Configuration.

+ Re-enable the AD Sync

  Set-MsolDirSyncEnabled -EnableDirSync $false

 

Credits: https://www.itpromentor.com/remove-hybrid-keep-sync/

Good Luck Guys.

 

 

 

 

 

November 6, 2018 at 11:34 am Leave a comment

Azure AD Password Sync Error with Event ID 611

Recently we faced an issue with the Password Sync from our On-Prem AD server . The strange issue was when we create/delete  an AD object the synchronization completes successfully except for the Password changes. In the event viewer on the AAD connect server we could see event ID 611.

Password synchronization failed for domain: test.com

Details:
System.DirectoryServices.Protocols.LdapException: The operation was aborted because the client side timeout limit was exceeded.

 

If you are facing the same issue , you need to modify the registy entry on the AAD connect server as below;

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Ldap

Modify the parameter LdapClientIntegrity to 0.

But I am still not sure , is it a Bug or not, If I come across any information about the cause of the error , I will update this post accordingly.

Good Luck.

 

October 23, 2018 at 4:06 pm Leave a comment

How to descommission the On-Prem Exchange server after the successful migration to O365.

When you performed a cutover or Hybrid migration to O365 , You need to uninstall the On-Prem Exchange Server. Even though MS recommends to keep at least one Exchange On-Prem Server(Does not require any license and special license available for this use cases), Some environments require them to be uninstall in that case we could follow the below steps.(Please note I have not included any screenshots because if you are reading this article you should have well familiarized with O365 Admin / EAC consoles.

  • Change the DNS records internally & externally to point it to O365.
  • Open EAC on Office365
  • Click on Mailflow -> Open Connectors
  • Disable or Delete the 2 Connectors ( Both Inbound & Outbound)
  • Click on Organization and remove the O365 to Onpremises …config.
  • Stop the AD Sync (Set-MsolDirSyncEnabled –EnableDirSync $false)
  • Remove all the unwanted or non migrated mailboxes from the On-Prem Server
  • Remove the Public Folders
  • Remove / Disable Arbitration Mailbox ( Get-Mailbox –Arbitration )
  • Optional(Remove OAB)
  • Uninstall Exchange
  • Re-enable ADSync (Set-MsolDirSyncEnabled –EnableDirSync $true)

http://galinlab.com/2017/07/06/decommissioning-exchange-after-cutover-migration-with-dirsyncadsync-is-enabled/Credits:

October 21, 2018 at 11:40 am Leave a comment

How to recover from FWS and DAG Member failure in 2 Node DAG

Hi Folks

Recently we had a situation where one of our customer was affected with a malware and  most of his servers became unusable. The impact caused the File Witness Servers( a Domain Controller) and one of the Exchange Node from the 2 Node DAG environment to become instable.

So after studying the impact we decided to do the below ;

  • Remove the Failed Node from the DAG and rebuild it from scratch and attach it to the DAG again.
  • Change the FWS to another server.

But unfortunately we were not able to proceed as we expected because the cluster service on the remaining node was not able to reach any cluster defined.  When I opened the Failover Cluster Manager I was not able to reach or connect it to the DAG Cluster (As it was not able to reach any the Quorum in our case it is the FWS.  The same was confirmed by the below command:

  • cluster node
    This will show the failed node as down and the survived DAG node in Joining state

To overcome the problem  you have to restart the cluster without quorum to do that type the below command on the  Exchange server

net stop clussvc

net start clussvc  /fq

 

Boom ..  everything  returned normal with Windows Clustering on the remaining node ( you could verify it with the same command ;  cluster node) . I was able to connect it to the DAG cluster via the Windows Clustering Manager.

Now the cluster is restored and I had to move the FWS to another server so I ran the command below which set the new FWS ( Source: https://practical365.com/exchange-server/recovering-a-failed-exchange-2016-database-availability-group-member/)

Set-DatabaseAvailabilityGroup -Identity “DAG-Name” -WitnessDirectory c:\FWS -WitnessServer “New Server Name”

Now  we were able to proceed with the remaining steps that is to
– remove the Mailbox Copies from the Failed Server
–  Move the Active Mailboxes from the Failed Server to the active Server

The commands I used are

  • Get-MailboxDatabaseCopyStatus -Server “Failed Exchange Server Name”  | Remove-MailboxDatabaseCopy -Confirm:$false
  • Move-ActiveMailboxDatabase “Mailbox Database Name” -ActivateOnServer “Exchange Server Name”  -SkipHealthChecks -SkipActiveCopyChecks -SkipClientExperienceChecks -SkipLagChecks -MountDialOverride:BESTEFFORT

Thereafter you could proceed with the remaining steps as mentioned below;

To remove the failed server from the DAG (-ConfigurationOnly switch will execute the command without trying to contact the failed server)

  • Remove-DatabaseAvailabilityGroupServer -Identity “DAG Name”  -MailboxServer “Failed Exchange Server Name” -ConfigurationOnly

Thereafter you need to remove the failed server from the Cluster Group to do that;

  • Get-ClusterNode “Failed Exchange Server Name”  | Remove-ClusterNode

Once you are able to pass through all the steps  , the only thing left is to rejoin the Failed Exchange Server to the same DAG. (Refer Article:https://practical365.com/exchange-server/recovering-a-failed-exchange-2016-database-availability-group-member/)

Hope this will help someone in a similar situation.

Good Luck

Muralee

November 21, 2017 at 12:28 pm Leave a comment

Older Posts


Archives

Categories

Follow Hope you like it.. on WordPress.com

Blog Stats

  • 35,624 hits

%d bloggers like this: