Posts filed under ‘Exchange and O365’
Azure AD Password Sync Error with Event ID 611
Recently we faced an issue with the Password Sync from our On-Prem AD server . The strange issue was when we create/delete an AD object the synchronization completes successfully except for the Password changes. In the event viewer on the AAD connect server we could see event ID 611.
Password synchronization failed for domain: test.com
Details:
System.DirectoryServices.Protocols.LdapException: The operation was aborted because the client side timeout limit was exceeded.
If you are facing the same issue , you need to modify the registy entry on the AAD connect server as below;
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Ldap
Modify the parameter LdapClientIntegrity to 0.
But I am still not sure , is it a Bug or not, If I come across any information about the cause of the error , I will update this post accordingly.
Good Luck.
How to login to Exchange Online Powershell
In most of the cases you may need to login O365 via PowerShell to manage the environments , Then you could use the below cmdlets to initiate the sessions.
+ Set-ExecutionPolicy RemoteSigned
+ $UserCredential = Get-Credential
+ $Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $UserCredential -Authentication Basic -AllowRedirection
+ Import-PSSession $Session -DisableNameChecking
+ Remove-PSSession $Session ( Need to run this to avoid the waiting time as Microsoft has a limit on the session)
That’s it now , you could start using the Powershell cmd’s. Please refer the below MS KB Article for the explanation of these commands, and prerequisites.
https://docs.microsoft.com/en-us/powershell/exchange/exchange-online/connect-to-exchange-online-powershell/connect-to-exchange-online-powershell?view=exchange-ps
In some cases when you try to run Import-Module MSOnline you will be see an error message no cmdlets found. In order to solve this issue you need to install Windows Management Framework 5.1. Once you install this update need to restart the computer. Thereafter you could use the below commands to Manage your environment.
+ $UserCredential = Get-Credential
+ Import-Module MSOnline
+ Connect-MsolService -Credential $UserCredential
+To Disable the Sync
Set-MsolDirSyncEnabled -EnableDirSync $false
+To view the Current status
(Get-MsolCompanyInformation).DirectorySynchronizationEnabled
+ To force a manual Sync after a Change in On-Prem Object.
Import-Module ADSync
Get-ADSyncScheduler
Start-ADSyncCycle -PolicyType Delta
How to descommission the On-Prem Exchange server after the successful migration to O365.
When you performed a cutover or Hybrid migration to O365 , You need to uninstall the On-Prem Exchange Server. Even though MS recommends to keep at least one Exchange On-Prem Server(Does not require any license and special license available for this use cases), Some environments require them to be uninstall in that case we could follow the below steps.(Please note I have not included any screenshots because if you are reading this article you should have well familiarized with O365 Admin / EAC consoles.
- Change the DNS records internally & externally to point it to O365.
- Open EAC on Office365
- Click on Mailflow -> Open Connectors
- Disable or Delete the 2 Connectors ( Both Inbound & Outbound)
- Click on Organization and remove the O365 to Onpremises …config.
- Stop the AD Sync (Set-MsolDirSyncEnabled –EnableDirSync $false)
- Remove all the unwanted or non migrated mailboxes from the On-Prem Server
- Remove the Public Folders
- Remove / Disable Arbitration Mailbox ( Get-Mailbox –Arbitration )
- Optional(Remove OAB)
- Uninstall Exchange
- Re-enable ADSync (Set-MsolDirSyncEnabled –EnableDirSync $true)
How to recover from FWS and DAG Member failure in 2 Node DAG
Hi Folks
Recently we had a situation where one of our customer was affected with a malware and most of his servers became unusable. The impact caused the File Witness Servers( a Domain Controller) and one of the Exchange Node from the 2 Node DAG environment to become instable.
So after studying the impact we decided to do the below ;
- Remove the Failed Node from the DAG and rebuild it from scratch and attach it to the DAG again.
- Change the FWS to another server.
But unfortunately we were not able to proceed as we expected because the cluster service on the remaining node was not able to reach any cluster defined. When I opened the Failover Cluster Manager I was not able to reach or connect it to the DAG Cluster (As it was not able to reach any the Quorum in our case it is the FWS. The same was confirmed by the below command:
- cluster node
This will show the failed node as down and the survived DAG node in Joining state
To overcome the problem you have to restart the cluster without quorum to do that type the below command on the Exchange server
net stop clussvc
net start clussvc /fq
Boom .. everything returned normal with Windows Clustering on the remaining node ( you could verify it with the same command ; cluster node) . I was able to connect it to the DAG cluster via the Windows Clustering Manager.
Now the cluster is restored and I had to move the FWS to another server so I ran the command below which set the new FWS ( Source: https://practical365.com/exchange-server/recovering-a-failed-exchange-2016-database-availability-group-member/)
Set-DatabaseAvailabilityGroup -Identity “DAG-Name” -WitnessDirectory c:\FWS -WitnessServer “New Server Name”
Now we were able to proceed with the remaining steps that is to
– remove the Mailbox Copies from the Failed Server
– Move the Active Mailboxes from the Failed Server to the active Server
The commands I used are
- Get-MailboxDatabaseCopyStatus -Server “Failed Exchange Server Name” | Remove-MailboxDatabaseCopy -Confirm:$false
- Move-ActiveMailboxDatabase “Mailbox Database Name” -ActivateOnServer “Exchange Server Name” -SkipHealthChecks -SkipActiveCopyChecks -SkipClientExperienceChecks -SkipLagChecks -MountDialOverride:BESTEFFORT
Thereafter you could proceed with the remaining steps as mentioned below;
To remove the failed server from the DAG (-ConfigurationOnly switch will execute the command without trying to contact the failed server)
- Remove-DatabaseAvailabilityGroupServer -Identity “DAG Name” -MailboxServer “Failed Exchange Server Name” -ConfigurationOnly
Thereafter you need to remove the failed server from the Cluster Group to do that;
- Get-ClusterNode “Failed Exchange Server Name” | Remove-ClusterNode
Once you are able to pass through all the steps , the only thing left is to rejoin the Failed Exchange Server to the same DAG. (Refer Article:https://practical365.com/exchange-server/recovering-a-failed-exchange-2016-database-availability-group-member/)
Hope this will help someone in a similar situation.
Good Luck
Muralee
How to Allow Mail Relay on Exchange 2016
Hi All
Being an Exchange Administrator you would often come across requirements to allow mail relay from Internal Application or Some Appliances in order to send emails alerts through your Exchange Server. In this case you need to do the below;
- Create a New Receive Connector with any name E.g: Anonymous Relay
- Select the Role as FrontEnd Transport
- Select the Usage Type as Custom
- Leave the bindings as default.
- In the Remote IP addresses that are allowed to use this connector enter the IP addresses of your appliance of application server which wants to use the Exchange for Mail Relay
Once the above steps are done , you need to run the below PS Commands:
-
Set-ReceiveConnector "Anonymous Relay" -PermissionGroups AnonymousUsers
-
Get-ReceiveConnector "Anonymous Relay" | Add-ADPermission -User "NT AUTHORITY\ANONYMOUS LOGON" -ExtendedRights "Ms-Exch-SMTP-Accept-Any-Recipient"
Source1 :https://technet.microsoft.com/en-us/library/mt668454(v=exchg.160).aspx
Source2:http://exchangeserverpro.com/exchange-2016-smtp-relay-connector/#comment-409194
UPDATE:1 With Exchange 2016 CU3 you will not be able to create this connector through the ECP , thus you need to use the below PS commands:
New-Receiveconnector -Name “Mail Relay” -RemoteIPRange (“Remote IP “) -TransportRole “FrontendTransport” -Bindings (“0.0.0.0:25”) -usage “Custom” -Server “Exchange Server Name”
Exchange 2010 Transport Service starts and stops automatically
Dears
Recently I had a problem in one of my customer sites where they have Exchange 2010 servers running on Windows 2008. The initial call I received is that the Inbound email has stopped. When I checked on the server the C:\ free space was less than 4 GB . After some googling I found out that it could be due to Exchange 2010 Back pressure and disabled the Resource Monitoring as below;
- Stopped Exchange Transport Services.
- Open the EdgeTransport.exe.config in the bin folder in Exchange Installation directory
- Edited the parameter <add key=”EnableResourceMonitoring” value=”true” /> to be as false
- Restarted the Transport services
The Mail flow started again , but unfortunately I received the call again saying that the Transport service has stopped , when I tried to start it manually it stops again. After few hours of troubleshooting we decided to contact MS Professional support as we want to bring up the environment as soon as possible. The MS Team went through several process and finally decided to patch the server with the latest Exchange 2010 Rollup 15 , Voila the services was restored.
I shared this , because there are few good articles with regard to MS Exchange Back pressure and MTA service but none of them was related to my experience so it will help somebody who have the same issue.
GoodLuck
Cross Forest Migration from Exchange 2003 to Exchange 2016 using CodeTwo
Hi All
After a long pause I thought add a new post about a recent project that I worked with .The project was to perform a cross forest migration from Exchange 2003 to Exchange 2016 .We had to bypass the double hop migration ,thus we chose CodeTwo exchange migration to move directly from Exchange 2003 to Exchange 2016.
In this article I have include the steps (not detailed steps as it could be easily found via a simple Google search) which will guide you from Cross Forest Active Directory Migration to Exchange Migration.
NOTE: There was no inter forest mail flow during the migration as we decided to move all the mailboxes over the weekend.
Source Forest:
Windows(2003) , Exchange (2003), Multiple SMTP Domains configured on Exchange 2003
Target Forest:Windows 2012(R2),Exchange(2016), Multiple SMTP Domains configured as the Accepted Domains.
The migration task was broken in to several sequential steps.
1)User Migration (via ADMT)
2)Group Migration(via ADMT)
3)Workstation Migration (via ADMT)
4)Mailbox Migration(via CodeTwo)
Based on the above we started preparing the prerequisites for the steps 1-3(Exchange 2016 was already installed on the Target Domain).
– Raised the Forest Functional level and Domain Functional level to
2003 on the Source Domain Controllers.
– Created conditional forwarding on source and target DNS servers
for both domain names to be resolved vice versa.
– Created two way trust between the source and target forest.
– Created a group called ADMTAdmin(on source) and added the
Domain Admin(Target). Thereafter created a group policy for the
Restricted group and added the ADMT Admin Group(This is to
have local administrator rights on all source forest workstations).
– Created another group policy to disable Windows Firewall.
– Install SQL Express and .NET Framework on the Target DC prior to
installing the ADMT.
– Install Password Export Server on the source Domain Controller
(This is to import the same password).This will require the below
steps.
– Create PAC Key on Target *
(admt key /option:create /sourcedomain:ebyader.com
/keyfile:”c:\PES.pes” /keypassword:*)* (use this extracted key
when installing the PES Application)
– PES service should be run as using target admin account) *
– Finally Add the Administrator use to remote builtin Administrator
group vice versa on each forest..
Once the above are done you can start migrating the users,groups and workstations. The ADMT will prepare the workstations to work on the new domain but accessing the same profile.The Outlook profile still will be pointed to the old exchange server.(In order to avoid the password prompt you could grant Mailbox right for the target AD account to his/her mailbox).
Any issues related to ADMT and the required troubleshooting steps were collected from the website: https://blog.thesysadmins.co.uk/admt-series-1-preparing-active-directory.html
Issues faced at this stage.
Issue 1: Users get blank profile with out any data.
Solution:
Ask the user to log off from the user workstation.
– Run Regedit from your computer. Connect Network Registry to the user workstation.
– Go to HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\ProfileList
– Go through the Profile list and identify the Source account. Copy the value from the ProfileImagePath key.
– Again go through the Profile list and identify the Target account. Paste the ProfileImagePath key value there.
– Restart the user workstation.
The ProfileImagePath key will be same value for both Source and Target user accounts. This ensures both source and target users will receive the same profile which is stored under C:\Documents and Settings\UserName.
Issue2: Outlook users will not be able to access their email as per attached error.
Solution: This is because the autodisover on Exchange 2016 will try to reconfigure the outlook. Thus you could set the autodiscover url to null, and remove the autodiscover DNS record created in the local DNS servers(Target).
Issue3: Failed to change domain affiliation, hr=800704f1 .
Solution: Enabled the Group Policy on the Target DC as
Default Domain Controller Policy-Computer Configuration-Policies-Admin Templates-System-Net Logon-Allow cryptography —NT 4.0
At this stage all the users ,groups and workstation were migrated to the new target forest but still accessing the old exchange mail flow.Thus we reached the climax of this project which is the exchange migration. we started preparing the prerequisites for Exchange Migration using CodeTwo
– Installed Windows 2012 R2 and joined to the Source domain.
– Login with the domain administrator account
– Install CodeTwo (please refer the user manual from CodeTwo , but
in my case I skipped all the prerequisites as I was not intended to
maintain any mail flow between two forests.
– Started Exchange Migration in batches and completed successfully.
– Informed the ISP to change the MX toward the new IP pointing to
Exchange 2016 server.
– Did a rescan (CodeTwo command to copy any missed or new
items from the source mailboxes to new mailboxes.
Issue Faced:When migrating with CodeTwo you will receive ErrorMessageSizeExceeded
Solution: EAC -> Mail Flow-> More Option -> OrganizationTransportSettings -> Change it to more than 100.
http://www.codetwo.com/kb/changing-the-message-size-limit/?sts=1646.
Finally we enabled the autodiscover and recreated the autodiscover DNS record. In order to minimize the work involved to recreate the profile in outlook we pushed the below Group Policy entries as it will enable the user to press “new” button on the popup that display when they double click on outlook icon.
————————————————————————————-
For Outlook 2016:
HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Outlook\AutoDiscover
As a policy:
HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\16.0\Outlook\AutoDiscover
For Outlook 2013:
HKEY_CURRENT_USER\Software\Microsoft\Office\15.0\Outlook\AutoDiscover
As a policy:
HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\15.0\Outlook\AutoDiscover
For Outlook 2010
HKEY_CURRENT_USER\Software\Microsoft\Office\14.0\Outlook\AutoDiscover
As a policy:
HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\14.0\Outlook\AutoDiscover
For Outlook 2007:
HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\Outlook\AutoDiscover
As a policy:
HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\12.0\Outlook\AutoDiscover
2. Once the key is created, right-click in the right pane and click New, click DWORD Value and type in “ZeroConfigExchange” (without the quotes) and then press ENTER.
3. Finally, right-click ZeroConfigExchange, click Modify and then in the Value data box, type 1, and then click OK
————————————————————————————-
In case if the outlook is not prompting create the below entry as well
HKEY_CURRENT_USER\Software\Microsoft\Exchange\Client\Options
String Value: PickLogonProfile
Data: 1
All these above Registry entries can be pushed via GroupPolicy.
Hope I have covered all the required steps in brief and anybody who wants to perform a cross forest migration using Code Two , can refer this article as guideline for their project.The other fact which I decided to write this article is that I did not find any single article/post on the Internet which covering all the cross forestmigration processes staring from Active Directory Objects to Mailboxes.
Cheers !!
Muralee
How to perform a Local Mailbox Move Request on Exchange 2013
Dears
I have prepared the below screen cast to guide you through the steps of moving an Active Mailbox.
1) Create a new Move Request via Exchange Shell (Orange2 Mailbox has been moved to MailDB1 Database)
2) Initialize or Start the Move Request
3) Check the Status of the Request.
How to create Exchange 2013 DAG and Test DAG Failover
Dear Folks
Today I decided to write a post explaining Exchange 2013 DAG Switchover/Switchback and Failover/Failback because I could not find a proper or an consolidated article in the Internet which explains the required steps to install and test the DAG.
This article will be divided in to 3 sections
Step 1 – Installation of Exchange Prerequisites and Exchange Installation
Step 2 – DAG creation
Step 3 – Testing (Switchover/Switchback and Failover/Failback)
Environment
1 DC (Windows 2012 R2 OS, Domain: Test.local)
2 Exchange Servers(Windows 2012 R2 OS, Exchange 2013 CU6)
Step 1 – Installation of Exchange Prerequisites and Exchange Installation
In this series we will walk you through the process of performing a clean installation of Exchange 2013 in single server.(Windows OS installation will not be covered)
> Begin with Installing the AD DS and AD LDS Roles to prepare the AD forest for the Exchange 2013 deployment.
> Thereafter open an elevated command prompt run the prepare schema command
> Then start with the AD Preparation in this you need to mention the Exchange Server Organization Name
> Now you could prepare the domain.
> We need to install the below prerequisite based on exchange roles that will be installed since my deployment is single server role (cas and mbox together)
Install-WindowsFeature AS-HTTP-Activation, Desktop-Experience, NET-Framework-45-Features, RPC-over-HTTP-proxy, RSAT-Clustering, RSAT-Clustering-CmdInterface, Web-Mgmt-Console, WAS-Process-Model, Web-Asp-Net45, Web-Basic-Auth, Web-Client-Auth, Web-Digest-Auth, Web-Dir-Browsing, Web-Dyn-Compression, Web-Http-Errors, Web-Http-Logging, Web-Http-Redirect, Web-Http-Tracing, Web-ISAPI-Ext, Web-ISAPI-Filter, Web-Lgcy-Mgmt-Console, Web-Metabase, Web-Mgmt-Console, Web-Mgmt-Service, Web-Net-Ext45, Web-Request-Monitor, Web-Server, Web-Stat-Compression, Web-Static-Content, Web-Windows-Auth, Web-WMI, Windows-Identity-Foundation ( For other roles please refer Technet Article http://technet.microsoft.com/en-us/library/bb691354%28v=exchg.150%29.aspx )
> Next we install the below mentioned components in the mentioned order as
1.Net Framework 4.5.2 (Only for CU7 , In case CU6 the required .NET Framework is pre installed)
2.Microsoft Unified Communication Managed API 4.0,Core Runtime 64-bit
> Finally begin the Exchange Installation by running setup.exe and follow the screen (selected option may vary based on your environment requirements) below are some of the customized options as per my LAB
Step 2 – DAG creation
Now we have installed the Exchange Servers. We will begin with the DAG creation.
> As a prerequisite the Exchange Installation or the Database and Log location should not to be identical in my case it was not
hence I need to move the Database and Log Files.
> Thereafter we need to pre create the CNO (Cluster Name Object) when we deploy DAG on Windows 2012 or Windows 2012 R2
1) Create a new computer object via Active Directory Users and Computer Snap-in
2) Disable the Computer Account created above and Press ” Yes” on the Prompt.
3) Enable “Advanced Features”
4) Right Click the Computer Object Created and add one of the Exchange Mailbox server and grant Full control
5) Add the File Witness Server in my case the Domain Controller in Exchange Trusted Sub System Group
6) Add the Exchange Trust Subsystem group on the Local Administrator Group of the File Witness Server.
Everything is done now we will begin with the DAG creation
7) We will create a DAG with the below parameteres
DAG Name : – DAG
IP Address :- 192.168.252.131
FSW Server : dc.test.local
NOTE:- I forgot the step 6 in my LAB and the FSW was not created , hence I need to re set the FSW property on my DAG after completing the Step6
8) Add the Mailbox Servers to the DAG.
9) Verify the Cluster Resource to confirm whether the above commands are executed correctly and the required cluster resources has been created.
10) Now we are going to make the Database Copies ,hence check the Database and Log Path on the Exchange Server.
11) Add the First Copy
12) Verify the Database Copy Status and repeat the steps for the other databases.
(If you notice any status other the Healthy you could give some time or simply restart the Information Store Service.)
Step 3 – Testing (Switchover/Switchback and Failover/Failback)
1) Now we have successfully created the DAG we will go through the testing .( While testing I prefer if you could open an E-Mail Client to test the email flow between 2 accounts)
We will perform a database switchover which is a planned activity and can be done via ECP , Hence login to ECP –> Servers –> Databases
Select the passive copy of the database on the Exchange2 server and click on activate it.
Test the e-mail flow .
2) Now we will test the automatic service recovery a beautiful feature of DAG.
In the below snapshot we will stop the IISAdmin service and confirm that Exchange will restart the service automatically
3) Now we will perform a Failover which is an unplanned activity,In order to do this we will crash or stop the process Microsoft.Exchange.Store.Worker
on theExchange2 Server
Once it done you will notice on the ECP that the Passive Database is activated automatically on the Exchange1 Server and now you could test the e-mail flow.
Done.
How to move the database and transaction log files on Exchange 2010 DAG servers
Hi Folks
In case if you want move the transaction logs and the exchange database to a new location on the disk , you have to follow certain guidelines to avoid exchange nightmare.
But I would like to recommend you do this before you establish the DAG ,
In-case if you have forgotten to do this before establishing the DAG you should do the below steps on the exchange servers that are part of the DAG in the mentioned order
– Disable the Circular logging ( In-case if it was enabled before)
– Then you need to remove the database copies on each database per each each server
Remove-MailboxDatabaseCopy -Identity DB1\MBX1 -Confirm:$False (This Eg:- will remove the copy of the database DB1 on the MBX1 server)
– Create the necessary folder on the new location
– Move the Mailbox databases as below
* Open Exchange Management Console
* Navigate to Organization Configuration -> Mailbox -> Database Management -> Select the Mailbox Database
* Right click on it select “Move Database Path” and enter the new locations
– On each DAG member server you will find the passive copy of the database for the other member servers,normally stored on the original location where the Active Mailbox is
stored ,You need to move these files to the new location , Keep in mind when you are moving the Passive Copies of the database it should be stored on the same drive letter where
the Active Mailbox database is stored on the active member.
Eg:- Server1 host the mailbox DB1 on C:\DB Folder , hence you need to maintain the same drive letter on the remote server(Server2) where it will host the passive copy for the DB1
Do the same on all the DAG members
– Now add the Mailbox database copies for all the servers.
Add-MailboxDatabaseCopy -Identity DB1 -MailboxServer Server2 -ActivationPreference 2
– On each server where the mailbox was moved run the below commands Net stop msftesql-Exchange Net start MSExchangeSearch
Tip:- I came across an issue where when I changed the drive letter on the server where it holds the mailbox , the database dismounted,therefore if you face the same issue use the
Move-Database command with the -ConfigurationOnly Parameter.