Nutanix AOS Upgrade Tips
Recently we were upgrading our Nutanix Cluster which was running an AOS version 4.5.2.3 to the latest 5.9. The process was seamless and non interruptive. I have listed the commands we have used along with Nutanix Engineer during the process for future reference.
Initial Check prior to AOS Upgrade
- ncli cluster info
- ncli host ls
- ncli ru ls
- ncli ms ls
- ncc –version
- cluster status | grep -v UP
- nodetool -h 0 ring | grep -i normal | wc -l
- svmips | wc -w
Once the output of above checks are fine Use the Software Upgrade feature from PRISM to upgrade the AOS.
To check the upgrade / pre-upgrade status and on which node is being picked up.& Confirm the versions after upgrade
- allssh ls -ltra ~/data/logs | grep -i preupgrade
- tail -F ~/data/logs/preupgrade.out
- use upgrade_status (to verify the status , less verbose mode)
- ncli –version
- stargate –version
- watch -d genesis status (to check the services status after the CVM reboot)
Optional: To delete the previously uploaded ISO
- cd ~/software_downloads/nos ( use it with allssh to run it on all the CVM’s)
Finally after AOS upgrade sometime the curator replication process and it takes some time to complete . Until it completes we cannot proceed with the other update , thus you could check via the below command;
- curator_cli get_under_replication_info
How to Capture & Analyze Network Traffic on ESXi
Being an ESXI Implementer or an Administrator , you may come across some situations where you need to make your hands dirty 🙂 , with deep network troubleshooting. I had a similar situation few months ago , which I would like to share it in this post.
We deployed the Horizon View (for VDI) in one of our customer’s ESXi Cluster ( 8 Nodes) environment, The Desktop users were complaining about they were not able to specific network .
Thus to further investigate we swapped the Physical Adapter to the on-board BroadCom cards (1Gps). Then we were able to re-establish the network. We thought to engage the VMware Support with the intention to find out the root cause and get a permanent fix. The VMware support was pretty awesome and they were able to nail it very quickly.
First they used the two built-in commands on ESXI , which are
- pktcap-uw (To capture the Network Packets)
- tcpdump-uw ( To read the captured Packets)
They ran the below commands on both the NIC cards to initially capture the traffic.
- pktcap-uw –uplink vmnic0 –dir 0 –mac 00:00:00:00:00:00 —vlan 18 -o /tmp/f.pcap
uplink – Name of the VMnic
dir – 0 means RX Traffic
mac – MAC address of the machine which you are troubleshooting
vlan – The VLAN ID
Thereafter we read the output of the above command using
- tcpdump-uw -ner /tmp/f.pcap
By comparing the output from both NIC’s we were able to narrow down the problem to the Mellanox cards. when tagged traffic passed by on a Mellanox Network Card (10 Gbps), the reply packet was not being tagged with the proper VLAN ID causing disruption to the network traffic.
Good Luck
Muralee
Veeam Backup Repository Sizing
Folks who wants to size the Backup Repository for their Veeam Deployment , could make use of the online sizing tool below for their calculations.
It is very user friendly and output is much descriptive.
Good Luck.
How to create an O365 Mailbox when there is no On-Prem Exchange Servers.
In some cases the IT department decomission the On-Prem Exchange Server after migrating the mailboxes to O365. (For detailed steps for uninstallation of Exchange Please refer my previous article : https://vands.pro/2018/11/06/how-to-decomission-on-prem-exchange-server-after-migrating-the-mailboxes-to-o365/.
OK, Coming back to mailbox creation ; All these steps need to be done in the ADUC
+ Create an AD User.
+ Type the email address in the email field.
+ Go to the Account Tab and select the correct domain name.
+ In the Attributes Editor modify the two parameter’s as below
proxyAddresses: SMTP: myemail@email.com
targetAddress: SMTP: myemail@companyname.onmicrosoft.com
+ Either perform a manual sync or wait for the next schedule.
+ After the Sync is completed you will be able to see the user in O365 Portal and need to assign the Exchange License to complete
the mailbox creation.
Credits: https://c7solutions.com/2014/07/creating-mailboxes-in-office-365-when-using-dirsync
How to Decomission On-Prem Exchange Server after migrating the Mailboxes to O365
Few weeks ago, we had project to migrate the Exchange 2010 Mailboxes to O365. After the migration completed we had to remove the On-Prem Exchange Servers(Even though it is not a supported scenario from Microsoft , still you could do this.) This will lead you to to adopt non standard methods when creating new mailboxes(Refer my post https://vands.pro/2018/11/06/how-to-create-an-o365-mailbox-when-there-is-no-on-prem-exchange-servers/ for detailed steps). Also if you still needed an Exchange Server in your premises to avoid complication you could contact MS O365 Support to obtain a free Exchange Hybrid License and install an Exchange Server.
Coming back to the original goal of this article you need to follow the below steps if you have decided to remove the Exchange Servers; Also note that uninstallation of Exchange will not impact the O365 mailboxes in any manner.
+ By this time you should have changed the MX and related DNS records on
the on-prem DNS and Public DNS Servers.
+ Set the AD Internal URL to null
Get-ClientAccessServer | Set-ClientAccessServer
-AutoDiscoverServiceInternalUri $Null
+ Then disable the AD Sync(Temporarily)
Set-MsolDirSyncEnabled -EnableDirSync $false
(Get-MsolCompanyInformation).DirectorySynchronizationEnabled – To
view the current status.
+ Open Exchange Shell in one of the Exchange Server(uninstallation should
be started in the order of MBOX,CAS,HUB & Edge)
+ Remove default Public folders
Get-PublicFolder “\” -Recurse -ResultSize:Unlimited |
Remove-PublicFolder -Recurse -ErrorAction:SilentlyContinue
+ Remove system Public folders
Get-PublicFolder “\Non_Ipm_Subtree” -Recurse -ResultSize:Unlimited |
Remove-PublicFolder -Recurse -ErrorAction:SilentlyContinue
+ Remove Offline Address Book
Get-OfflineAddressBook | Remove-OfflineAddressBook
+ Remove send connectors
Get-SendConnector | Remove-SendConnector
+ Remove Public Folder Database
Get-PublicFolderDatabase | Remove-PublicFolderDatabase
+ + Remove Arbitration Mailbox
Get-Mailbox -Arbitration | Disable-Mailbox -Arbitration -DisableLastArbitrationMailboxAllowed
+ Disable / Delete All non migrated Mailboxes
Get-Mailbox | Disable-Mailbox
+ Open CMD as Administrator and Go to the bin folder in the Exchange Installation Folder in your C: drive(Depend on your environment)
setup.com /m:unistall.
+ Once all the Exchange Roles are uninstalled disjoin the servers from the AD.
+ Re-run the AD Sync Tool and remove the Tick on Exchange Hybrid Configuration.
+ Re-enable the AD Sync
Set-MsolDirSyncEnabled -EnableDirSync $false
Credits: https://www.itpromentor.com/remove-hybrid-keep-sync/
Good Luck Guys.
Azure AD Password Sync Error with Event ID 611
Recently we faced an issue with the Password Sync from our On-Prem AD server . The strange issue was when we create/delete an AD object the synchronization completes successfully except for the Password changes. In the event viewer on the AAD connect server we could see event ID 611.
Password synchronization failed for domain: test.com
Details:
System.DirectoryServices.Protocols.LdapException: The operation was aborted because the client side timeout limit was exceeded.
If you are facing the same issue , you need to modify the registy entry on the AAD connect server as below;
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Ldap
Modify the parameter LdapClientIntegrity to 0.
But I am still not sure , is it a Bug or not, If I come across any information about the cause of the error , I will update this post accordingly.
Good Luck.
How to login to Exchange Online Powershell
In most of the cases you may need to login O365 via PowerShell to manage the environments , Then you could use the below cmdlets to initiate the sessions.
+ Set-ExecutionPolicy RemoteSigned
+ $UserCredential = Get-Credential
+ $Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $UserCredential -Authentication Basic -AllowRedirection
+ Import-PSSession $Session -DisableNameChecking
+ Remove-PSSession $Session ( Need to run this to avoid the waiting time as Microsoft has a limit on the session)
That’s it now , you could start using the Powershell cmd’s. Please refer the below MS KB Article for the explanation of these commands, and prerequisites.
https://docs.microsoft.com/en-us/powershell/exchange/exchange-online/connect-to-exchange-online-powershell/connect-to-exchange-online-powershell?view=exchange-ps
In some cases when you try to run Import-Module MSOnline you will be see an error message no cmdlets found. In order to solve this issue you need to install Windows Management Framework 5.1. Once you install this update need to restart the computer. Thereafter you could use the below commands to Manage your environment.
+ $UserCredential = Get-Credential
+ Import-Module MSOnline
+ Connect-MsolService -Credential $UserCredential
+To Disable the Sync
Set-MsolDirSyncEnabled -EnableDirSync $false
+To view the Current status
(Get-MsolCompanyInformation).DirectorySynchronizationEnabled
+ To force a manual Sync after a Change in On-Prem Object.
Import-Module ADSync
Get-ADSyncScheduler
Start-ADSyncCycle -PolicyType Delta
How to descommission the On-Prem Exchange server after the successful migration to O365.
When you performed a cutover or Hybrid migration to O365 , You need to uninstall the On-Prem Exchange Server. Even though MS recommends to keep at least one Exchange On-Prem Server(Does not require any license and special license available for this use cases), Some environments require them to be uninstall in that case we could follow the below steps.(Please note I have not included any screenshots because if you are reading this article you should have well familiarized with O365 Admin / EAC consoles.
- Change the DNS records internally & externally to point it to O365.
- Open EAC on Office365
- Click on Mailflow -> Open Connectors
- Disable or Delete the 2 Connectors ( Both Inbound & Outbound)
- Click on Organization and remove the O365 to Onpremises …config.
- Stop the AD Sync (Set-MsolDirSyncEnabled –EnableDirSync $false)
- Remove all the unwanted or non migrated mailboxes from the On-Prem Server
- Remove the Public Folders
- Remove / Disable Arbitration Mailbox ( Get-Mailbox –Arbitration )
- Optional(Remove OAB)
- Uninstall Exchange
- Re-enable ADSync (Set-MsolDirSyncEnabled –EnableDirSync $true)
How to expire Veeam Backup Jobs
When using Veeam B&R , if you had to face a situation where you need to expire the old backups to free up some disk space you need to follow slightly a different approach. As mostly in other cases (especially VERITAS) you could simply change the retention period to a lower value and need to restart the services , you could notice the backup files have been disappeared. In Veeam you need to go through the below steps.
- Remove the backup files manually (recommended to clear the files created by the last incremental jobs until you reach to the last full backup).
- Reduce the backup pointer to a lower value in your backup job.
- Then you need to start the jobs manually or wait for the next schedule.
VCSA6.7 and Veeam B&R Issues
Recently we were upgrading our ESXi Infrastructure from ESXi 6.0 to 6.7.During this process we kicked off the migration process with our VCenter Server 6.0 with the intention to move it to a VCSA 6.7 . Everything went well . But on the following day we started receiving Backup job failure alerts from Veeam Server.
After few google searches we came to know that the Veeam B&R need to be upgraded with U3 to be fully be compatible with Photon based VCSA 6.7 .
Good Luck with your VSphere Upgrades.
Hope you like it.. 