Posts filed under ‘VMware’
Cross VCenter VMotion fails with the error “A specified parameter was not correct: path”
Recently , we had to migrate some VM’s from a VSphere 6.7(17499825) to 7.0 U2 cluster. We leveraged the Import VM’s feature that utilizes the Cross VCenter VMotion capability.
The source: VCenter 7.0
The destination: VCenter 6.7
Even though , the required access between the clusters were allowed(based on https://kb.vmware.com/s/article/2106952) , when we tried to trigger the import VM’s there were no process or tasks being generated in both vcenters. However , we noticed a failed task on the VM that we are trying to import in the destination VCenter( under the tasks and events for the particular VM )wiith the message “A specified parameter was not correct: path”.
We started reviewing the hostd.log on both sides but no avail. Finally , out of exhaustion , we tried to changed the source datastore to a different one(In our scenario the datastore names were identical on both sides) and voila the import VM’s got kicked in and completed successfully. Still , I am not sure the cause for this behavior where the import of VM’s failing due to identical datastore names on both sides , so we have opened a case with VMware support to find out the root cause. I will updating this post as soon as I receive the response.
Tip: Also during the import VM’s wizard (3rd step) where we select the desired to host to place the imported VM on the source cluster we were receiving a SSL related error. This was resolved after disabling the SSL inspection on our firewall
How to migrate Windows Cluster VMs (incl. RDMs) from old to new storage with minimal downtime?
Recently , we did a storage upgrade for one of our customer. The VM migration was pretty easy for almost 90% of the workloads. However , the remaining 10 % of the VM’s were having RDM disks ,and for the standalone RDM disks (non clustered workloads) we were able to leverage the Storage VMotion to convert the RDM’s to VMDK and migrate it.
On the other hand , for clustered VM’s , Initially we were planning to utilize the RP4VM to migrate the VM’s(Failover to the Replica). But this approach was not fruitful as we faced a compatibility issue and there was no workaround and we dropped the idea.
During this time our colleague from the Storage Team suggested this alternate plan and we tested it and it was successful. The steps are here as listed.
make a note of the RDM mapping to the VMs (note the SCSI ID assigned to the RDM in the VM configuration)
# shutdown the VMs of that Virtual cluster.
# unmap the RDMs.
# Create the Luns on the destination array (must be at least as large as the source !!!)
# present to ESXhost (rescan, …)
# use Storage vMotion to move the VM to the new datastores.(This is to move the VM with the OS disk and any other non RDM based disks)
# use ESX CLI vmkfstools to copy the RDM content to the new lun and this will automatically create the new RDM vmdk pointer file (destination.vmdk)
vmkfstolls -i <srcdisk.vmdk> -d rdmp:device <destination.vmdk>
Example:
vmkfstools -i TestVM_RDM1.vmdk -d rdmp:/vmfs/devices/disks/vml.02000100006006016044440000f8b164674b51e111565241494420 TestVM_NewRDM.vmdk
==== remap the new LUN as RDMp with the same SCSI ID====
=== don’t forget to set the bus sharing if it disappeared===
Source:
Kudos to EricDeWitte1 (Contributor)
Datastore conflicts with an existing datastore in the datacenter that has the same URL” error in vCenter Server
Hi Guys.
Let me share my recent experience with a VMware Upgrade on a Nutanix Cluster. We planned the upgrade from vSphere 6.7 U3 to vSphere 7.0 U1.
As usual we started the Nutanix LCM upgrade and brought the cluster to the latest Firmware & BIOS drivers.
Thereafter , I started upgrading the ESXi , I used the offline .zip file for the upgrade. I followed the below steps.
- Uploaded the offline bundle VMware-ESXI-7.0U1-16850804-depot.zip to the datastore.
- Ran the below command to findout the profile name
esxcli software sources profile list -d /xxxxxxx/xxxxxx/VMware-ESXI-7.0U1-16850804-depot.zip - Get the profile name from the output of the above command which is ESXi-7.0.1-1685 and used it in command as below
esxcli software profile update -d /xxxxxxx/xxxxxx/VMware-ESXI-7.0U1-16850804-depot.zip -p ESXi-7.0.1-1685
Everything went well and rebooted the server but,
- CVM is shown inaccessible.
- Local datastore was missing from the server.
As adviced by the Nutanix support , restarted the server for the second time. and I was able to see the local datastore. Still we could not exit from the maintenance mode. As it was giving the error “Datastore conflicts with an existing datastore in the datacenter that has the same URL” error in vCenter Server”.
The issue was resolved after following the VMware KB https://kb.vmware.com/s/article/79623.
To worsen up our situation , for some reason the ESXi version got reverted automatically to the previous build ESXi 6.7 after following the steps in the KB. So we decided to check the boot.cfg files in bootbank & altbootbank locations.
#tail -2 /*bootbank/boot.cfg
It was showing only the ESXi 7.0 U1 only.
Again we contacted the VMware support and resolved the issue with the below steps.
Checked whether achi is disabled or not by typing the command on the ESXi shell.
#grep ahci /etc/vmware/esx.conf
/vmkernel/module/vmw_ahci/enabled = “false”
The output shows the achi is disabled and Enabled it by typing
#esxcfg-module -e vmw_ahci
Thereafter ran the upgrade again using the same steps and rebooted the server.Voila we were able to see the correct version. For some reason the default ahci driver provided from VMware has been disabled.
Sources:
https://anthonyspiteri.net/datastore-esxi7-upgrade-supermicro/
After updating vCSA to 6.7 U2 or higher, unable to log into the VAMI page- “Invalid Credentials” or “Unable to Login”
We recently had a situation , where we were not able to login to VAMI Page of the VCSA . As it was continuously providing the unable to login error. However , we know that the password is correct.
When we logged in to the SSH of the VCSA , we noticed that the applmgmt service is not started. We had to manually start it. Then the login to the VAMI got restored. However , when the appliance is rebooted the same issues pops up again. So ,we followed the VMware KB #68149(https://kb.vmware.com/s/article/68149) , but no success.
Finally , we decided to contact the VMware support and the resolution was very quick . It was due to sqlite DB used by the vmware-statsmonitor serivce(In our case , when we reboot the VCSA both the statsmonitor & applmgemt services were not coming up). The DB was reaching around 500 MB in size. The resolution was simple , he moved the DB file to a temporary location and restarted the vmware-statsmonitor service. We noticed a new appliance_stats.sqlite file got created in the same path /var/vmware/applmgmt/.
The DB file is located in /var/vmware/applmgmt/appliance_stats.sqlite.
We rebooted the VCSA and confirmed that we can login to the VAMI successfully.
Please note that this process will remove the previous stats collected on the VCSA.
Good luck.
How to renew vSphere 6.5 & 6.7 certificates.
When the VCenter Certificate is expired , you will be blocked from logging in to the VCenter . However , the Appliance Management will continue to work. Be noted that there a 2 categories of certificates.
- VMware Security Token Service (STS)
- Solution , Machine , Root and Other certificates.
Import Notes:
- You could avoid all these messy steps , had you monitor and check for the
warnings on the VCenter Administration page for Certificate expiry events. - For Windows based VCenter , you can refer the same KB’s mentioned here for the detailed steps.
- You may face an error when uploading the scripts to the VCSA via WinSCP . The Solution is provided in the same KB’s.
- Certificate Manager may fail during the process , you could refer the https://mueller-tech.com/2019/06/28/replacing-expired-certificates/ for the solution.
I used the below mentioned steps to confirm the expiry date for both of these certificates
STS – Please refer the KB:
https://kb.vmware.com/s/article/79248 (It will require to download a script – checksts.py)
Others – Run the below command in the VCSA.
for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list –store $i –text | egrep “Alias|Not After”; done
In my situation , both of the certificate types were expired and I had to replace all of them. To replace the STS certifcate , you could utilize a script provided by VMWare (fixsts.sh) using the KB : https://kb.vmware.com/s/article/76719
Once it is done , you need to restart the VCenter services using the below commands.
service-control –stop –all
service-control –start –all
service-control –status.
Thereafter , you could proceed to replace the other certificates using the VSphere Certificate Manager https://kb.vmware.com/s/article/2112283
How to update ESXi 6.x using an Offline Bundle
We all know there are multiple ways to upgrade the ESXi to the desired version. In this post I am going to discuss about upgrading the ESXi using an offline bundle. In my opinion this method is much convenient when you are having a single ESXi host , or you want a more controlled upgrade process. The steps are very minimal.(Please make sure that , you have powered off all the VM’s and the Host is put to the Maintenance mode)
– Download the ESXi bundle and upload it to a Datastore.
– Login to the ESXi via SSH.
Run the below command to verify the bundle
#esxcli software sources profile list -d /vmfs/volumes/datastore1/VMware-ESXi-6.7.0-Update3-15160138-HPE-Gen9plus-670.U3.10.5.5.25-Mar2020-depot.zip
Output:
Thereafter , run the below command to install the file.
#esxcli software profile update -p HPE-ESXi-6.7.0-Update3-Gen9plus-670.U3.10.5.5.25 -d /vmfs/volumes/datastore/VMware-ESXi-6.7.0-Update3-15160138-HPE-Gen9plus-670.U3.10.5.5.25-Mar2020-depot.zip
-p xxxxxxxxx (Name of the ESXi bundle as per the output).
Once the process is completed you need to reboot the server(The host may restart automatically for the 2nd time as well).
Update1: In case if you receive an error “Could not find a trusted signer” ., when you try to install you can run the same command with –no-sig-check as below:
#esxcli software profile update -p HPE-ESXi-6.7.0-Update3-Gen9plus-670.U3.10.5.5.25 -d /vmfs/volumes/datastore/VMware-ESXi-6.7.0-Update3-15160138-HPE-Gen9plus-670.U3.10.5.5.25-Mar2020-depot.zip –no-sig-check
Advanced Troubleshooting of ESXi Server 6.x for vSphere Gurus
Hi Folks
You could refer the attached document for hints that will help you in troubleshooting ESXi environments. This document covers mainly 3 areas.
- Which log files to review and when.
- ESXi commands to isolate and troubleshoot issues.
- Configuration Files.
Thanks.
Source: vmworld.
VMware PowerCLI
In this post , I am going to cover the PowerCLI module for VMware. Whenever, I came across a new cmd-let , I will update this post.
First things first, You need to install the PowerCLI. Now , the Windows Powershell have the VMware PowerCLI module. So you could simply install it by.
PS> Install-Module -Name VMware.PowerCLI
Then import it before using the Power CLI.
# To verify the version:
PS> Get-PowerCLIVersion
# To login to VCenter
PS> Connect-VIServer -Server “vcenterhostname”
# To Suppress the Certificate Warning/Error
Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false
#To list the VM’s with their creation date.
Get-VM | fl Name,CreateDate
How to create a RHEL 7 template in VSphere ESXi 6.7
Unlike for Windows , RHEL based template creation requires additional steps to make it work. During this process , I came across very valuable information from the linuxtechi blog . I am summarizing the steps and some additional steps that I followed during the whole. process. ( But , I am not adding the steps that you need to follow in ESXi to convert a VM in to template)
Source: https://www.linuxtechi.com/create-vm-template-ovirt-environment/
Environment Details:
- RHEL 7.3
- ESXi 6.7
+ Create a RHEL 7.3 VM
+ Install the Operating System and all other Packages needed.
+ Yum update it (If you have a valid RHEL subscription).
Thereafter , we need to follow the below steps to generalize the VM by removing any VM specific configuration and you need to do the below:
+ Remove the SSH host keys
# rm -f /etc/ssh/ssh_host_*
+Clear the history
# history -c
+Clear Yum
#yum clean all
Update#1: In case , if the sys-unconfig command does not work, you must use the virt-sysprep command . Details steps can be found in the below article
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-guest_virtual_machine_disk_access_with_offline_tools-using_virt_sysprep
NOTE: VM Customization specification is mandatory to avoid the VM’s getting the same hostids. Steps are as below;
+ Login to the VCenter.
+ Open Policies & Profiles.
+ Select VM Customization Specification.
+ Provide the details , based on your environment.
+ In the Network screen , select “Manually Select Custom Settings”.
+ Click on Add.
+ In the IPv4 section , select ” Prompt the user for an IPv4 address when the
specification is used “.
Good Luck .
Reset to device, \Device\RaidPort0, was issued” error in the Windows event log
Environment: VSphere ESXi 6.7 on HP DL 380 (Single Server)
Problem: The VM’s getting hanged / frozen. Cannot Login to Windows nor issue any Power off commands. During the investigation , we found out that the VM’s were recording Event ID 129 with the Warning message “Reset to device, \Device\RaidPort0, was issued” , just before the VM becoming unresponsive.
We were referring the VMware KB https://kb.vmware.com/s/article/2063346 , and confirmed the LSI_SAS driver is updated to the latest version. But , luckily in our case , this deployment was a temporary one as we are planning to move this VM’s to a stable VSphere Cluster running on Nutanix. After few days ,moving the VM’s to the Nutanix environment , we noticed that the VM’s were functioning well with out any issues.
So for those who are having a similar issue , you need to check the underlying storage structure . As it could cause similar issues like this.
NOTE: During this unresponsive state , you could notice the Disk Latency stays at more than 20. This definitely a problem for a VM’s responsiveness.
Hope you like it.. 