Cross VCenter VMotion fails with the error “A specified parameter was not correct: path”

Recently , we had to migrate some VM’s from a VSphere 6.7(17499825) to 7.0 U2 cluster. We leveraged the Import VM’s feature that utilizes the Cross VCenter VMotion capability.


The source: VCenter 7.0
The destination: VCenter 6.7

Even though , the required access between the clusters were allowed(based on https://kb.vmware.com/s/article/2106952) , when we tried to trigger the import VM’s there were no process or tasks being generated in both vcenters. However , we noticed a failed task on the VM that we are trying to import in the destination VCenter( under the tasks and events for the particular VM )wiith the message “A specified parameter was not correct: path”.

We started reviewing the hostd.log on both sides but no avail. Finally , out of exhaustion , we tried to changed the source datastore to a different one(In our scenario the datastore names were identical on both sides) and voila the import VM’s got kicked in and completed successfully. Still , I am not sure the cause for this behavior where the import of VM’s failing due to identical datastore names on both sides , so we have opened a case with VMware support to find out the root cause. I will updating this post as soon as I receive the response.

Tip: Also during the import VM’s wizard (3rd step) where we select the desired to host to place the imported VM on the source cluster we were receiving a SSL related error. This was resolved after disabling the SSL inspection on our firewall

March 28, 2022 at 12:11 pm 6 comments

How to migrate Windows Cluster VMs (incl. RDMs) from old to new storage with minimal downtime?

Recently , we did a storage upgrade for one of our customer. The VM migration was pretty easy for almost 90% of the workloads. However , the remaining 10 % of the VM’s were having RDM disks ,and for the standalone RDM disks (non clustered workloads) we were able to leverage the Storage VMotion to convert the RDM’s to VMDK and migrate it.

On the other hand , for clustered VM’s , Initially we were planning to utilize the RP4VM to migrate the VM’s(Failover to the Replica). But this approach was not fruitful as we faced a compatibility issue and there was no workaround and we dropped the idea.

During this time our colleague from the Storage Team suggested this alternate plan and we tested it and it was successful. The steps are here as listed.

make a note of the RDM mapping to the VMs (note the SCSI ID assigned to the RDM in the VM configuration) 

# shutdown the VMs of that Virtual cluster.

# unmap the RDMs.

# Create the Luns on the destination array (must be at least as large as the source !!!)

# present to ESXhost (rescan, …)

# use Storage vMotion to move the VM to the new datastores.(This is to move the VM with the OS disk and any other non RDM based disks)

# use ESX CLI vmkfstools to copy the RDM content to the new lun and this will automatically create the new RDM vmdk pointer file (destination.vmdk)

vmkfstolls -i <srcdisk.vmdk> -d rdmp:device <destination.vmdk>

Example:
vmkfstools -i TestVM_RDM1.vmdk -d rdmp:/vmfs/devices/disks/vml.02000100006006016044440000f8b164674b51e111565241494420 TestVM_NewRDM.vmdk

   ==== remap the new LUN as RDMp with the same SCSI ID====

      === don’t forget to set the bus sharing if it disappeared===

Source:

https://www.dell.com/community/VMware/How-to-migrate-Windows-Cluster-VMs-incl-RDMs-from-old-to-new/td-p/6859292

Kudos to EricDeWitte1 (Contributor)

February 14, 2022 at 12:22 pm Leave a comment

How To Find My Public IP Address From Command Line On a Linux

For machines with GUI , Interface there are several ways to obtain the Public IP address. However , in Linux shell environment you need to rely on the commands(Even though there are 3rd party tools available).

Open a terminal window and type the below command:

#dig +short myip.opendns.com @resolver1.opendns.com

OR

dig TXT +short o-o.myaddr.l.google.com @ns1.google.com

Source:https://www.cyberciti.biz/

January 20, 2022 at 11:05 am Leave a comment

What is the most secure way to allow a user read access to a log file

I am running Splunk Enterprise on a Linux Server in our environment. Our Security Standards , prevents the Splunk from running under root User. In this scenario , we will come across a situation , where we will not be able to read the critical Linux logs like audit , messages , secure etc.

There are several methods to achieve this , But in this post I am using the ACL method.

First , you need to set the acl’s on these files located in the /var/log folder. My Splunk is running under the user splunkadmin

# setfacl -m g:splunkadmin:r /var/log/messages
# setfacl -m g:splunkadmin:r /var/log/secure
# setfacl -m g:splunkadmin:r /var/log/maillog

The above changes will not retain , when the logs are rotated. Thus you need to create postrotate action for these logs to retain the ACL's.

1) Create a text file in the folder /etc/logrotate.d/Splunk_ACLs ( My file name is SplunkACLs).
2)  Add the below entries in the file 
{
    postrotate
        /usr/bin/setfacl -m g:splunk:r/var/log/maillog
        /usr/bin/setfacl -m g:splunk:r /var/log/messages
        /usr/bin/setfacl -m g:splunk:r /var/log/secure
     endscript
}

You can verify the ACLS using
#getfacl /var/log/messages

On the other hand , The steps for audit files are different .

Check the current permissions
#ls -l /var/log/audit/audit.log
-rw——- 1 root root 3531590 Jun 1 00:20 /var/log/audit/audit.log

Then edit the auditd.conf file and change the log_group parameter to splunkadmin instead of root.
log_group = splunkadmin

Restart the auditd services
#service auditd restart

You can re-verify the permissions
#ls -l /var/log/audit/audit.log
-rw-r—– 1 root splunkadmin 3532862 Jun 1 00:24 /var/log/audit/audit.log

Source:

https://www.thegeekdiary.com/how-to-change-the-default-permissions-on-var-log-audit-audit-log-file-in-centos-rhel/

https://newbedev.com/what-is-the-most-secure-way-to-allow-a-user-read-access-to-a-log-file

January 5, 2022 at 1:23 pm Leave a comment

Datastore conflicts with an existing datastore in the datacenter that has the same URL” error in vCenter Server

Hi Guys.

Let me share my recent experience with a VMware Upgrade on a Nutanix Cluster. We planned the upgrade from vSphere 6.7 U3 to vSphere 7.0 U1.

As usual we started the Nutanix LCM upgrade and brought the cluster to the latest Firmware & BIOS drivers.

Thereafter , I started upgrading the ESXi , I used the offline .zip file for the upgrade. I followed the below steps.

  • Uploaded the offline bundle VMware-ESXI-7.0U1-16850804-depot.zip to the datastore.
  • Ran the below command to findout the profile name
    esxcli software sources profile list -d /xxxxxxx/xxxxxx/VMware-ESXI-7.0U1-16850804-depot.zip
  • Get the profile name from the output of the above command which is ESXi-7.0.1-1685 and used it in command as below
    esxcli software profile update -d /xxxxxxx/xxxxxx/VMware-ESXI-7.0U1-16850804-depot.zip -p ESXi-7.0.1-1685

Everything went well and rebooted the server but,

  • CVM is shown inaccessible.
  • Local datastore was missing from the server.

As adviced by the Nutanix support , restarted the server for the second time. and I was able to see the local datastore. Still we could not exit from the maintenance mode. As it was giving the error “Datastore conflicts with an existing datastore in the datacenter that has the same URL” error in vCenter Server”.

The issue was resolved after following the VMware KB https://kb.vmware.com/s/article/79623.

To worsen up our situation , for some reason the ESXi version got reverted automatically to the previous build ESXi 6.7 after following the steps in the KB. So we decided to check the boot.cfg files in bootbank & altbootbank locations.

#tail -2 /*bootbank/boot.cfg
It was showing only the ESXi 7.0 U1 only.

Again we contacted the VMware support and resolved the issue with the below steps.

Checked whether achi is disabled or not by typing the command on the ESXi shell.
#grep ahci /etc/vmware/esx.conf
/vmkernel/module/vmw_ahci/enabled = “false”

The output shows the achi is disabled and Enabled it by typing
#esxcfg-module -e vmw_ahci

Thereafter ran the upgrade again using the same steps and rebooted the server.Voila we were able to see the correct version. For some reason the default ahci driver provided from VMware has been disabled.

Sources:
https://anthonyspiteri.net/datastore-esxi7-upgrade-supermicro/

https://kb.vmware.com/s/article/79623

December 9, 2021 at 2:53 pm Leave a comment

The DNS server was unable to open Active Directory”

Recently , One of our customer reported an issue stating that the Exchange Services are failing and Outlook clients are getting disconnected. We noticed few DC related events (Kerberos) on both the exchange servers. Thus , we ran the “netdom query fsmo” command on the 2 Exchange Servers and got the below error:

The same error appeared on all the other domain joined servers. Therefore ,we decided to check the DC’s.

When we reviewed the event viewer on the 2 DC’s , there were DNS related errors(Event ID 4000)

Further , we could not open the DNS MMC snap-ins and pinging the hostname by DC was failing as well. However, the DNS service is started state. In addition to this , there were errors on KDC consistency as well. After , troubleshooting for few minutes ,we go hold of the Microsoft KB :https://docs.microsoft.com/en-us/troubleshoot/windows-server/networking/dns-zones-do-not-load-event-4000-4007 and followed the steps mentioned to resolve the issue successfully.

Few points to consider:
– You will find an additional “d: in the word password in the below command. Do not change it.
netdom resetpwd /server: /userd: netdom resetpwd /server: /userd: /passwordd:*
– In my case I had to run this command on the PDC and the other DC as well
– Stop the KDC service prior to running the command.
– First I started on the PDC and restarted it and ensured the DNS snap-in was accessible and the pinging
by hostname was working.
– Finally , I continued the same steps on the remaining domain controllers.

October 5, 2021 at 10:04 am Leave a comment

After updating vCSA to 6.7 U2 or higher, unable to log into the VAMI page- “Invalid Credentials” or “Unable to Login”

We recently had a situation , where we were not able to login to VAMI Page of the VCSA . As it was continuously providing the unable to login error. However , we know that the password is correct.

When we logged in to the SSH of the VCSA , we noticed that the applmgmt service is not started. We had to manually start it. Then the login to the VAMI got restored. However , when the appliance is rebooted the same issues pops up again. So ,we followed the VMware KB #68149(https://kb.vmware.com/s/article/68149) , but no success.

Finally , we decided to contact the VMware support and the resolution was very quick . It was due to sqlite DB used by the vmware-statsmonitor serivce(In our case , when we reboot the VCSA both the statsmonitor & applmgemt services were not coming up). The DB was reaching around 500 MB in size. The resolution was simple , he moved the DB file to a temporary location and restarted the vmware-statsmonitor service. We noticed a new appliance_stats.sqlite file got created in the same path /var/vmware/applmgmt/.

The DB file is located in /var/vmware/applmgmt/appliance_stats.sqlite.

We rebooted the VCSA and confirmed that we can login to the VAMI successfully.

Please note that this process will remove the previous stats collected on the VCSA.

Good luck.

September 8, 2021 at 9:57 am Leave a comment

How to check the AD Schema version

We all know , how to check the FFL & DFL version using the AD Snap-ins.However , if you want to check the AD schema version , you need to run regsvr32 schmmgmt.dll to active the MMC snap-in to get the required details.

Most of the time this process will fail and need additional troubleshooting steps. Instead of that , you can use the below PowerShell command to easily get the AD schema version. In the meantime , you could still use the ADSI Edit as well.

Get-ADObject (Get-ADRootDSE).schemaNamingContext -Property objectVersion

Output of the above command will give you a value for the ObjectVersion. This value need to be compared with below to obtain the correct schema version.

Source: https://support.globalsign.com/aeg/aeg-how-check-active-directory-schema-version

August 31, 2021 at 12:08 pm Leave a comment

How to use KMS based activation for Windows images

Hi All

In our Hosting environment , we had to deploy large number of Windows Servers. We need to ensure that the Windows images are remain active within our datacenters only. As we cannot use MAK based activation. Because , if the customer moves out from us the licenses should be not active . In other words , it is customer responsibility to reactivate the Windows in his new environment.

So we came up with the idea of KMS based activation using VMware Templates and this is how we did it.

  • Install the KMS Service on a Windows 2019 and activate it with the KMS Key.
    ( I am not writing the steps here .Since a simple Google search result will yield lot’s how-to articles on KMS installation.)
  • Created a Windows 2019 VM.
  • Activated the Windows 2019 with the GVLK. You can refer the below URL for the GVLK keys.
    https://docs.microsoft.com/en-us/windows-server/get-started/kmsclientkeys
  • Ran the Sysprep and converted it to a template.
  • Thereafter , we created a custom specification file to perform the KMS activation on Windows image. All these settings are mandatory .Otherwise , the KMS commands will fail to run.

    + In the Windows License Page , clear all the settings.

+ In the Administrator Password and set a Password ,

+ In the Commands to run once Page, enter the below commands
cscript c:\windows\system32\slmgr.vbs /skms “kms server ip”:1688
cscript c:\windows\system32\slmgr.vbs /ato

That’s it. Now you can go ahead and deploy the Windows images and get it activated by KMS.

Further reading:

1)Use the below command on your Windows images to verify the CMID(Client Machine ID is not duplicated).

Get-WmiObject -class SoftwareLicensingService | Select-object ClientMachineID

Duplicated CMID will prevent VM’s from getting activated.

2) On the KMS client you can review the below Registry key to verify the correct KMS settings have been applied.

HKLM/SOFTWARE/MICROSOFT/WINDOWSNT/CURRENTVERSION/SOFTWAREPROTECTIONPLATFORM/
–KeyManagementServiceName
–KeyManagementServicePort

3) Use the below commands on the KMS Server & Clients to verify the Windows Activation.

slmgr /dlv

slmgr /dli


August 4, 2021 at 9:18 am Leave a comment

How to renew vSphere 6.5 & 6.7 certificates.

When the VCenter Certificate is expired , you will be blocked from logging in to the VCenter . However , the Appliance Management will continue to work. Be noted that there a 2 categories of certificates.

  • VMware Security Token Service (STS)
  • Solution , Machine , Root and Other certificates.

Import Notes:

  1. You could avoid all these messy steps , had you monitor and check for the
    warnings on the VCenter Administration page for Certificate expiry events.
  2. For Windows based VCenter , you can refer the same KB’s mentioned here for the detailed steps.
  3. You may face an error when uploading the scripts to the VCSA via WinSCP . The Solution is provided in the same KB’s.
  4. Certificate Manager may fail during the process , you could refer the https://mueller-tech.com/2019/06/28/replacing-expired-certificates/ for the solution.

I used the below mentioned steps to confirm the expiry date for both of these certificates

STS – Please refer the KB:
https://kb.vmware.com/s/article/79248 (It will require to download a script – checksts.py)

Others – Run the below command in the VCSA.
for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list –store $i –text | egrep “Alias|Not After”; done

In my situation , both of the certificate types were expired and I had to replace all of them. To replace the STS certifcate , you could utilize a script provided by VMWare (fixsts.sh) using the KB : https://kb.vmware.com/s/article/76719

Once it is done , you need to restart the VCenter services using the below commands.

service-control –stop –all

service-control –start –all

service-control –status.

Thereafter , you could proceed to replace the other certificates using the VSphere Certificate Manager https://kb.vmware.com/s/article/2112283

July 13, 2021 at 3:38 pm Leave a comment

Older Posts


Archives

Categories

Follow Hope you like it.. on WordPress.com

Blog Stats

  • 53,979 hits

%d bloggers like this: