Posts filed under ‘VMware’

After updating vCSA to 6.7 U2 or higher, unable to log into the VAMI page- “Invalid Credentials” or “Unable to Login”

We recently had a situation , where we were not able to login to VAMI Page of the VCSA . As it was continuously providing the unable to login error. However , we know that the password is correct.

When we logged in to the SSH of the VCSA , we noticed that the applmgmt service is not started. We had to manually start it. Then the login to the VAMI got restored. However , when the appliance is rebooted the same issues pops up again. So ,we followed the VMware KB #68149(https://kb.vmware.com/s/article/68149) , but no success.

Finally , we decided to contact the VMware support and the resolution was very quick . It was due to sqlite DB used by the vmware-statsmonitor serivce(In our case , when we reboot the VCSA both the statsmonitor & applmgemt services were not coming up). The DB was reaching around 500 MB in size. The resolution was simple , he moved the DB file to a temporary location and restarted the vmware-statsmonitor service. We noticed a new appliance_stats.sqlite file got created in the same path /var/vmware/applmgmt/.

The DB file is located in /var/vmware/applmgmt/appliance_stats.sqlite.

We rebooted the VCSA and confirmed that we can login to the VAMI successfully.

Please note that this process will remove the previous stats collected on the VCSA.

Good luck.

September 8, 2021 at 9:57 am Leave a comment

How to renew vSphere 6.5 & 6.7 certificates.

When the VCenter Certificate is expired , you will be blocked from logging in to the VCenter . However , the Appliance Management will continue to work. Be noted that there a 2 categories of certificates.

  • VMware Security Token Service (STS)
  • Solution , Machine , Root and Other certificates.

Import Notes:

  1. You could avoid all these messy steps , had you monitor and check for the
    warnings on the VCenter Administration page for Certificate expiry events.
  2. For Windows based VCenter , you can refer the same KB’s mentioned here for the detailed steps.
  3. You may face an error when uploading the scripts to the VCSA via WinSCP . The Solution is provided in the same KB’s.
  4. Certificate Manager may fail during the process , you could refer the https://mueller-tech.com/2019/06/28/replacing-expired-certificates/ for the solution.

I used the below mentioned steps to confirm the expiry date for both of these certificates

STS – Please refer the KB:
https://kb.vmware.com/s/article/79248 (It will require to download a script – checksts.py)

Others – Run the below command in the VCSA.
for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list –store $i –text | egrep “Alias|Not After”; done

In my situation , both of the certificate types were expired and I had to replace all of them. To replace the STS certifcate , you could utilize a script provided by VMWare (fixsts.sh) using the KB : https://kb.vmware.com/s/article/76719

Once it is done , you need to restart the VCenter services using the below commands.

service-control –stop –all

service-control –start –all

service-control –status.

Thereafter , you could proceed to replace the other certificates using the VSphere Certificate Manager https://kb.vmware.com/s/article/2112283

July 13, 2021 at 3:38 pm Leave a comment

How to update ESXi 6.x using an Offline Bundle

We all know there are multiple ways to upgrade the ESXi to the desired version. In this post I am going to discuss about upgrading the ESXi using an offline bundle. In my opinion this method is much convenient when you are having a single ESXi host , or you want a more controlled upgrade process. The steps are very minimal.(Please make sure that , you have powered off all the VM’s and the Host is put to the Maintenance mode)

– Download the ESXi bundle and upload it to a Datastore.
– Login to the ESXi via SSH.

Run the below command to verify the bundle

#esxcli software sources profile list -d /vmfs/volumes/datastore1/VMware-ESXi-6.7.0-Update3-15160138-HPE-Gen9plus-670.U3.10.5.5.25-Mar2020-depot.zip

Output:

Thereafter , run the below command to install the file.

#esxcli software profile update -p HPE-ESXi-6.7.0-Update3-Gen9plus-670.U3.10.5.5.25 -d /vmfs/volumes/datastore/VMware-ESXi-6.7.0-Update3-15160138-HPE-Gen9plus-670.U3.10.5.5.25-Mar2020-depot.zip

-p xxxxxxxxx (Name of the ESXi bundle as per the output).

Once the process is completed you need to reboot the server(The host may restart automatically for the 2nd time as well).

Update1: In case if you receive an error “Could not find a trusted signer” ., when you try to install you can run the same command with –no-sig-check as below:

#esxcli software profile update -p HPE-ESXi-6.7.0-Update3-Gen9plus-670.U3.10.5.5.25 -d /vmfs/volumes/datastore/VMware-ESXi-6.7.0-Update3-15160138-HPE-Gen9plus-670.U3.10.5.5.25-Mar2020-depot.zip –no-sig-check

September 21, 2020 at 3:00 pm Leave a comment

Advanced Troubleshooting of ESXi Server 6.x for vSphere Gurus

Hi Folks

You could refer the attached document for hints that will help you in troubleshooting ESXi environments. This document covers mainly 3 areas.

  • Which log files to review and when.
  • ESXi commands to isolate and troubleshoot issues.
  • Configuration Files.

Thanks.

Source: vmworld.

June 8, 2020 at 9:51 am Leave a comment

VMware PowerCLI

In this post , I am going to cover the PowerCLI module for VMware. Whenever, I came across a new cmd-let , I will update this post.

First things first, You need to install the PowerCLI. Now , the Windows Powershell have the VMware PowerCLI module. So you could simply install it by.


PS> Install-Module -Name VMware.PowerCLI

Then import it before using the Power CLI.

# To verify the version:
PS> Get-PowerCLIVersion

# To login to VCenter
PS> Connect-VIServer -Server “vcenterhostname”

# To Suppress the Certificate Warning/Error
Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false

#To list the VM’s with their creation date.
Get-VM | fl Name,CreateDate


March 23, 2020 at 1:25 pm Leave a comment

How to create a RHEL 7 template in VSphere ESXi 6.7

Unlike for Windows , RHEL based template creation requires additional steps to make it work. During this process , I came across very valuable information from the linuxtechi blog . I am summarizing the steps and some additional steps that I followed during the whole. process. ( But , I am not adding the steps that you need to follow in ESXi to convert a VM in to template)

Source: https://www.linuxtechi.com/create-vm-template-ovirt-environment/

Environment Details:

  • RHEL 7.3
  • ESXi 6.7

+ Create a RHEL 7.3 VM

+ Install the Operating System and all other Packages needed.

+ Yum update it (If you have a valid RHEL subscription).

Thereafter , we need to follow the below steps to generalize the VM by removing any VM specific configuration and you need to do the below:

+ Remove the SSH host keys
# rm -f /etc/ssh/ssh_host_*

+Clear the history
# history -c

+Clear Yum
#yum clean all

Update#1: In case , if the sys-unconfig command does not work, you must use the virt-sysprep command . Details steps can be found in the below article
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-guest_virtual_machine_disk_access_with_offline_tools-using_virt_sysprep

NOTE: VM Customization specification is mandatory to avoid the VM’s getting the same hostids. Steps are as below;

+ Login to the VCenter.
+ Open Policies & Profiles.
+ Select VM Customization Specification.
+ Provide the details , based on your environment.
+ In the Network screen , select “Manually Select Custom Settings”.
+ Click on Add.
+ In the IPv4 section , select ” Prompt the user for an IPv4 address when the
specification is used “.

Good Luck .

 

 

October 7, 2019 at 12:07 pm 1 comment

Reset to device, \Device\RaidPort0, was issued” error in the Windows event log

Environment: VSphere ESXi 6.7 on HP DL 380 (Single Server)

Problem: The VM’s getting hanged / frozen. Cannot Login to Windows nor issue any Power off commands. During the investigation , we found out that the VM’s were recording Event ID 129 with the Warning message “Reset to device, \Device\RaidPort0, was issued” , just before the VM becoming unresponsive.

We were referring the VMware KB https://kb.vmware.com/s/article/2063346 , and confirmed the LSI_SAS driver is updated to the latest version. But , luckily in our case , this deployment was a temporary one as we are planning to move this VM’s to a stable VSphere Cluster running on Nutanix. After few days ,moving the VM’s to the Nutanix environment , we noticed that the VM’s were functioning well with out any issues.

So for those who are having a similar issue , you need to check the underlying storage structure . As it could cause similar issues like this.

NOTE: During this unresponsive state , you could notice the Disk Latency stays at more than 20. This definitely a problem for a VM’s responsiveness.


March 24, 2019 at 12:08 pm Leave a comment

How to enable EVC when VCenter Server is running on VM in a Nutanix Cluster

As part of the Nutanix best practices we need to enable the EVC on the VSphere Cluster.  In that sense , when the VCenter Server it self a VM , you will be dragged in to  a chicken and egg situation.  Because , when a host contains powered on VM , you will not be able to add the host to the EVC enabled Cluster. Thus , to overcome this condition , you could follow the below guidelines. (You may need to disable the Admission Control temporarily and enable it again until you finish all the steps)

1) Add the hosts to the DataCenter .

2) Create the HA / DRS Cluster .

3) Enable EVC on the cluster based on your processor architecture.

4) Pick up any host and shutdown the running VM’s and the CVM ( Please keep in mind , you can shutdown only one CVM at a time).

5) Then drag & drop the host to the Cluster , the  host will be added to the cluster without any hassle.

6) Power on the VM’s and the CVM ( wait till the CVM completes the boot)

7) Now , VMotion the VCenter VM to the host which is part of the Cluster already.

8) That’s it repeat  steps 4 ,5 & 6 for the remaining hosts.

Hint:

# In case if you have forgotten to enable EVC before you put the Cluster in to production , and now you are in a situation , that you need to expand your Nutanix Cluster and enabling EVC becomes mandatory to add the new nodes to the existing ESXi cluster.In this case , you could do the additional steps given below to achieve the intended result. ( Again , you may need to disable the Admission Control temporarily and enable it again until you finish all the steps)

 

1) Create a new Cluster (without EVC)

2) Select a host and VMotion  all the Production VM’s running on that host to other remaining hosts.

3) Shutdown the CVM

4) Put the host on to the Maintenance Mode

5) Drag and Drop the host to the new Cluster

6) Exit from the Maintenance Mode & Power on the CVM.

7) Then VMotion the VCenter VM & Other VM’s to this host.

8) Do the steps 2 – 6 for other remaining hosts.

9) Reconfigure your old cluster with proper EVC mode.

10) Then repeat 2 – 6 for all the hosts.

Source :

Refer https://www.virten.net/2013/04/intel-cpu-evc-matrix/ for the guidelines on EVC modes

Video Reference : https://www.youtube.com/watch?v=DSfzafr1ndA

 

 

 

March 18, 2019 at 2:24 pm Leave a comment

AsBuilt Report for VSphere

Hi Folks

Until recent years , I was struggling to build a proper AsBuilt Document for VSphere environments. As the manual process requires capturing screenshots and time consuming word document preparations.

Last week , I came across 2 blogs talking about this AsBuilt tool for VMware which turned out to be  very handy and must have tool for VMware installations .

For those who want to read more about this tool, could visit the 2 blogs that are listed at the bottom of this page.

You need Windows PowerShell. Once you are ready with the PowerShell run the below commands to build your AsBuilt document .

 

1) Install the PSCribo Module

 #Install-Module PSCribo

2)Download the AsBuilt PowerShell Scripts via https://github.com/tpcarman/As-Built-Report

2.1)Extract it to a Folder

#Import-Module C:\As-Built-Report-dev\AsBuiltReport.psd1

3)Install PowerCLI Module

#Find-Module -Name VMware.PowerCLI

#Install-Module -Name VMware.PowerCLI

3.1)Run the below command to bypass SSL warning for VCenter/ESXi

#Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false

4) Below command will create the Report

New-AsBuiltReport -Target vcenterip -Credential (get-credential) -Type vSphere -Format HTML,Word -TimeStamp imeStamp -Healthchecks -AsBuiltConfigPath C:\As-Built-Report-dev\Src\Public\Reports\vSphere\vsphere.json

Source:

https://www.timcarman.net/as-built-report/

As Built Report – working with it in my lab

 

January 24, 2019 at 3:29 pm Leave a comment

How to Capture & Analyze Network Traffic on ESXi

Being an ESXI  Implementer or an Administrator , you may come across some situations where you need to make your hands dirty 🙂 , with deep network troubleshooting.  I had a similar situation few months ago , which I would like to share it in this post.

We deployed the Horizon View (for VDI) in one of our customer’s ESXi Cluster ( 8 Nodes) environment, The Desktop users were complaining about they were not able to specific network .

 Thus to further investigate we swapped the Physical Adapter to the on-board BroadCom cards (1Gps). Then we were able to re-establish the network. We thought to engage the VMware Support with the intention to find out the root cause and get a permanent fix. The VMware support was pretty awesome and they were able to nail it very quickly.

First they used the two built-in commands on ESXI , which are

  • pktcap-uw (To capture the Network Packets)
  • tcpdump-uw ( To read the captured Packets)

They ran the below commands on both the NIC cards to initially capture the traffic.

  • pktcap-uw –uplink vmnic0 –dir 0 –mac 00:00:00:00:00:00 —vlan 18 -o /tmp/f.pcap

uplink –  Name of the VMnic

dir      –  0  means RX Traffic

mac   –  MAC address of the machine which you are troubleshooting

vlan   –  The VLAN ID

Thereafter we read the  output of the above command using 

  •     tcpdump-uw -ner /tmp/f.pcap

By comparing the output from both NIC’s  we were able to narrow down the problem to the Mellanox cards. when tagged traffic passed by on a Mellanox Network Card (10 Gbps), the reply packet was not being tagged with the proper VLAN ID causing disruption to the network traffic.

 

Good Luck

Muralee

 

 

December 12, 2018 at 10:52 am Leave a comment

Older Posts


Archives

Categories

Follow Hope you like it.. on WordPress.com

Blog Stats

  • 47,082 hits

%d bloggers like this: