Reset to device, \Device\RaidPort0, was issued” error in the Windows event log
Environment: VSphere ESXi 6.7 on HP DL 380 (Single Server)
Problem: The VM’s getting hanged / frozen. Cannot Login to Windows nor issue any Power off commands. During the investigation , we found out that the VM’s were recording Event ID 129 with the Warning message “Reset to device, \Device\RaidPort0, was issued” , just before the VM becoming unresponsive.
We were referring the VMware KB https://kb.vmware.com/s/article/2063346 , and confirmed the LSI_SAS driver is updated to the latest version. But , luckily in our case , this deployment was a temporary one as we are planning to move this VM’s to a stable VSphere Cluster running on Nutanix. After few days ,moving the VM’s to the Nutanix environment , we noticed that the VM’s were functioning well with out any issues.
So for those who are having a similar issue , you need to check the underlying storage structure . As it could cause similar issues like this.
NOTE: During this unresponsive state , you could notice the Disk Latency stays at more than 20. This definitely a problem for a VM’s responsiveness.
How to enable EVC when VCenter Server is running on VM in a Nutanix Cluster
As part of the Nutanix best practices we need to enable the EVC on the VSphere Cluster. In that sense , when the VCenter Server it self a VM , you will be dragged in to a chicken and egg situation. Because , when a host contains powered on VM , you will not be able to add the host to the EVC enabled Cluster. Thus , to overcome this condition , you could follow the below guidelines. (You may need to disable the Admission Control temporarily and enable it again until you finish all the steps)
1) Add the hosts to the DataCenter .
2) Create the HA / DRS Cluster .
3) Enable EVC on the cluster based on your processor architecture.
4) Pick up any host and shutdown the running VM’s and the CVM ( Please keep in mind , you can shutdown only one CVM at a time).
5) Then drag & drop the host to the Cluster , the host will be added to the cluster without any hassle.
6) Power on the VM’s and the CVM ( wait till the CVM completes the boot)
7) Now , VMotion the VCenter VM to the host which is part of the Cluster already.
8) That’s it repeat steps 4 ,5 & 6 for the remaining hosts.
Hint:
# In case if you have forgotten to enable EVC before you put the Cluster in to production , and now you are in a situation , that you need to expand your Nutanix Cluster and enabling EVC becomes mandatory to add the new nodes to the existing ESXi cluster.In this case , you could do the additional steps given below to achieve the intended result. ( Again , you may need to disable the Admission Control temporarily and enable it again until you finish all the steps)
1) Create a new Cluster (without EVC)
2) Select a host and VMotion all the Production VM’s running on that host to other remaining hosts.
3) Shutdown the CVM
4) Put the host on to the Maintenance Mode
5) Drag and Drop the host to the new Cluster
6) Exit from the Maintenance Mode & Power on the CVM.
7) Then VMotion the VCenter VM & Other VM’s to this host.
8) Do the steps 2 – 6 for other remaining hosts.
9) Reconfigure your old cluster with proper EVC mode.
10) Then repeat 2 – 6 for all the hosts.
Source :
Refer https://www.virten.net/2013/04/intel-cpu-evc-matrix/ for the guidelines on EVC modes
Video Reference : https://www.youtube.com/watch?v=DSfzafr1ndA
ESXTOP Thresholds
Hi Guys
Below table provides the recommended thresholds from ESXi . These values can be monitored via ESXTOP commands.
Metrics and Thresholds
| Display | Metric | Threshold | Explanation |
| CPU | %RDY | 10 | Overprovisioning of vCPUs, excessive usage of vSMP or a limit(check %MLMTD) has been set. Note that you will need to expand the VM Group to see how this is distributed across vCPUs. If you have many vCPUs than per vCPU may be low and this may not be an issue. 10% is per world! |
| CPU | %CSTP | 3 | Excessive usage of vSMP. Decrease amount of vCPUs for this particular VM. This should lead to increased scheduling opportunities. |
| CPU | %MLMTD | 0 | The percentage of time the vCPU was ready to run but deliberately wasn’t scheduled because that would violate the “CPU limit” settings. If larger than 0 the world is being throttled due to the limit on CPU. |
| CPU | %SWPWT | 5 | VM waiting on swapped pages to be read from disk. Possible cause: Memory overcommitment. |
| MEM | MCTLSZ | 1 | If larger than 0 hosts is forcing VMs to inflate balloon driver to reclaim memory as host is overcommited. |
| MEM | SWCUR | 1 | If larger than 0 hosts has swapped memory pages in the past. Possible cause: Overcommitment. |
| MEM | SWR/s | 1 | If larger than 0 host is actively reading from swap(vswp). Possible cause: Excessive memory overcommitment. |
| MEM | SWW/s | 1 | If larger than 0 host is actively writing to swap(vswp). Possible cause: Excessive memory overcommitment. |
| MEM | CACHEUSD | 0 | If larger than 0 hosts has compressed memory. Possible cause: Memory overcommitment. |
| MEM | ZIP/s | 0 | If larger than 0 hosts is actively compressing memory. Possible cause: Memory overcommitment. |
| MEM | UNZIP/s | 0 | If larger than 0 host has accessing compressed memory. Possible cause: Previously host was overcommited on memory. |
| MEM | N%L | 80 | If less than 80 VM experiences poor NUMA locality. If a VM has a memory size greater than the amount of memory local to each processor, the ESX scheduler does not attempt to use NUMA optimizations for that VM and “remotely” uses memory via “interconnect”. Check “GST_ND(X)” to find out which NUMA nodes are used. |
| NETWORK | %DRPTX | 1 | Dropped packets transmitted, hardware overworked. Possible cause: very high network utilization |
| NETWORK | %DRPRX | 1 | Dropped packets received, hardware overworked. Possible cause: very high network utilization |
| DISK | GAVG | 25 | Look at “DAVG” and “KAVG” as the sum of both is GAVG. |
| DISK | DAVG | 25 | Disk latency most likely to be caused by the array. |
| DISK | KAVG | 2 | Disk latency caused by the VMkernel, high KAVG usually means queuing. This is the ESXi storage stack, the vSCSI layer and the VMM. Check “QUED”. |
| DISK | QUED | 1 | Queue maxed out. Possibly queue depth set to low, or controller overloaded. Check with array vendor for optimal queue depth value. (Enable this via option “F” aka QSTATS |
| DISK | ABRTS/s | 1 | Aborts issued by guest(VM) because storage is not responding. For Windows VMs this happens after 60 seconds by default. Can be caused for instance when paths failed or array is not accepting any IO for whatever reason. |
| DISK | RESETS/s | 1 | The number of commands resets per second. |
| DISK | ATSF | 1 | The number of failed ATS commands, this value should be 0 |
| DISK | ATS | 1 | The number of successful ATS commands, this value should go up over time when the array supports ATS |
| DISK | DELETE | 1 | The number of successful UNMAP commands, this value should go up over time when the array supports UNMAP! |
| DISK | DELETE_F | 1 | The number of failed UNMAP commands, this value should be 0 |
| DISK | CONS/s | 20 | SCSI Reservation Conflicts per second. If many SCSI Reservation Conflicts occur performance could be degraded due to the lock on the VMFS. |
| VSAN | SDLAT | 5 | Standard deviation of latency, when above 10ms latency contact support to analyze vSAN Observer details to find out what is causing the delay |
Source: http://www.yellow-bricks.com/esxtop/#esxtop-thresholds
Latency between the Nutanix CVM’s
Recently we noticed the Prism was throwing an error stating that there is latency between CVM’s . To investigate the issue we raised a support call with the Nutanix Team. I am sharing the procedures followed by the Nutanix Team as it may help somebody who are facing a similar issue.
# Login to Controller VM
# cd ~nutanix/data/logs/sysstats (This location will contain the ping_hosts & ping_gateway logs)
# tailf ping_hosts.INFO
In our case we noticed there was unreachable on one of the CVM’s
x.x.x.1 : 0.187 ms
x.x.x.2 : Unreachable
x.x.x.3 : 0.028 ms
So we consulted the Network Team and found out that the Switch port where one of the node is connected , conatined lots of errors and we had to replace the cable.
That’s it the problem got resolved.
How to Configure Sendmail & SpamAssassin for SPF Check
We had a Sendmail Server (8.14.7) running on CentOS Server, The server acts as a Secondary MX and SMART hosts for many domains. In this scenario we decided to install the SpamAssassin to force the Sendmail server to validate SPF records prior to accepting the email. I have written the below post to explain the whole process with few notes on troubleshooting I had to perform during the installation & configuration stages.
-Sendmail (already installed and running)
-SpamAssassin v. 3.4.0 (already installed with CentOS , use spamassassin -V to check the version)
– Spam-ass milter
So let’s start with the process;
+ Install spam-ass milter
# yum install perl-Mail-SPF perl-Mail-DKIM perl-Razor-Agent pyzor poppler-utils re2c ( These are the prerequisites)
# Download the RPM from https://centos.pkgs.org/7/epel-x86_64/spamass-milter-0.4.0-7.el7.x86_64.rpm.html and install by rpm -i “rpm name”
+ Start the spamassassin & spamass-milter services
# systemctl start spamassassin
# systemctl start spamass-milter.service
Now we need to force sendmail daemon to use the milter for antispam processing. Add the below lines in sendmail.mc (** do not forget to backup the files before modifying it)
======================================================================================
dnl #
dnl # SPAMASSASSIN dnl
dnl **
dnl ** enable spamassassin-milter to scan for spam using spamassassin **
dnl **
INPUT_MAIL_FILTER(`spamassassin’, `S=unix:/var/run/spamass-milter/spamass-milter.sock, F=, T=C:15m;S:4m;R:4m;E:10m’)dnl
define(`confMILTER_MACROS_CONNECT’,`t, b, j, _, {daemon_name}, {if_name}, {if_addr}’)dnl
define(`confMILTER_MACROS_HELO’,`s, {tls_version}, {cipher}, {cipher_bits}, {cert_subject}, {cert_issuer}’)dnl
dnl # END LOCAL ADDITIONS
dnl #
======================================================================================
+ save the file & quit it
+ Compile the Sendmail configuration & restart the sendmail services.
# m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf ( or you could simply type make)
# systemctl restart sendmail
To confirm whether all these components are working fine with the relevant SPF check you need to run ,
# spamassassin -D < /usr/share/doc/spamassassin-3.4.0/sample-spam.txt 2>&1 |grep -i spf
Thereafter we could analyze whether the email are being filtered properly with the SPF Check, to check that run
# grep spf /var/log/maillog
if it is not functioning well you should look for the errors & start troubleshooting it. In my case it was throwing the below error;
“Mar 4 15:34:20 mail spamd[11685]: spf: lookup failed: addr is not a string at /usr/share/perl5/vendor_perl/IO/Socket/IP.pm line 662.”
After few minutes of googling , we found out that , it was a bug in the perl-socket module in CentOS 7 , thus you need to
# yum install epel
# yum update perl-Socket –enablerepo=cr
You need to restart the sendmail , spamassassin & spamass-milter services for the changes to take effect and review the log again for any errors.
+ A new cron.d job will be created automatically for the spamassassin update in the /etc/cron.d/sa-update file.
Few advice, do not modify any files in /usr/share/spamassassin , since these files will be overwritten with spamassassin updates. Thus always modify the /etc/mail/local.cf for any customizations and it is a system wide configuration.
Secondly you could refer the below samples , that you could use for any customization and whitelisting stuff with in spamassassin.
========================================================================
# How many hits before a message is considered spam.
required_hits 5.0
# Text to prepend to subject if rewrite_subject is used
rewrite_header Subject [*****SPAM*****]
# Encapsulate spam in an attachment
report_safe 1
# Enable the Bayes system
use_bayes 1
# Enable Bayes auto-learning
bayes_auto_learn 1
bayes_path /home/spamd/
bayes_file_mode 0666
# Enable or disable network checks
skip_rbl_checks 0
use_razor2 0
use_dcc 0
use_pyzor 0
# Mail using languages used in these country codes will not be marked
# as being possibly spam in a foreign language.
# ok_languages all
# Mail using locales used in these country codes will not be marked
# as being possibly spam in a foreign language.
# ok_locales all
# Whitelist important senders
whitelist_from *@xyz.xx
========================================================================
That’s it , but during this process i came across useful blog sites and forums posts that helped me to work on this task and they are listed below for your reference as well.
https://blesseddlo.wordpress.com/2010/04/01/sendmail-spamassassin-spamass-milter-milter-greylist/
https://www.rosehosting.com/blog/how-to-install-spamassassin-on-a-virtual-server-with-centos-6/
https://www.jethrocarr.com/2013/10/26/spf-with-spamassassin/
http://forums.sentora.org/showthread.php?tid=1118
https://it.megocollector.com/linux/install-spamassassin-on-centos-6/
http://forum.icewarp.com/forum/showthread.php?1809-Spamassassin-SPF-and-spoofing
https://centos.org/forums/viewtopic.php?t=60477
https://vamsoft.com/support/tools/spf-policy-tester (This will validate you SPF check in the email server)
http://spamassassin.1065346.n5.nabble.com/return-path-test-td1869.html
https://www.howtoforge.com/community/threads/spamassassin-version.74/
Update1:
In January 2018 , barracuda removed the RBL from the SA ruleset (it was under 72_active.cf in /usr/share/spamassassin)
To add this rule , you need to register via the below URL; http://barracudacentral.org/account/register
and then you need to manually edit the local.cf file add the below texts and restart the services
ifplugin Mail::SpamAssassin::Plugin::DNSEval
header __RCVD_IN_BRBL eval:check_rbl(‘brbl’,’bb.barracudacentral.org’)
tflags __RCVD_IN_BRBL net
header __RCVD_IN_BRBL_2 eval:check_rbl_sub(‘brbl’, ‘127.0.0.2’)
meta RCVD_IN_BRBL __RCVD_IN_BRBL_2 && !RCVD_IN_BRBL_LASTEXT
describe RCVD_IN_BRBL Received is listed in Barracuda RBL bb.barracudacentral.org
score RCVD_IN_BRBL 1.2
tflags RCVD_IN_BRBL net
header RCVD_IN_BRBL_LASTEXT
eval:check_rbl(‘brbl-lastexternal’, ‘bb.barracudacentral.org’)
describe RCVD_IN_BRBL_LASTEXT Last external is listed in Barracuda RBL bb.barracudacentral.org
score RCVD_IN_BRBL_LASTEXT 2.2
tflags RCVD_IN_BRBL_LASTEXT net
endif
Update 2:
Recently we were blacklisted by backscatter and the reason for listing was , sending out NDR for non valid emails. Thus we have add the below line in the local.cf configuration file
whitelist_bounce_relays myrelay.mydomain.net (Replace it with your outgoing email server name) If you have multiple servers , you could add them all here in multiple lines . Once the above is added and the spamassassin is restarted , issue the below command to verify for any config errors #spamassassin --lint The below URL contains additional information to test the backscatter rule via sample bounce messages. https://wiki.apache.org/spamassassin/VBounceRuleset https://forums.untangle.com/feedback/11356-backscatter-spamassassin.html
Update 3:
After some time we realized the above settings , does not fulfill our requirement and had to modify the sendmail.mc as below
Original Config
define(confPRIVACY_FLAGS',authwarnings,novrfy,noexpn,restrictqrun’)dnl
Change it to
define(confPRIVACY_FLAGS',authwarnings,nobodyreturn’)dnl#
Compile the sendmail and restart the sendmail services.
Nutanix: fatal mounting installer media
Last week , we were doing the foundation on NX-1365-G6 block. The foundation process hangs at 26% with error fatal mounting installer media. When this happens, the Nutanix nodes are being powered off. I have attached 2 screenshots below that depicts the problem we faced.
You could see it in the images , that the IPMITool is trying to restart the server and failing to do so.
Therefore , to overcome this situation , we logged in to IPMI on each node and did a Uni Reset & Factory Default via the Maintenance Menu. Thereafter we restarted the foundation from scratch and it got completed successfully.
AsBuilt Report for VSphere
Hi Folks
Until recent years , I was struggling to build a proper AsBuilt Document for VSphere environments. As the manual process requires capturing screenshots and time consuming word document preparations.
Last week , I came across 2 blogs talking about this AsBuilt tool for VMware which turned out to be very handy and must have tool for VMware installations .
For those who want to read more about this tool, could visit the 2 blogs that are listed at the bottom of this page.
You need Windows PowerShell. Once you are ready with the PowerShell run the below commands to build your AsBuilt document .
1) Install the PSCribo Module
#Install-Module PSCribo
2)Download the AsBuilt PowerShell Scripts via https://github.com/tpcarman/As-Built-Report
2.1)Extract it to a Folder
#Import-Module C:\As-Built-Report-dev\AsBuiltReport.psd1
3)Install PowerCLI Module
#Find-Module -Name VMware.PowerCLI
#Install-Module -Name VMware.PowerCLI
3.1)Run the below command to bypass SSL warning for VCenter/ESXi
#Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false
4) Below command will create the Report
New-AsBuiltReport -Target vcenterip -Credential (get-credential) -Type vSphere -Format HTML,Word -TimeStamp imeStamp -Healthchecks -AsBuiltConfigPath C:\As-Built-Report-dev\Src\Public\Reports\vSphere\vsphere.json
Source:
https://www.timcarman.net/as-built-report/
As Built Report – working with it in my lab
