Latency between the Nutanix CVM’s
Recently we noticed the Prism was throwing an error stating that there is latency between CVM’s . To investigate the issue we raised a support call with the Nutanix Team. I am sharing the procedures followed by the Nutanix Team as it may help somebody who are facing a similar issue.
# Login to Controller VM
# cd ~nutanix/data/logs/sysstats (This location will contain the ping_hosts & ping_gateway logs)
# tailf ping_hosts.INFO
In our case we noticed there was unreachable on one of the CVM’s
x.x.x.1 : 0.187 ms
x.x.x.2 : Unreachable
x.x.x.3 : 0.028 ms
So we consulted the Network Team and found out that the Switch port where one of the node is connected , conatined lots of errors and we had to replace the cable.
That’s it the problem got resolved.
How to Configure Sendmail & SpamAssassin for SPF Check
We had a Sendmail Server (8.14.7) running on CentOS Server, The server acts as a Secondary MX and SMART hosts for many domains. In this scenario we decided to install the SpamAssassin to force the Sendmail server to validate SPF records prior to accepting the email. I have written the below post to explain the whole process with few notes on troubleshooting I had to perform during the installation & configuration stages.
-Sendmail (already installed and running)
-SpamAssassin v. 3.4.0 (already installed with CentOS , use spamassassin -V to check the version)
– Spam-ass milter
So let’s start with the process;
+ Install spam-ass milter
# yum install perl-Mail-SPF perl-Mail-DKIM perl-Razor-Agent pyzor poppler-utils re2c ( These are the prerequisites)
# Download the RPM from https://centos.pkgs.org/7/epel-x86_64/spamass-milter-0.4.0-7.el7.x86_64.rpm.html and install by rpm -i “rpm name”
+ Start the spamassassin & spamass-milter services
# systemctl start spamassassin
# systemctl start spamass-milter.service
Now we need to force sendmail daemon to use the milter for antispam processing. Add the below lines in sendmail.mc (** do not forget to backup the files before modifying it)
======================================================================================
dnl #
dnl # SPAMASSASSIN dnl
dnl **
dnl ** enable spamassassin-milter to scan for spam using spamassassin **
dnl **
INPUT_MAIL_FILTER(`spamassassin’, `S=unix:/var/run/spamass-milter/spamass-milter.sock, F=, T=C:15m;S:4m;R:4m;E:10m’)dnl
define(`confMILTER_MACROS_CONNECT’,`t, b, j, _, {daemon_name}, {if_name}, {if_addr}’)dnl
define(`confMILTER_MACROS_HELO’,`s, {tls_version}, {cipher}, {cipher_bits}, {cert_subject}, {cert_issuer}’)dnl
dnl # END LOCAL ADDITIONS
dnl #
======================================================================================
+ save the file & quit it
+ Compile the Sendmail configuration & restart the sendmail services.
# m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf ( or you could simply type make)
# systemctl restart sendmail
To confirm whether all these components are working fine with the relevant SPF check you need to run ,
# spamassassin -D < /usr/share/doc/spamassassin-3.4.0/sample-spam.txt 2>&1 |grep -i spf
Thereafter we could analyze whether the email are being filtered properly with the SPF Check, to check that run
# grep spf /var/log/maillog
if it is not functioning well you should look for the errors & start troubleshooting it. In my case it was throwing the below error;
“Mar 4 15:34:20 mail spamd[11685]: spf: lookup failed: addr is not a string at /usr/share/perl5/vendor_perl/IO/Socket/IP.pm line 662.”
After few minutes of googling , we found out that , it was a bug in the perl-socket module in CentOS 7 , thus you need to
# yum install epel
# yum update perl-Socket –enablerepo=cr
You need to restart the sendmail , spamassassin & spamass-milter services for the changes to take effect and review the log again for any errors.
+ A new cron.d job will be created automatically for the spamassassin update in the /etc/cron.d/sa-update file.
Few advice, do not modify any files in /usr/share/spamassassin , since these files will be overwritten with spamassassin updates. Thus always modify the /etc/mail/local.cf for any customizations and it is a system wide configuration.
Secondly you could refer the below samples , that you could use for any customization and whitelisting stuff with in spamassassin.
========================================================================
# How many hits before a message is considered spam.
required_hits 5.0
# Text to prepend to subject if rewrite_subject is used
rewrite_header Subject [*****SPAM*****]
# Encapsulate spam in an attachment
report_safe 1
# Enable the Bayes system
use_bayes 1
# Enable Bayes auto-learning
bayes_auto_learn 1
bayes_path /home/spamd/
bayes_file_mode 0666
# Enable or disable network checks
skip_rbl_checks 0
use_razor2 0
use_dcc 0
use_pyzor 0
# Mail using languages used in these country codes will not be marked
# as being possibly spam in a foreign language.
# ok_languages all
# Mail using locales used in these country codes will not be marked
# as being possibly spam in a foreign language.
# ok_locales all
# Whitelist important senders
whitelist_from *@xyz.xx
========================================================================
That’s it , but during this process i came across useful blog sites and forums posts that helped me to work on this task and they are listed below for your reference as well.
https://blesseddlo.wordpress.com/2010/04/01/sendmail-spamassassin-spamass-milter-milter-greylist/
https://www.rosehosting.com/blog/how-to-install-spamassassin-on-a-virtual-server-with-centos-6/
https://www.jethrocarr.com/2013/10/26/spf-with-spamassassin/
http://forums.sentora.org/showthread.php?tid=1118
https://it.megocollector.com/linux/install-spamassassin-on-centos-6/
http://forum.icewarp.com/forum/showthread.php?1809-Spamassassin-SPF-and-spoofing
https://centos.org/forums/viewtopic.php?t=60477
https://vamsoft.com/support/tools/spf-policy-tester (This will validate you SPF check in the email server)
http://spamassassin.1065346.n5.nabble.com/return-path-test-td1869.html
https://www.howtoforge.com/community/threads/spamassassin-version.74/
Update1:
In January 2018 , barracuda removed the RBL from the SA ruleset (it was under 72_active.cf in /usr/share/spamassassin)
To add this rule , you need to register via the below URL; http://barracudacentral.org/account/register
and then you need to manually edit the local.cf file add the below texts and restart the services
ifplugin Mail::SpamAssassin::Plugin::DNSEval
header __RCVD_IN_BRBL eval:check_rbl(‘brbl’,’bb.barracudacentral.org’)
tflags __RCVD_IN_BRBL net
header __RCVD_IN_BRBL_2 eval:check_rbl_sub(‘brbl’, ‘127.0.0.2’)
meta RCVD_IN_BRBL __RCVD_IN_BRBL_2 && !RCVD_IN_BRBL_LASTEXT
describe RCVD_IN_BRBL Received is listed in Barracuda RBL bb.barracudacentral.org
score RCVD_IN_BRBL 1.2
tflags RCVD_IN_BRBL net
header RCVD_IN_BRBL_LASTEXT
eval:check_rbl(‘brbl-lastexternal’, ‘bb.barracudacentral.org’)
describe RCVD_IN_BRBL_LASTEXT Last external is listed in Barracuda RBL bb.barracudacentral.org
score RCVD_IN_BRBL_LASTEXT 2.2
tflags RCVD_IN_BRBL_LASTEXT net
endif
Update 2:
Recently we were blacklisted by backscatter and the reason for listing was , sending out NDR for non valid emails. Thus we have add the below line in the local.cf configuration file
whitelist_bounce_relays myrelay.mydomain.net (Replace it with your outgoing email server name) If you have multiple servers , you could add them all here in multiple lines . Once the above is added and the spamassassin is restarted , issue the below command to verify for any config errors #spamassassin --lint The below URL contains additional information to test the backscatter rule via sample bounce messages. https://wiki.apache.org/spamassassin/VBounceRuleset https://forums.untangle.com/feedback/11356-backscatter-spamassassin.html
Update 3:
After some time we realized the above settings , does not fulfill our requirement and had to modify the sendmail.mc as below
Original Config
define(confPRIVACY_FLAGS',authwarnings,novrfy,noexpn,restrictqrun’)dnl
Change it to
define(confPRIVACY_FLAGS',authwarnings,nobodyreturn’)dnl#
Compile the sendmail and restart the sendmail services.
Nutanix: fatal mounting installer media
Last week , we were doing the foundation on NX-1365-G6 block. The foundation process hangs at 26% with error fatal mounting installer media. When this happens, the Nutanix nodes are being powered off. I have attached 2 screenshots below that depicts the problem we faced.
You could see it in the images , that the IPMITool is trying to restart the server and failing to do so.
Therefore , to overcome this situation , we logged in to IPMI on each node and did a Uni Reset & Factory Default via the Maintenance Menu. Thereafter we restarted the foundation from scratch and it got completed successfully.
AsBuilt Report for VSphere
Hi Folks
Until recent years , I was struggling to build a proper AsBuilt Document for VSphere environments. As the manual process requires capturing screenshots and time consuming word document preparations.
Last week , I came across 2 blogs talking about this AsBuilt tool for VMware which turned out to be very handy and must have tool for VMware installations .
For those who want to read more about this tool, could visit the 2 blogs that are listed at the bottom of this page.
You need Windows PowerShell. Once you are ready with the PowerShell run the below commands to build your AsBuilt document .
1) Install the PSCribo Module
#Install-Module PSCribo
2)Download the AsBuilt PowerShell Scripts via https://github.com/tpcarman/As-Built-Report
2.1)Extract it to a Folder
#Import-Module C:\As-Built-Report-dev\AsBuiltReport.psd1
3)Install PowerCLI Module
#Find-Module -Name VMware.PowerCLI
#Install-Module -Name VMware.PowerCLI
3.1)Run the below command to bypass SSL warning for VCenter/ESXi
#Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false
4) Below command will create the Report
New-AsBuiltReport -Target vcenterip -Credential (get-credential) -Type vSphere -Format HTML,Word -TimeStamp imeStamp -Healthchecks -AsBuiltConfigPath C:\As-Built-Report-dev\Src\Public\Reports\vSphere\vsphere.json
Source:
https://www.timcarman.net/as-built-report/
As Built Report – working with it in my lab
Nutanix AOS Upgrade Tips
Recently we were upgrading our Nutanix Cluster which was running an AOS version 4.5.2.3 to the latest 5.9. The process was seamless and non interruptive. I have listed the commands we have used along with Nutanix Engineer during the process for future reference.
Initial Check prior to AOS Upgrade
- ncli cluster info
- ncli host ls
- ncli ru ls
- ncli ms ls
- ncc –version
- cluster status | grep -v UP
- nodetool -h 0 ring | grep -i normal | wc -l
- svmips | wc -w
Once the output of above checks are fine Use the Software Upgrade feature from PRISM to upgrade the AOS.
To check the upgrade / pre-upgrade status and on which node is being picked up.& Confirm the versions after upgrade
- allssh ls -ltra ~/data/logs | grep -i preupgrade
- tail -F ~/data/logs/preupgrade.out
- use upgrade_status (to verify the status , less verbose mode)
- ncli –version
- stargate –version
- watch -d genesis status (to check the services status after the CVM reboot)
Optional: To delete the previously uploaded ISO
- cd ~/software_downloads/nos ( use it with allssh to run it on all the CVM’s)
Finally after AOS upgrade sometime the curator replication process and it takes some time to complete . Until it completes we cannot proceed with the other update , thus you could check via the below command;
- curator_cli get_under_replication_info
How to Capture & Analyze Network Traffic on ESXi
Being an ESXI Implementer or an Administrator , you may come across some situations where you need to make your hands dirty 🙂 , with deep network troubleshooting. I had a similar situation few months ago , which I would like to share it in this post.
We deployed the Horizon View (for VDI) in one of our customer’s ESXi Cluster ( 8 Nodes) environment, The Desktop users were complaining about they were not able to specific network .
Thus to further investigate we swapped the Physical Adapter to the on-board BroadCom cards (1Gps). Then we were able to re-establish the network. We thought to engage the VMware Support with the intention to find out the root cause and get a permanent fix. The VMware support was pretty awesome and they were able to nail it very quickly.
First they used the two built-in commands on ESXI , which are
- pktcap-uw (To capture the Network Packets)
- tcpdump-uw ( To read the captured Packets)
They ran the below commands on both the NIC cards to initially capture the traffic.
- pktcap-uw –uplink vmnic0 –dir 0 –mac 00:00:00:00:00:00 —vlan 18 -o /tmp/f.pcap
uplink – Name of the VMnic
dir – 0 means RX Traffic
mac – MAC address of the machine which you are troubleshooting
vlan – The VLAN ID
Thereafter we read the output of the above command using
- tcpdump-uw -ner /tmp/f.pcap
By comparing the output from both NIC’s we were able to narrow down the problem to the Mellanox cards. when tagged traffic passed by on a Mellanox Network Card (10 Gbps), the reply packet was not being tagged with the proper VLAN ID causing disruption to the network traffic.
Good Luck
Muralee
Veeam Backup Repository Sizing
Folks who wants to size the Backup Repository for their Veeam Deployment , could make use of the online sizing tool below for their calculations.
It is very user friendly and output is much descriptive.
Good Luck.
