DCPromo Fails – The directory service is missing mandatory configuration information

Last week , we worked on a AD migration project. This project involved deploying a Windows 2016 based Domain Controller and then decommission the Windows 2008R2 domain controller.

We successfully transferred the FSMO roles . During the decommissioning process when we ran the dcpromo command we received the error “The directory service is missing mandatory configuration information”

During the troubleshooting the  MS KB (https://docs.microsoft.com/en-us/troubleshoot/windows-server/identity/dcpromo-demotion-fails) was pointing us to correct direction. The issue was related to  fsmoroleowner attribute  on CN=Infrastructure is not set properly. In my case it was pointing to the server which I am trying to demote.

You can see this by opening ADSI Edit;

Right click the ADSI Edit root and click on Connect to…
Use the following connection point: DC=DomainDNSZones,DC=abc,DC=local  (Replace it with your actual AD DNS Zone)
Click on Default Naming Context [DC.abc.local] to populate it.
Click on DC=DomainDNSZones,DC=abc,DC=local folder.
Double click on CN=Infrastructure.
Locate the fSMORoleOwner attribute

Ensure you connect to DC=ForestDNSZones as well to verify the attribute.

In my case DomainZones was showing the correct DC .But the ForestDNSZones pointing to the Windows 2008R2 Server.

I have tried the manual method using the ADSI Edit to change the value. However it was failing with the error  “The role owner attribute could not be read”

In this case you need to refer the MS KB https://docs.microsoft.com/en-us/troubleshoot/windows-server/identity/dcpromo-demotion-fails to create the .vbs file to fix this issue.( I have seen suggestions to run the dcpromo /forceremoval instead and then run a metadata cleanup. I do not recommend this approach)

The script provided in the KB does not work due to incorrect end statements. Luckily the Blogger veducate.co.uk (https://veducate.co.uk/dcpromo-fails-missing-mandatory-configuration/) have provided a fixed version.

NOTE: You need to  run these commands from the current owner of the FSMO roles.

Create a .vbs file via CMD
fsutil file createnew fixfsmo.vbs 0

Copy the below contents to the file

const ADS_NAME_TYPE_1779 = 1

set inArgs = WScript.Arguments

if (inArgs.Count = 1) then
‘ Assume the command line argument is the NDNC (in DN form) to use.
NdncDN = inArgs(0)
Wscript.StdOut.Write “usage: cscript fixfsmo.vbs NdncDN”
End if

if (NdncDN <> “”) then

‘ Convert the DN form of the NDNC into DNS dotted form.
Set objTranslator = CreateObject(“NameTranslate”)
objTranslator.Init ADS_NAME_INITTYPE_GC, “”
objTranslator.Set ADS_NAME_TYPE_1779, NdncDN
strDomainDNS = objTranslator.Get(ADS_NAME_TYPE_CANONICAL)
strDomainDNS = Left(strDomainDNS, len(strDomainDNS)-1)

Wscript.Echo “DNS name: ” & strDomainDNS

‘ Find a domain controller that hosts this NDNC and that is online.
set objRootDSE = GetObject(“LDAP://” & strDomainDNS & “/RootDSE”)
strDnsHostName = objRootDSE.Get(“dnsHostName”)
strDsServiceName = objRootDSE.Get(“dsServiceName”)
Wscript.Echo “Using DC ” & strDnsHostName

‘ Get the current infrastructure fsmo.
strInfraDN = “CN=Infrastructure,” & NdncDN
set objInfra = GetObject(“LDAP://” & strInfraDN)
Wscript.Echo “infra fsmo is ” & objInfra.fsmoroleowner

‘ If the current fsmo holder is deleted, set the fsmo holder to this domain controller.

if (InStr(objInfra.fsmoroleowner, “\0ADEL:”) > 0) then

‘ Set the fsmo holder to this domain controller.
objInfra.Put “fSMORoleOwner”, strDsServiceName

‘ Read the fsmo holder back.
set objInfra = GetObject(“LDAP://” & strInfraDN)
Wscript.Echo “infra fsmo changed to:” & objInfra.fsmoroleowner

End if

End if


Run the file twice as below
1) cscript fixfsmo.vbs dc=forestdnszones,dc=abc,dc=local

2) cscript fixfsmo.vbs dc=domaindnszones,dc=abc,dc=local

Voila , The fsmoroleowner attribute got updated with the correct server name , and  I was able to demote the server successfully.

Source: (Helped me to fix the syntax errors on script provided by Microsoft)

DCPromo Fails – The directory service is missing mandatory configuration information




November 19, 2020 at 2:56 pm Leave a comment

How to create a Ubuntu 18.04.5 template in VSphere ESXi 6.7

As usual you need to create a VM and install Ubuntu in to it. Thereafter you need to follow the below steps to strip out the unique data being propagated to the VM’s you create from the template.

+Update the OS
#sudo apt -y update
#sudo apt -y upgrade

+Clear audit logs
truncate -s0 /var/log/wtmp
truncate -s0 /var/log/lastlog

+Clear the tmp
#rm -rf /tmp/*
rm -rf /var/tmp/*

+Clear the SSH
#rm -f /etc/ssh/ssh_host_*

+Reset the hostname
#sed -i ‘s/preserve_hostname: false/preserve_hostname: true/g’ /etc/cloud/cloud.cfg
#truncate -s0 /etc/hostname
#hostnamectl set-hostname localhost

+Clean apt
#apt clean

+Remove the default *.yaml file from the /etc/netplan. The VMware customizaion will create it’s own file 99-netcfg-vmware.yaml.
#rm -f /etc/netplan/*.yaml

+Reset the machine id
#echo -n > /etc/machine-id

+Clear the history & shutdown the VM
#history -c
#shutdown -h now

On the vCenter you need to create a VM Customization specification to be used when deploying the VM’s from the template(Refer:How to create a RHEL 7 template in VSphere ESXi 6.7 for steps).

However , when I created the VM’s from this template I realised that

  • All the VM’s are having the same hostid. Some applications uses the hostid in their licensing .Thus, it has to be unique across the VM’s. Since I did not find a way to solve this while deploying the VM’, I had to do a manual modification as mentioned below on the VM’s after it being created.

    #vi /etc/hosts
    replace the line “ localhost” with
    your IP FQDN Hostname (E.g: server.test.local server).

  • You need to manually select the “Connected” option in The VM network Adapter in the VM properties.

October 14, 2020 at 4:19 pm Leave a comment

The Microsoft Exchange Replication service couldn’t find a valid configuration for database ‘mydb’ on server ‘server1’. Error: An Active Manager operation failed. Error: The active copy for database could not be determined:

Recently one of our customer had an Exchange Server outage. Whereas one of the node from 2 node DAG was not able to communicate with other server. We tried to rebooting the server but no luck . The symptoms were

– Cluster service was not starting.
– On the other node FCM was not able to connect to the DAG cluster.
– Unable to open the ECP/OWA.
– Outlook / Mobile Users cannot access their mailboxes.

The environment consist of 2 Exchange Server 2016 & DAG.

So we started with removing the failed node by typing the below commands in the healthy exchange server:
# Open Exchange PowerShell
#Get-ClusterNode -Name node1 | Remove-Clusternode
# net stop clussvc
# net start clussvc 
# Remove-DatabaseAvailabilityGroupServer -Identity “DAG Name” -MailboxServer “Failed Exchange Server Name” -ConfigurationOnly
# Get-ClusterNode “Failed Exchange Server Name” | Remove-ClusterNode

However , still no luck. Then we tried to remove the mailbox database copies form the failed node(I believe the copies were active on the second server) we got a new error stating that “Mailbox databse copies cannot be disabled on the Databases with circular logging enabled. So we had to remove the circular logging and then delete the database copies.

#Get-MailboxDatabse | Set-MailboxDatabase -CircularLoggingEnabled $False
# Get-MailboxDatabaseCopy -Identity “DatabaseName” |Remove-MailboxDatabaseCopy -Identity “DatabaseName\ServerName”

Now it is the time for mounting the databses: When we ran the below command

#Get-MailboxDatabase | Mount-Database

We received the below error”

Failed to mount database “———-“. Error: An Active Manager operation failed. Error: An Active Manager operation
encountered an error. To perform this operation, the server must be a member of a database availability group, and the
database availability group must have quorum. Error: Automount consensus not reached (Reason: FSW boot time did not
match (FSW-Remote: 2020-10-01T11:37:09.4930830Z FSW-Reg: 0001-01-01T00:00:00.0000000)). [Server:———-]
+ CategoryInfo : InvalidOperation: (—-:ADObjectId) [Mount-Database], InvalidOperationException
+ FullyQualifiedErrorId : [Server=————,RequestId=5022acdd-0c48-4584-b2eb-1d0a2c692f0d,TimeStamp=10/7/2020
11:09:14 AM] [FailureCategory=Cmdlet-InvalidOperationException] BCF4F25E,Microsoft.Exchange.Management.SystemConfi

Here , we wanted to check the DAG status

#Get-DatabaseAvailabilityGroup -status

Output :

WARNING: Unable to get Primary Active Manager information due to an Active Manager call failure. Error: An Active
Manager operation failed. Error: An Active Manager operation encountered an error. To perform this operation, the
server must be a member of a database availability group, and the database availability group must have quorum. Error:
Automount consensus not reached (Reason: FSW boot time did not match (FSW-Remote: 2020-10-01T11:37:09.4930830Z FSW-Reg:
0001-01-01T00:00:00.0000000)). [Server: ———-]

Name : DAGName
Memberservers: {Server1}
Operational Servers {}

The output confirmed that there a no operatinal servers. So we had to restart the DAG with the surviving node.

#Start-DatabaseAvailabilityGroup -Identity “DAGNAME” -MailboxServer ServerName”

Now the Get-DatabaseAvailabilityGroup -status cmd will gave the below output:

Name : DAGName
Memberservers: {Server1}
Operational Servers {Server1}

Now it is the time for mounting all the databases:

Get-MailboxDatabase | Mount-Database

Voila , The MailboxDatabases were started mounting and the ECP access is restored.

October 7, 2020 at 3:39 pm Leave a comment

Free Radius : Auth: (0) Invalid user (Rejected: User-Name contains multiple ..s): [xxx]

Recently we performed a Yum update on our Free Radius Server. The newer version is FreeRADIUS Version 3.0.13. At that time , we started receiving complains from the users that they are not able to login to the network devices.

After reviewing the logs , we noticed the error “Auth: (0) Invalid user (Rejected: User-Name contains multiple ..s)” being logged during the authentication. This is due to the changes applied in the filter file(/etc/raddb/policy.d/filter ).

Before Update:

if (&User-Name =~ /\\.\\./ ) {

After Update:

if (&User-Name =~ /\.\./ ) {

The new Regex syntax style doesn’t escape backslashes anymore. So you need to ensure that the correct_escapes = true property is set in /etc/raddb/radiusd.conf.


September 27, 2020 at 11:34 am Leave a comment

How to update ESXi 6.x using an Offline Bundle

We all know there are multiple ways to upgrade the ESXi to the desired version. In this post I am going to discuss about upgrading the ESXi using an offline bundle. In my opinion this method is much convenient when you are having a single ESXi host , or you want a more controlled upgrade process. The steps are very minimal.(Please make sure that , you have powered off all the VM’s and the Host is put to the Maintenance mode)

– Download the ESXi bundle and upload it to a Datastore.
– Login to the ESXi via SSH.

Run the below command to verify the bundle

#esxcli software sources profile list -d /vmfs/volumes/datastore1/VMware-ESXi-6.7.0-Update3-15160138-HPE-Gen9plus-670.U3.


Thereafter , run the below command to install the file.

#esxcli software profile update -p HPE-ESXi-6.7.0-Update3-Gen9plus-670.U3. -d /vmfs/volumes/datastore/VMware-ESXi-6.7.0-Update3-15160138-HPE-Gen9plus-670.U3.

-p xxxxxxxxx (Name of the ESXi bundle as per the output).

Once the process is completed you need to reboot the server(The host may restart automatically for the 2nd time as well).

Update1: In case if you receive an error “Could not find a trusted signer” ., when you try to install you can run the same command with –no-sig-check as below:

#esxcli software profile update -p HPE-ESXi-6.7.0-Update3-Gen9plus-670.U3. -d /vmfs/volumes/datastore/VMware-ESXi-6.7.0-Update3-15160138-HPE-Gen9plus-670.U3. –no-sig-check

September 21, 2020 at 3:00 pm Leave a comment

How to properly update kernel in RHEL/CentOS 7/8 Linux

When you perform a yum update in CentOS 7 or higher , you will notice the newer kernel version is not reflected. You can use the below commands to verify the current version.

#uname -r

You can confirm whether the new kernel is installed successfully or not by using the command below;

#yum list kernel
#grep saved /boot/grub2/grubenv

In order to apply the kernel you need to restart the server.

In some instances you need to manually set the correct kernel version to boot by using the below commands;

#awk -F\’ ‘$1==”menuentry ” {print i++ ” : ” $2}’ /etc/grub2.cfg

(If you have multiple versions installed all of them will be listed with 0 , 1 & etc. In the example the newer version is 0,)

#sudo grub2-set-default 0

Source1: https://www.howtoforge.com/
Source2: https://www.golinuxcloud.com/

September 20, 2020 at 2:25 pm Leave a comment

“ Error 432 4.3.2 STOREDRV.Deliver; recipient thread limit exceeded in Exchange 2016

One of our customer complained that the Internal E-mails are not being delivered. When we checked the Queue Viewer we observed the emails are stuck at the queue with the error “Error- 432 4.3.2 STOREDRV.Deliver; recipient thread limit exceeded”

This could be due to large number of emails being sent internally(either to one mailbox / several mailboxes).In our case , the customer’s processing server was down for some days, and when it was restored it started sending all the backlog emails.

To overcome this issue we had to temporarily disable the throttling , by adding the below text in the EdgeTransport.exe.config file(located in the Exchange Bin Folder)

add key=”MailboxDeliveryThrottlingEnabled” value=”False”

(Some articled recommends to add the value in the MSExchangedelivery.exe.config file instead).

Thereafter , you need to restart the MS Exchange Transport & Exchange Mailbox Transport Delivery services.

In case if you don’t completely disable the throttling you could add the below keys.

add key=”RecipientThreadLimit” value=”2″
add key=”MaxMailboxDeliveryPerMdbConnections” value=”3″

Good Luck.

September 17, 2020 at 3:33 pm Leave a comment

How to modify the iSCSI initiator ID in Linux

When you deploy Linux VM’s using a Template (in ESXi) ,you may come across a situation , where the iSCI initiator ID on these VM’s will be identical. To resolve this issue we need to modify the iscsi initiator ID’s.

In case if you have logged in to the iscsi session already you need to log out first.

#iscsiadm -m node -T iqn.xxxxxxxxxxxxxx -p iscsiserver-ip -u


backup the file initiatorname.iscsi
#cp /etc/iscsi/initiatorname.iscsi /etc/iscsi/initiatorname.iscsi.bak

#echo “InitiatorName=`/sbin/iscsi-iname`” > /etc/iscsi/initiatorname.iscsi

You can login again to the iscsi session
#iscsiadm -m node -T iqn.xxxxxxxxxxxxxx -p iscsiserver-ip -l


September 15, 2020 at 8:47 am Leave a comment

How to view the Network Configuration in AHV

Use the following commands to view the configuration of the network elements.

Before you begin

Log on to the Acropolis host with SSH.

  • To show interface properties such as link speed and status, log on to the Controller VM, and then list the physical interfaces.

    nutanix@cvm$ manage_ovs show_interfaces

    Output similar to the following is displayed:
name mode link speed 
eth0 1000 True 1000 
eth1 1000 True 1000 
eth2 10000 True 10000 
eth3 10000 True 10000
  • To show the ports and interfaces that are configured as uplinks, log on to the Controller VM, and then list the uplink configuration.

    nutanix@cvm$ manage_ovs –bridge_name bridge show_uplinks

    Replace bridge with the name of the bridge for which you want to view uplink information. Omit the –bridge_name parameter if you want to view uplink information for the default OVS bridge br0.Output similar to the following is displayed:
Bridge: br0
  Bond: br0-up
    bond_mode: active-backup
    interfaces: eth3 eth2 eth1 eth0
    lacp: off
    lacp-fallback: false
    lacp_speed: slow
  • To show the bridges on the host, log on to any Controller VM with SSH and list the bridges:

    nutanix@cvm$ manage_ovs show_bridges

    Output similar to the following is displayed
  • To show the configuration of an OVS bond, log on to the Acropolis host with SSH, and then list the configuration of the bond.

    root@ahv# ovs-appctl bond/show bond_name

    For example, show the configuration of bond0.

    root@ahv# ovs-appctl bond/show bond0

    Output similar to the following is displayed:
---- bond0 ----
bond_mode: active-backup
bond may use recirculation: no, Recirc-ID : -1
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
lacp_status: off
active slave mac: 0c:c4:7a:48:b2:68(eth0)

slave eth0: enabled
        active slave
        may_enable: true

slave eth1: disabled
        may_enable: false

September 7, 2020 at 12:17 pm Leave a comment

How to fix the disk usage warning when /home partition or /home/nutanix directory is full

Source: https://portal.nutanix.com/page/documents/kbs/details?targetId=kA0600000008dpDCAQ


This article describes ways to safely free up space if /home or /home/nutanix becomes full or does not contain enough space to facilitate an AOS upgrade or PCVM upgrade.

Versions affected:

ALL Prism Central Versions, ALL AOS VersionTroubleshootingUpgrade


WARNING: DO NOT treat the Nutanix CVM (Controller VM) or PCVM as a normal Linux machine. DO NOT perform “rm -rf /home” on any of the CVMs or PCVM. It could lead to data loss scenarios. Contact Nutanix Support in case you have any doubts.

This condition can be reported in two scenarios:

  • The NCC health checkdisk_usage_check reports that the /home partition usage is above a certain threshold (by default 75%)
  • The pre-upgrade check test_nutanix_partition_space checks if all nodes have a minimum of 5.6 GB space on the /home/nutanix directory before performing an upgrade

The following error messages will be generated in Prism by the test_nutanix_partition_space pre-upgrade check:

Not enough space on /home/nutanix directory on Controller VM [ip]. Available = x GB : Expected = x GB
Failed to calculate minimum space required
Failed to get disk usage for cvm [ip], most likely because of failure to ssh into cvm
Unexpected output from df on Controller VM [ip]. Please refer to preupgrade.out for further information

Nutanix reserves space on the SSD-tier of each CVM for its infrastructure. These files and directories are located in the /home folder that you see when you log in to a CVM. The size of the /home folder is capped at 40 GB so that the majority of the space on SSD is available for user data.

Due to the limited size of the /home partition, it is possible for it to run low on free space and trigger Prism Alerts, NCC Health Check failures or warnings, or Pre-Upgrade Check failures. These guardrails exist to prevent /home from becoming completely full, as this causes data processing services like Stargate to become unresponsive. Clusters with multiple CVMs having 100% full /home partition will often result in downtime for user VMs.

The Scavenger service running on each CVM is responsible for the automated clean-up of old logs in /home and improvements to its scope were made in AOS 5.5.9, 5.10.1, and later releases. For customers running earlier AOS releases, or in special circumstances, it may be necessary to manually clean up files out of certain directories in order to bring space usage in /home down to a level that will allow future AOS upgrades.

When cleaning up unused binaries and old logs on a CVM, it is important to note that all the user data partitions on each drive associated with a given node are also mounted within /home. This is why we strongly advise against using undocumented commands like “rm -rf /home”, since this will also wipe the user data directories mounted within this path. The purpose of this article is to guide you through identifying the files that are causing the CVM to run low on free space and removing only those which can be safely deleted.


WARNING: DO NOT treat the Nutanix CVM (Controller VM) as a normal Linux machine. DO NOT perform “rm -rf /home” on any of the CVMs. It could lead to data loss scenarios. Contact Nutanix Support in case you have any doubts.

Step 1: Parsing the space usage for “/home”.

Log in to CVM, download KB-1540_clean_v7.sh to /home/nutanix/tmp directory, make it executable and run it.

KB-1540_clean_v7.sh has some checks (MD5, compatibility, etc.) and deploys KB-1540_clean_v7.sh script accordingly.

nutanix@cvm:~$ cd ~/tmp
nutanix@cvm:~/tmp$ wget http://download.nutanix.com/kbattachments/1540/KB-1540_clean_v7.sh
nutanix@cvm:~/tmp$ mv KB-1540_clean_v7.sh KB-1540_clean.sh
nutanix@cvm:~/tmp$ chmod +x KB-1540_clean.sh
nutanix@cvm:~/tmp$ ./KB-1540_clean.sh

You can select to deploy the script to the local CVM or all CVMs.

Select package to deploy
     1 : Deploy the tool only to the local CVM
     2 : Deploy the tool to all of the CVMs in the cluster
    Selection (Cancel="c"):

Run the script to get a clear distribution of partition space usage in /home.

nutanix@cvm:~/tmp$ ./nutanix_home_clean.sh

Step 2: Check for files that can be deleted from within the list of approved directories.

PLEASE READ: The following are the ONLY directories within which it is safe to remove files. Take note of the specific guidance for removing files from each directory. Do not use any other commands or scripts to remove files. Do not use “rm -rf” under any circumstances.

  1. Removing Old Logs and Core Files Before removing old logs, check to see if you have any open cases with pending RCAs (Root Cause Analysis). The existing logs might be necessary for resolving those cases and you should check with the owner from Nutanix Support before cleaning up /home. Only delete the files inside these directories. Do not delete the directories themselves.
    • /home/nutanix/data/cores/
    • /home/nutanix/data/binary_logs/
    • /home/nutanix/data/ncc/installer/
    • /home/nutanix/data/log_collector/
    Use this syntax for deleting files within each of these directories: nutanix@cvm:~$ rm /home/nutanix/data/cores/*
  2. Removing Old ISOs and Software Binaries Begin by confirming the version of AOS that is currently installed on your cluster by running the command below. Make sure never to remove any files that are associated with your current AOS version. You will find this under the “Cluster Version” field in the output of the command shown below. nutanix@cvm:~$ ncli cluster info Example output: Cluster Name : Axxxxa Cluster Version : 5.10.2 Only delete the files inside these directories. Do not delete the directories themselves.
    • /home/nutanix/software_uncompressed/ – Delete any old versions other than the versions you are currently upgrading. The software_uncompressed folder is only in use when the pre-upgrade is running and should be removed after a successful upgrade. If you see a running cluster which is currently not upgrading, it is safe to remove everything underneath software_uncompressed
    • /home/nutanix/foundation/isos/ – Old ISOs of hypervisors or Phoenix.
    • /home/nutanix/foundation/tmp/ – Temporary files that can be deleted.
    Use this syntax for deleting files within each of these directories: nutanix@cvm:~$ rm /home/nutanix/foundation/isos/* If you see large files in the software_downloads directory that are not needed for any planned upgrades, do not remove those from the command-line. Instead, use the Prism Upgrade Software UI to accomplish as shown below. This example lists multiple versions of AOS which consume around 5 GB each, simply click on the ‘X’ to delete the files. Then click on each of the following tabs including File Server, Hypervisor, NCC, and Foundation to locate further downloads you may not require. It is possible that Enable Automatic Download is checked. This is located below the above screenshot (on the AOS tab). Left unmonitored, the cluster will download multiple versions, consuming more space in the home directory.

Step 3: Check space usage in /home to see that it is now below 70%.

You can use the “df -h” command to check on the amount of free space in /home. To accommodate a potential AOS upgrade, usage should ideally be below 70%.

nutanix@cvm:~$ allssh "df -h /home"

Example output:

================== x.x.x.x =================
/dev/md2         40G  8.4G   31G  22% /home
================== x.x.x.x =================
/dev/md2         40G  8.5G   31G  22% /home
================== x.x.x.x =================
/dev/md2         40G   19G   21G  49% /home

Cleaned up files from the approved directories but still see high usage in /home?

Contact Nutanix Support and submit the script log bundle (/tmp/home_kb1540_<cvm_name>_<timestamp>.tar.gz). One of our Systems Reliability Engineers (SREs) will promptly assist you with identifying the source of and solution to the problem at hand. Under no circumstances should you remove files from any other directories aside from those found here as these may be critical to the CVM infrastructure or may contain user data.

For the home partition exceeding its limit on the PCVM refer to the KB-8950 to troubleshoot.

September 7, 2020 at 12:11 pm Leave a comment

Older Posts



Follow Hope you like it.. on WordPress.com

Blog Stats

  • 35,624 hits

%d bloggers like this: