A server-side database availability group administrative operation failed. Error The operation failed. CreateCluster errors may result from incorrectly configured static addresses

Recently , we were working with one of our customer to build the Exchange 2019 DAG. When we add the 2 Exchange Nodes to the DAG , the process got failed with the error “A server-side database availability group administrative operation failed. Error The operation failed. CreateCluster errors may result from incorrectly configured static addresses – Error: Windows Failover Clustering isn’t installed on ‘xxxxxx’.. [Server: xxxxxxxxx] – Restart the server to complete the failover cluster installation also check for the duplicate mac , incorrect IP address, if it is a VM make sure you have done sysprep.”

However , we were pretty sure that the servers were created using a fresh installation instead of syprepped images. Also , we noticed that the Server Manager on both the servers were displaying a message ” Restart is Pending”.

Therefore , we simply restarted the server , and thereafter we were able to add the 2 exchange servers to the DAG.

November 17, 2022 at 2:13 pm Leave a comment

How to migrate the AAD Connect to a new Server

For those who running AAD Connect to Synchronize their On-Premise AD , may come across a requirement to upgrade the OS on the AAD connect server. In this situation , the safest approach would be to install a new server with latest OS version and then migrate the AAD connect with the below mentioned steps safely without facing lengthy outages.

  • Prepare a new server with the Operating System.
  • Install the AAD Connect (Preferably the same version as the existing or the latest version available).
  • Now , Prior to proceed with the migration you need to ensure two things.

    – Take a backup of existing configuration using the AAD Connect –> View or Export configuration
    –>Click on Export Settings. This will create a json file in X:\Program Data\AADConnect. (Copy it to the new server)
    – Secondly use the Azure AD Sync Configuration Documenter to collect the existing configuration in HTML format.(https://github.com/Microsoft/AADConnectConfigDocumenter/releases) . The Installation instruction can be found in https://github.com/Microsoft/AADConnectConfigDocumenter/wiki.

Thereafter , proceed with the installation of AADC on the new server , select the Customize option (instead of Express) –> Import Synchronization settings ,and import the configuration using the json filed copied in the above step and press Next.

In the last screen of the AADC Installation wizard , select the below options.
– Start the synchronization process when the configuration completes.
– Enable staging mode.

Now , you using the AADC configuration documenter capture the settings on the new AADC server. Then run the below command to compare both the configuration.(Refer the wiki for instructions) after copying both the files in the same location.

AzureADConnectSyncDocumenterCmd.exe “AADC-SERVER-OLD” “AADC-SERVER-NEW”

After reviewing the output and confirming the configuration are identical proceed with the final steps in the migration.

– Enable the staging mode on the Old AADC server.(by running the AADC – Tasks- Configure Staging mode).
– Disable the staging mode on the New AADC server.
– Perform a test and confirm the synchronization is working as expected.
– Uninstall the AADC from the old server and proceed with the decommission.

October 8, 2022 at 10:52 pm Leave a comment

The trust relationship between this workstation and the primary domain has failed

Very frequently , you may have faced the above error while trying to login to domain from your PC / Server login. Additionally ,when you review the Event viewer the NETLOGON Event ID 3210 , 130 & 8019 would have been logged as well.

In this case(Incase , if you have credentials for the local administrator).
– Login as Local Administrator
– Open the Powershell
– Run the PS command Test-ComputerSecureChannel ( The output will be false)
This will prove the Secure channel to the AD is broken, unlike earlier , whereas we need to disjoin the workstation/station from the domain and rejoin it , we can use the same PS command to fix the issue.

– Type : $cred=Get-Credential (Enter the username and the password of domain administrator or any user who has rights to add computers to the domain)
– Test-ComputerSecureChannel -Credential $cred -Repair (You will get the output as True).

Restart the computer and everything should be working as normal.

NOTE: – Incase if you skip the credential variable and try to enter the credentials manually , the command will fail.

May 31, 2022 at 11:56 am Leave a comment

How to Remove office 365 mailbox without deleting user account

We were planning to deploy an Exchange Hybrid Configuration with On-Premise Exchange 2019 servers. Thereafter , we will be moving some of the mailboxes to On-Premise Exchange
Servers and disable only the Exchange Online mailbox to keep the user account intact with the license so he could use other Applications (like SharePoint Online , OneDrive and Desktop Apps ,etc).
The environment is already synced via Azure AD Connect.

Since the Exchange Online license is assigned to the user , the user could have one mailbox in Office 365 and one in an on-premises Exchange ,causing mail delivery issues . Thus , to solve the issue
we need to

1) Remove the Exchange Online License for the user.
2) Clear the mailbox info:
Set-User user@tga.gov.sa -PermanentlyClearPreviousMailboxInfo
3) Resync and Ensure the MSExchangeGuid is Synced to point that now the Mailbox is in On-Premise Server.
4) Re-assign the Exchange Online License in O365.
5) Now the user will not be having a mailbox , But, he will be able to access the SharepointOnline , OneDrive , the DesktopApps , etc.

OR , we could use the below steps as well (Recommended by the MS Support)

– Migrate the O365 mailbox to On-Premise Exchange Server.
– Go to the License & Apps under the User property in O365 portal.
– Unselect the Exchange Online (Plan 2) from the Apps list.
– This will ensure that the user mailbox is only in On-Premise Exchange Server and still he will be able to access the SharepointOnline, OneDrive and DesktopApp ,etc)

May 31, 2022 at 11:42 am Leave a comment

Cross VCenter VMotion fails with the error “A specified parameter was not correct: path”

Recently , we had to migrate some VM’s from a VSphere 6.7(17499825) to 7.0 U2 cluster. We leveraged the Import VM’s feature that utilizes the Cross VCenter VMotion capability.


The source: VCenter 7.0
The destination: VCenter 6.7

Even though , the required access between the clusters were allowed(based on https://kb.vmware.com/s/article/2106952) , when we tried to trigger the import VM’s there were no process or tasks being generated in both vcenters. However , we noticed a failed task on the VM that we are trying to import in the destination VCenter( under the tasks and events for the particular VM )wiith the message “A specified parameter was not correct: path”.

We started reviewing the hostd.log on both sides but no avail. Finally , out of exhaustion , we tried to changed the source datastore to a different one(In our scenario the datastore names were identical on both sides) and voila the import VM’s got kicked in and completed successfully. Still , I am not sure the cause for this behavior where the import of VM’s failing due to identical datastore names on both sides , so we have opened a case with VMware support to find out the root cause. I will updating this post as soon as I receive the response.

Tip: Also during the import VM’s wizard (3rd step) where we select the desired to host to place the imported VM on the source cluster we were receiving a SSL related error. This was resolved after disabling the SSL inspection on our firewall

March 28, 2022 at 12:11 pm 6 comments

How to migrate Windows Cluster VMs (incl. RDMs) from old to new storage with minimal downtime?

Recently , we did a storage upgrade for one of our customer. The VM migration was pretty easy for almost 90% of the workloads. However , the remaining 10 % of the VM’s were having RDM disks ,and for the standalone RDM disks (non clustered workloads) we were able to leverage the Storage VMotion to convert the RDM’s to VMDK and migrate it.

On the other hand , for clustered VM’s , Initially we were planning to utilize the RP4VM to migrate the VM’s(Failover to the Replica). But this approach was not fruitful as we faced a compatibility issue and there was no workaround and we dropped the idea.

During this time our colleague from the Storage Team suggested this alternate plan and we tested it and it was successful. The steps are here as listed.

make a note of the RDM mapping to the VMs (note the SCSI ID assigned to the RDM in the VM configuration) 

# shutdown the VMs of that Virtual cluster.

# unmap the RDMs.

# Create the Luns on the destination array (must be at least as large as the source !!!)

# present to ESXhost (rescan, …)

# use Storage vMotion to move the VM to the new datastores.(This is to move the VM with the OS disk and any other non RDM based disks)

# use ESX CLI vmkfstools to copy the RDM content to the new lun and this will automatically create the new RDM vmdk pointer file (destination.vmdk)

vmkfstolls -i <srcdisk.vmdk> -d rdmp:device <destination.vmdk>

Example:
vmkfstools -i TestVM_RDM1.vmdk -d rdmp:/vmfs/devices/disks/vml.02000100006006016044440000f8b164674b51e111565241494420 TestVM_NewRDM.vmdk

   ==== remap the new LUN as RDMp with the same SCSI ID====

      === don’t forget to set the bus sharing if it disappeared===

Source:

https://www.dell.com/community/VMware/How-to-migrate-Windows-Cluster-VMs-incl-RDMs-from-old-to-new/td-p/6859292

Kudos to EricDeWitte1 (Contributor)

February 14, 2022 at 12:22 pm Leave a comment

How To Find My Public IP Address From Command Line On a Linux

For machines with GUI , Interface there are several ways to obtain the Public IP address. However , in Linux shell environment you need to rely on the commands(Even though there are 3rd party tools available).

Open a terminal window and type the below command:

#dig +short myip.opendns.com @resolver1.opendns.com

OR

dig TXT +short o-o.myaddr.l.google.com @ns1.google.com

Source:https://www.cyberciti.biz/

January 20, 2022 at 11:05 am Leave a comment

What is the most secure way to allow a user read access to a log file

I am running Splunk Enterprise on a Linux Server in our environment. Our Security Standards , prevents the Splunk from running under root User. In this scenario , we will come across a situation , where we will not be able to read the critical Linux logs like audit , messages , secure etc.

There are several methods to achieve this , But in this post I am using the ACL method.

First , you need to set the acl’s on these files located in the /var/log folder. My Splunk is running under the user splunkadmin

# setfacl -m g:splunkadmin:r /var/log/messages
# setfacl -m g:splunkadmin:r /var/log/secure
# setfacl -m g:splunkadmin:r /var/log/maillog

The above changes will not retain , when the logs are rotated. Thus you need to create postrotate action for these logs to retain the ACL's.

1) Create a text file in the folder /etc/logrotate.d/Splunk_ACLs ( My file name is SplunkACLs).
2)  Add the below entries in the file 
{
    postrotate
        /usr/bin/setfacl -m g:splunk:r/var/log/maillog
        /usr/bin/setfacl -m g:splunk:r /var/log/messages
        /usr/bin/setfacl -m g:splunk:r /var/log/secure
     endscript
}

You can verify the ACLS using
#getfacl /var/log/messages

On the other hand , The steps for audit files are different .

Check the current permissions
#ls -l /var/log/audit/audit.log
-rw——- 1 root root 3531590 Jun 1 00:20 /var/log/audit/audit.log

Then edit the auditd.conf file and change the log_group parameter to splunkadmin instead of root.
log_group = splunkadmin

Restart the auditd services
#service auditd restart

You can re-verify the permissions
#ls -l /var/log/audit/audit.log
-rw-r—– 1 root splunkadmin 3532862 Jun 1 00:24 /var/log/audit/audit.log

Source:

https://www.thegeekdiary.com/how-to-change-the-default-permissions-on-var-log-audit-audit-log-file-in-centos-rhel/

https://newbedev.com/what-is-the-most-secure-way-to-allow-a-user-read-access-to-a-log-file

January 5, 2022 at 1:23 pm Leave a comment

Datastore conflicts with an existing datastore in the datacenter that has the same URL” error in vCenter Server

Hi Guys.

Let me share my recent experience with a VMware Upgrade on a Nutanix Cluster. We planned the upgrade from vSphere 6.7 U3 to vSphere 7.0 U1.

As usual we started the Nutanix LCM upgrade and brought the cluster to the latest Firmware & BIOS drivers.

Thereafter , I started upgrading the ESXi , I used the offline .zip file for the upgrade. I followed the below steps.

  • Uploaded the offline bundle VMware-ESXI-7.0U1-16850804-depot.zip to the datastore.
  • Ran the below command to findout the profile name
    esxcli software sources profile list -d /xxxxxxx/xxxxxx/VMware-ESXI-7.0U1-16850804-depot.zip
  • Get the profile name from the output of the above command which is ESXi-7.0.1-1685 and used it in command as below
    esxcli software profile update -d /xxxxxxx/xxxxxx/VMware-ESXI-7.0U1-16850804-depot.zip -p ESXi-7.0.1-1685

Everything went well and rebooted the server but,

  • CVM is shown inaccessible.
  • Local datastore was missing from the server.

As adviced by the Nutanix support , restarted the server for the second time. and I was able to see the local datastore. Still we could not exit from the maintenance mode. As it was giving the error “Datastore conflicts with an existing datastore in the datacenter that has the same URL” error in vCenter Server”.

The issue was resolved after following the VMware KB https://kb.vmware.com/s/article/79623.

To worsen up our situation , for some reason the ESXi version got reverted automatically to the previous build ESXi 6.7 after following the steps in the KB. So we decided to check the boot.cfg files in bootbank & altbootbank locations.

#tail -2 /*bootbank/boot.cfg
It was showing only the ESXi 7.0 U1 only.

Again we contacted the VMware support and resolved the issue with the below steps.

Checked whether achi is disabled or not by typing the command on the ESXi shell.
#grep ahci /etc/vmware/esx.conf
/vmkernel/module/vmw_ahci/enabled = “false”

The output shows the achi is disabled and Enabled it by typing
#esxcfg-module -e vmw_ahci

Thereafter ran the upgrade again using the same steps and rebooted the server.Voila we were able to see the correct version. For some reason the default ahci driver provided from VMware has been disabled.

Sources:
https://anthonyspiteri.net/datastore-esxi7-upgrade-supermicro/

https://kb.vmware.com/s/article/79623

December 9, 2021 at 2:53 pm Leave a comment

The DNS server was unable to open Active Directory”

Recently , One of our customer reported an issue stating that the Exchange Services are failing and Outlook clients are getting disconnected. We noticed few DC related events (Kerberos) on both the exchange servers. Thus , we ran the “netdom query fsmo” command on the 2 Exchange Servers and got the below error:

The same error appeared on all the other domain joined servers. Therefore ,we decided to check the DC’s.

When we reviewed the event viewer on the 2 DC’s , there were DNS related errors(Event ID 4000)

Further , we could not open the DNS MMC snap-ins and pinging the hostname by DC was failing as well. However, the DNS service is started state. In addition to this , there were errors on KDC consistency as well. After , troubleshooting for few minutes ,we go hold of the Microsoft KB :https://docs.microsoft.com/en-us/troubleshoot/windows-server/networking/dns-zones-do-not-load-event-4000-4007 and followed the steps mentioned to resolve the issue successfully.

Few points to consider:
– You will find an additional “d: in the word password in the below command. Do not change it.
netdom resetpwd /server: /userd: netdom resetpwd /server: /userd: /passwordd:*
– In my case I had to run this command on the PDC and the other DC as well
– Stop the KDC service prior to running the command.
– First I started on the PDC and restarted it and ensured the DNS snap-in was accessible and the pinging
by hostname was working.
– Finally , I continued the same steps on the remaining domain controllers.

October 5, 2021 at 10:04 am Leave a comment

Older Posts


Archives

Categories

Follow Hope you like it.. on WordPress.com

Blog Stats

  • 63,246 hits

%d bloggers like this: