VCSA 6.5 reset root password

It’s always a very annoying situation when you can’t login into a system anymore because you either don’t know the root password anymore or the system is not able to log you in. The latter can happen if the root mountpoint of the VCSA 6.5 appliance filled up.

As VCSA (vCenter Server Appliance) 6.5 is build on top of Photon OS, you can’t use the same standard procedure you know from Debian, Ubuntu or RedHat. But relax, it’s very simple to reset the root password or clean up a filled up root partition.

First thing to do is to restart your vCenter appliance and wait for the Photon OS Splash screen during boot. 

Hit the letter e to enter the boot menu.

Then change to the GNU GRUB boot menu editor and hit enter.

Next is to add the following string behind the line that starts with linux: rw init=/bin/bash

Hit the F10 function key to boot the changed entry.

Clean up the root partition

In many cases you should check the root partition usage, using the df -h command. Very often the log files grew large and filled up your partition. One well known guy is the audit.log in /var/log/audit. So type ls -sh /var/log/audit to check the file size and rm /var/log/audit/*.log to clean it up.

Reset the root password

If you’re sure that you have to reset the root password, please follow these steps:

passwd 

then enter a strong password twice and remember it.

umount /

reboot -f

It is important to add the -f behind the reboot.

vCenter Server and PSC Deployment Types

Windows based vCenter Server or VCSA can be deployed either with an embedded PSC or an external PSC. You can also deploy a PSC as an appliance or install it on Windows. Depending on your infrastructure needs, it can be mix and match of both.

Before proceeding with installation of vCenter Server Appliance or vCenter Server for Windows, you must determine the deployment model that is suitable for your environment. The deployment types can be categorised in mainly 3 categories:

vCenter Server with an Embedded Platform Services Controller

This is a standalone deployment type that has its own vCenter Single Sign-On domain with a single site. vCenter Server with an embedded PSC is suitable for small environments. Other vCenter Server or PSC instances could not be joined to this vCenter Single Sign-On domain.

vCenter Server with an embedded PSC offers following advantages:

1. The connection between vCenter Server and the Platform Services Controller is not over the network, and vCenter Server is not prone to outages caused by connectivity and name resolution issues between vCenter Server and the Platform Services Controller.
2. If you install vCenter Server on Windows virtual machines or physical servers, you need fewer Windows licenses.
3. You manage fewer virtual machines or physical servers.

Installing vCenter Server with an embedded Platform Services Controller is a recommended solution for small-scale environments.

You can configure the vCenter Server Appliance with an embedded PSC in vCenter High Availability configuration.

Note: A vcenter Server deployed with an embedded PSC can be reconfigured later to change the deployment type to vCenter Server with an external Platform Services Controller. I will cover this part later.

vCenter Server with an External Platform Services Controller

When you deploy or install a Platform Services Controller instance, you can create a vCenter Single Sign-On domain or join an existing vCenter Single Sign-On domain. Joined PSC instances replicate their infrastructure data, such as authentication and licensing information, and can span multiple vCenter Single Sign-On sites.

Multiple vCenter Servers can be registered to one external PSC instance. The vCenter Server instances assume the vCenter Single Sign-On site of the Platform Services Controller instance with which they are registered. All vCenter Server instances that are registered with one common or different joined Platform Services Controller instances which are connected in Enhanced Linked Mode.

Below image shows 2 vCenter Server registered to a common External PSC

Graphic Thanks to VMware.com

Mixed Operating Systems Environment

In a mixed OS environment you can have:

• A vCenter Server instance installed on Windows can be registered with either a PSC installed on Windows or a PSC appliance.
• A vCenter Server Appliance can be registered with either a PSC installed on Windows or a PSC appliance.
• Both vCenter Server and the vCenter Server Appliance can be registered with the same Platform Services Controller.

Below image shows mixed OS environment With an External PSC on Windows

Example of a mixed OS environment with an external PSC Appliance

Thats it for this post. Now we have an idea about available deployment models for vCenter Server/PSC deployment, we will try to explore them in coming posts of this series.

Back to Basics: Install, configure and use vSphere Replication

One of the coolest features that has been included with vSphere 5.1 in my opinion is vSphere Replication. (Make sure to read the what’s new paper) The reason for it being is that it now brings “advanced” technology to everyone (Essentials Plus and upwards). I have used vSphere Replication in 5.0 and it was nice, but with 5.1 the installation and configuration process has been improved. For instance the database is now included in the appliance and it isn’t as DNS sensitive as it was with 5.0. This makes installing and configuring it a matter of minutes.

I am going to assume you have “vSphere Replication” traffic enabled on a VMkernel NIC, if you do not know how to create a VMkernel NIC check this article

Lets get started. I downloaded the vSphere Replication virtual appliance and imported and configured it in just a couple of steps using the vSphere 5.1 Web Client:

• Go to your cluster under “vCenter” —> “Hosts and Clusters”.
• Right click the cluster object and click “All vCenter Actions” —> “Deploy OVF Template”
• As a source I select the ova file I downloaded, now click “Next”
• Validate the details and click “Next”
• If you agree “Accept” the EULA and click “Next”
• Select the “Name and folder” this virtual machine will needs to be placed in and click “Next”
• Select the “Datastore” it needs to be provisioned to and click “Next”
• Select the “Network” it needs to be connected to and click “Next”
• Provide an administrative “password” and enter the “Networking properties” and click “Next”
  
• Click “Next” on the vService bindings, when the binding status is “ok”
• Click “Finish”

Now the vSphere Replication appliance is ready to be powered on. Depending on where you are replicating to there might be some additional steps required. If you are replicating to a second vCenter Server you will need to deploy a vSphere Replication appliance in that environment as well. Note that you will need to link two appliances together before you can replicate anything.

I don’t have a second vCenter Server and I just want to replicate virtual machines to a secondary remote storage device as a form of backup. So I will go ahead and replicate a virtual machine.

• Go to your cluster under “vCenter” —> “Hosts and Clusters”.
• Right click one of your virtual machines, I will use the vCenter Server as an example, and select “All vSphere Replication Actions” and then click on “Configure Replication”.
  
• As a target site select the vCenter Server itself and click “next”.
• As a target location select a datastore and click “next”.
• Decide what the RPO (recovery point objective) should be, I selected 15 minutes and click “next”.
  
• Click “finish”.

Now replication will be configured and the virtual machine will be replicated with an RPO of 15 minutes. Next lets check on the progress of the replica:

• Click on the “Home” button.
• Click on “vSphere Replication” in the upper right.
• Click on “Sites” and then on your vCenter Server instance, in my case “vcenter-tm01”.
• On the “Summary” tab you can see that a virtual machine replication is in progress.
• If you click “View details” you can see some more specifics. It is the first time it is being synced so it will do a full sync as indicated.
• When it is finished it should show a nice green check.

Now if needed you can recover this virtual machine. You can also pause syncing or stop it completely. There is also the option to force an instant sync or even reconfigure the replication process. All of this can be found as follows:

• Click on the “Home” button.
• Click on “vSphere Replication” in the upper right.
• Click on “Sites” and then on your vCenter Server instance, in my case “vcenter-tm01”.
• Click on the “Monitor” tab and next on “Incoming Replications”.
• Right click the appropriate virtual machine.
  
• If you select “Recover” you will notice your virtual machine needs to be powered off before you can recover it.
• Select the “folder” you want to recover your virtual machine to and click “Next”.
• Select the “cluster” and click “Next”.
• Note that your virtual machine will be powered on, but with a disconnected network, click “Finish”.
• Now you should see the status change to “Recovering” and when it is done to “Recovered”.

vSphere 6.5 – How to deploy VCSA 6.5

With the release of vSphere 6.5, users are no longer dependent on the Client Integration Plugin, which was used to deploy the VCSA 6.0.

And trust me, it was a pain. There were a lot of issues with Browser compatibility, plugin not being detected. Well, the team has heard the feedback and the new VCSA 6.5 deployment is no longer dependent on CIP and is deprecated almost, except the Windows credentials log in still requires CIP from the Web Client.

The new VCSA 6.5 installer can be used from Windows, Linux, and Mac and is part of the VCSA ISO image that you will be able to download in the near future.

Once you download the ISO image, extract the contents and navigate to the folder vcsa-cli-installer/win32 and click on the installer.exe file.

As discussed before, the installation of VCSA is a two step process:

• Stage 1: Deploying the VCSA 6.5
• Stage 2: Configuring the VCSA 6.5

Stage 1: Deploying the VCSA 6.5

Deploying the VCSA is made very easy with this release, once you click on the installer.exe file, the new UI will be popped up, select the first option to Install the VCSA.

This brings up the Stage 1 of the installation which is deploying the appliance.

Next, you will have to accept the EULA and click Next to continue.

Select the Deployment, I am going to select the Embedded installation where the Platform Services Controller and the vCenter Server are installed on the same machine.

You will now have to provide the details of an existing vCenter Server or an ESXi host on which you want to deploy the new VCSA 6.5 appliance.

I am providing the details of an ESXi host that I have in my test lab.

Accept the Certificate Warning as I haven’t made any changes to default certificates of the host.

Provide a name for the Virtual Machine and set up the root password which will be used to configure other settings.

Select the Deployment size depending upon your needs, I will select the Tiny size deployment.

Select Destination datastore on which you want to deploy the appliance.

In the next step, you will have to configure the Network Settings like the IP Address, Subnet Mask, Default Gateway, DNS Servers and the System Name for the appliance.

Review the details that you have provided and click Finish to kick off the Deployment of the VCSA 6.5 appliance and finish the Stage 1 of the installation.

Once the deployment is completed, you would see the message and this completes the Stage 1 of the VCSA installation.

From here on, you can either click to continue to start the Stage 2 or close and connect to the VAMI (port 5480) to start stage 2 at a later stage.

But I would recommend on continuing to stage 2.

Stage 2: Configuring the VCSA 6.5

Stage 2 starts with the below screen giving you an overview of what needs to be configured for an Embedded type of installation.

You get two options to synchronize the appliance, either provide the details of an NTP server or choose to synchronize the time with the host on which the appliance is running.

Best practice is to provide an NTP server information used in your environment.

I have also enabled the SSH access to the appliance for a future use case. I do not recommend this but since this is my lab, I am OK with this.

Now, you will need to provide the details of the SSO configuration.

You can wish to join the VMware Customer Experience Program, it helps us with feedback.

Check the Review and click Finish to start configuring the VCSA appliance.

Once the appliance finishes the configuration, you would receive a message which says that everything is configured and now the vCenter can be accessed using the Web Client and new HTML 5 Client.

I hope this has been informative and thank you for reading!

Windows Server 2008 Repair Steps for No Boot Issues

Cases

Most of the no boot cases are caused by either of the following reasons:   

• Disk metadata corruption in the master boot record (MBR), partition table, or boot sector of Windows RE partition.
• Missing or corrupt boot manager.
• Missing or corrupt boot configuration data.

Startup Repair Process

Startup Repair will try to repair computers that are unbootable because of the following reasons:

• Registry corruption
• Missing or damaged system and driver files
• Disk metadata corruption (MBR, partition table, and boot sector)
• File system metadata corruption
• Installation of problematic or incompatible drivers
• Installation of incompatible Windows service packs and patches
• Corrupt boot configuration data
• Bad memory and hard disk hardware (detection only)

 http://technet.microsoft.com/en-us/library/cc722188(WS.10).aspx  
How Windows RE Works

 Log file location: Startup Repair Log:After Startup Repair has run, a text log with diagnostic information and repair result is generated within the recovery console. This log file is located at %WINDIR%\System32\LogFiles\Srt\SrtTrail.txt

•  We can execute startrep.exe tool in recovery console to address corrupt registry issues if any in Windows Server 2008 R2

 If the Windows registry is slightly or moderately corrupted, you may be able to restart the computer in Safe mode and use System Restore to restore the registry of the computer to the last known good configuration. However, if the Windows registry is severely corrupted, all types of logon will be prevented. Attempting to log on to Windows causes the system to fail and then to restart. In this situation, you will need to boot the system into the Recovery Console instead of into Windows. Once in the Recovery Console, you can use the Startup Repair tool. Startup Repair automates common diagnostic and repair tasks of unbootable Windows installations.

a)       Boot from your Windows Vista or Windows Server 2008 DVD b)       On the first screen of Setup choose Next c)       In the lower left of the screen choose "repair your computer" d)       On the System Recovery Options screen choose your installation of Windows and then click next e)       Click "Command prompt" f)        Go to X:\sources\recovery> startrep.exe g)       This will take a while but will address any registry related issues.

 Ref: http://technet.microsoft.com/en-us/library/cc734097(WS.10).aspx  
Event ID 4005 — Windows Logon Availability

• Sfc Scannow command address file based corruptions if any.

a)       Boot from your Windows Vista or Windows Server 2008 DVD b)       On the first screen of Setup choose Next c)       In the lower left of the screen choose "repair your computer" d)       On the System Recovery Options screen choose your installation of Windows and then click next e)       Click "Command prompt" f)        Type in the following command: SFC.EXE /scannow /offbootdir=c:\ /offwindir=c:\windows g)       When complete type in exit to see if Windows starts up now.

 Ref: http://blogs.technet.com/b/askcore/archive/2007/12/18/using-system-file-checker-sfc-to-fix-issues.aspx  
Using System File Checker (SFC) To Fix Issues

• Chkdsk to address disk and filesystem based issues.

Boot the machine in recovery console mode and run chkdsk on C: drive [Local Fixed Disk]chkdsk C: /f and then chkdsk c: /r 
/f - Fixes errors on the disk. The disk must be locked. If chkdsk cannot lock the drive, a message appears that asks you if you want to check the drive the next time you restart the computer.
/r - Locates bad sectors and recovers readable information. The disk must be locked. /r includes the functionality of /f, with the additional analysis of physical disk errors.

 http://technet.microsoft.com/en-us/library/cc730714(WS.10).aspx  
Chkdsk

vCenter Server for Windows and vCSA compared

The most glaring difference, as the name implies, is that vCenter Server for Windows will only run on a Microsoft Server operating system.. On the other hand, vCSA is a SuSe Linux Enterprise Server 64-bit based appliance which comes lock, stock and barrel ready for importation.

The second important difference is related to vSphere Update Manager (VUM) which, as you probably know, can be installed alongside on the same server running vCenter Server for Windows. Unfortunately the same does not apply to vCSA meaning that you’d still be needing a Windows box if your plans include deploying VUM. My prediction is that vCSA – VUM integration will be made available a couple of updates down the line. Failing that, it will be addressed in the next major release.

Thirdly, your database options are limited to the following:

• vCenter Server for Windows – you can either install the bundled PostgreSQL database for environments with no more than 20 hosts and 200 virtual machines or in the case of larger environments, use a Microsoft SQL or Oracle DBMS.
• vCSA – The bundled PostgreSQL database which now supports environments of up to 1000 hosts and 10,000 virtual machines. This will probably suffice for most scenarios but if need be, you can still use an external Oracle DBMS.

Similarities between vCenter Server for Windows and vCSA

• The hardware requirements are identical to both. Figure 1, reproduced from VMware’s documentation, specifies the minimum CPU and Memory requirements according to the environment size.

Figure 1 – vCenter Server Minimum Hardware Requirements

• Both can be managed with the traditional vSphere or web based client. In the case of vCSA you can also SSH to it or use the Direct Console User Interface to change stuff such as the root password and network settings (pretty much like you would do on an ESXi host). With the release of vCenter Server 6.0 Update 1 it is now possible to administer vCSA using the newly released HTML 5 based Appliance Management User Interface. One great feature is the ability to patch / upgrade vCSA using an ISO image or URL based patching.

Figure 2 – Appliance Management User Interface

• At last, and I emphasize this as in the past vCSA presented some serious scalability limitations preventing a more thorough uptake, both flavors enjoy the same set of scalability metrics. These are listed by way of Figure 3 reproduced from the “What’s New in the VMware vSphere 6.0 Platform” whitepaper.

Figure 3 – vCenter Server Scalability Numbers

Weighing advantages and disadvantages

vCSA Pros

• Quicker to deploy since everything is preconfigured and you don’t need to install anything apart from running the actual installer, supplying a couple of details and sipping tea as you sit back watching it do its thing.
• The bundled database is scalable enough to cater for most environments unless of course you’re planning on deploying 1000+ hosts and/or 10000+ vms.
• You do not need separate licenses for Microsoft Windows Server so overall this might turn out to be a cheaper option.
• From a security perspective, vCSA traditionally presents a smaller attack surface since it runs on Linux though it should be said that Microsoft have upped their ante considerably in this department.

vCSA Cons

• As with many appliances vCSA is somewhat of a black-box. You’ll be needing some intermediate Linux skills when troubleshooting or VMware support when things go dark.
• You still need to have a separate Windows box on which to install and run VUM.

vCenter Server for Windows Pros

• In my opinion it’s easier to troubleshoot vCenter Server for Windows if anything due to a larger user-base and the fact that fixing Windows related issues is generally easier, from my experience at least.
• VUM can be happily installed on the same server running vCenter so you don’t need to worry about managing two servers.

vCenter Server for Windows Cons

• For larger environments, you’ll definitely be needing an external database meaning more license and hardware related costs.
• While deployment is still easy, it definitely takes longer to install and configure vCenter Server for Windows.

Conclusion

The bottom line is that VMware is seemingly moving towards relegating vCenter Server for Windows to history. This is evident with the features that come with version 6 and the recently released Update 1. It’s only a matter of time before VMware completely close any functionality and feature gaps that remain between the two flavors. If they can address the VUM co-existence issue, I’m pretty sure more people would opt for vCSA without as much as giving it a second thought.

Storage performance troubleshooting with ESXTOP [Guide]

As you know ESXTOP is an utility bundled with ESXi allowing to monitor/troubleshoot performance of network, CPU or storage. This post is about storage performance troubleshooting with ESXTOP.

Se will focus on storage as many times the storage is the main problem of latency. A weakest performance element in the whole chain. VMs performs slowly, but where the latency comes from? Is it at the VM level, LUN level or Disk level (hba).

While ESXTOP is command line utility, note that there is a nice free tool which has GUI, from VMware called Visual ESXTOP. It integrates into vSphere web client. Note that I have not personaly tested the tool with the latest vSphere 6.0 U1 release.

What to monitor/troubleshoot?
Per LUN
Per VM
per Disk (HBA mode)
Let’s monitor a LUN with ESXTOP
1. Start ESXTOP and press U to switch to disk view (LUN mode).




2. Press F (Field Order) to modify fields which you want to display. Then hit Enter to validate.


3. Press S and then 3 (or other smaller/bigger value) to set the auto-update time to every 3 seconds…. In order to view the whole device name (the complete naa identifier) you’ll have to enlarge the column pres Shift + L and enter “32”. (or other larger number).


Let’s try to monitor Disk View (hba mode).
1. Start ESXTOP utility and press D to switch to Disk view (hba mode). In order to view the whole device name (the complete naa identifier) you’ll have to enlarge the column pres Shift + L and enter “32”. (or other larger number).


2. From here you can hit F (Field Order) to modify fields which you want to display. (You can see small star diplayed next to each visible field….). When OK, you can hit Enter.


3. Press S and then 3 to set the auto-update time for 3 sec. (you can enter smaller/bigger value as you wish).


Monitor VM performance (Per VM)
1. Start ESXTOP and press  V  to switch to disk view.


2. Again, Press F (Field Order) to modify fields which you want to display. Then hit Enter to validate.

3. And again, press S and then 3 (or other smaller/bigger value) to set the auto-update time to every 3 seconds…. In order to view the whole device name (the complete naa identifier) you’ll have to enlarge the column pres Shift + L and enter “32”. (or other larger number).


What represents the different columns?
Now let’s start with ESXTOP utility by identifying the different columns.

CMDS/s –  sum of commands per second with IOPS (Input/Output Operations Per Second). Here are also other SCSI commands like SCSI reservations, locks, vendor string requests, unit attention commands etc. All those are flowing to or are coming from the device or virtual machine which is monitored.

DAVG/cmd  –  Average response time in milliseconds per command which is sent to the device.

KAVG/cmd –  How many time this command spend in the VMkernel.

GAVG/cmd  – Response time at the guest operating system level. Here comes a formula: DAVG + KAVG = GAVG
[10/9, 16:45] vExpert-Jayesh: CPU

When troubleshooting CPU performance for your virtual machines the following counters are the most important.

%USED, %RDY, %CSTP

%USED tells you how much time did the virtual machine spend executing CPU cycles on the physical CPU.

%RDY is a Key Performance Indicator! Always start with this one. This one defines how much time your virtual machine wanted to execute CPU cycles but could not get access to the physical CPU. It tells you how much time did you spend in a “queue”. I normally expect this value to be better than 5% (this equals 1000ms in the vCenter Performance Graphs read about it here)

%CSTP tells you how much time a virtual machine is waiting for a virtual machine with multiple vCPU to catch up. If this number is higher than 3% you should consider lowering the amount of vCPU in your virtual machine.
[10/9, 16:45] vExpert-Jayesh: Memory

When troubleshooting memory performance this is the counters you want to focus on from a virtual machine perspective.

MCTL?, MCTLSZ, SWCUR, SWR/s, SWW/s

MCTL? This column is either YES or NO. If Yes it means that the balloon driver is installed. The Balloon driver is automatically installed with VMware tools and should be in every virtual machine. If it says No in this column then figure out why.

MCTLSZ The column show you how inflated the balloon is in the virtual machine. If it says 500MB it translates to the balloon driver inside the guest operating system has “stolen” 500MB from Windows/Linux etc. You would expect to see a value of 0 (zero) in this column

SWCUR tells you how much memory the virtual machine has in the .vswp file.  If you see a number of 500MB here it means that 500MB is from the swap file. This does not necessarily equals to bad performance. To figure out if you virtual machine is suffering from hypervisor swapping you need to look at the next two counters. In a healthy environment you would want this value to på 0 (zero)

SWR/s This value tells you the Read activity to your swap file. If you see a number here, then your virtual machine is suffering from hypervisor swapping.

SWW/s This value tells you the Write activity to your swap file. You want to see the number 0 (zero) here. Every number above 0 is BAD.