VMware – ESXi scratch partition and logging

Scratch partition

When you install ESXi, a scratch partition is automatically created when local disks are used and at least 5GB of free diskspace is available for ESXi to install on. This “scratch partition” is used for storing

  1. vm-support dumps
  2. log files
  3. userworld swapfiles (if enabled).

When using Boot-from-SAN or USB scratch partitions are not automatically created and the above mentioned files will be stored in a RAM disk. This means that after a reboot these files will be gone. Therefore, from an operational point of view it is a good idea to manually specify or script the scratch partition after the installation. You could use a VMFS volume for it and create multiple folders per server to store the respective files. Now, although ESXi requires 4GB for the scratch partition on local disk this doesn’t mean it is a hard requirement to allocate 4GB per server on a shared volume. I recommend creating a single shared volume for all servers and to allocate roughly 20GB to this volume. Closely monitor this volume to avoid filling it up with log files. Configuring the scratch partition is straightforward and can be done in eight simple steps:

  1. In the vSphere Client inventory, left-click the host.
  2. Click the Configuration tab.
  3. Click Advanced Settings under Software.
  4. Select ScratchConfig in the tree control.
  5. In the ScratchConfig.ConfiguredScrathLocation text box, enter the directory to be used for the scratch partition. (this needs to be unique for every ESX host)
  6. Enable the ScratchConfig.ConfiguredSwapState (recommended as this is a requirement for HA: see KB 1004177)
  7. Click OK.
  8. Put the ESXi host in maintenance mode and reboot the host

 

Logging

Logging is important for both troubleshooting and compliance. VMware ESXi exposes logs by using a host syslog capability from

  1. the host agent (hostd)
  2. VMware vCenter agent (vpxa)
  3. VMkernel (messages)

In many environments today Syslog is neglected but especially in ESXi environments there is real value in configuring this. Syslog will allow you to aggregate all logs into a single location. This will simplify the troubleshooting process immense when needed. Especially when Boot-from-SAN or USB is used as a boot media this setting can be critical as a scratch disk is not created by default. You can imagine situations where customers would run into problems which were impossible to troubleshoot as the scratch partition was not defined and neither was syslog, resulting in new log files after each reboot.

Users can configure syslog to write logs in different ways:

  1. configure the syslog to the scratch partition of the host (default configuration in ESXi 4.1). Be aware of the fact that the logs are stored in memory and they do not persist reboots unless they are redirected to a log host/file
  2. configure the syslog to a file on any datastore accessible to the VMware ESXi host.
  3. configure the syslog to a syslog server for enterprise central logging ==> recommended

It is a best practice to leverage the syslog capabilities that ESXi 4.1 offers. Using a syslog server will simplify troubleshooting and ensure that log files are accessible even when a VMware ESXi host has physically failed. Many syslog servers also enable easier correlation between events. Configuring the syslog client is straightforward and can be done in seven simple steps:

  1. In the vSphere Client inventory, left-click the host.
  2. Click the Configuration tab.
  3. Click Advanced Settings under Software.
  4. Select Syslog in the tree control.
  5. In the Syslog.Remote.Hostname text box, enter the name of the remote host where syslog data will be forwarded. If no value is specified, no data is forwarded.
  6. In the Syslog.Remote.Port text box, enter the port on the remote host where syslog data will be forwarded. By default Syslog.Remote.Port is set to 514, the default UDP port used by syslog. Changes to Syslog.Remote.Port take effect only if Syslog.Remote.Hostname is configured.
  7. Click OK.

 

A second capability that VMware ESXi offers is specifying a location for local log files. Local refers to a non-syslog solution. Local log files don’t need to be stored on a local drive, but rather on any datastore. It is a best practice for environments without a syslog server to specify a remote VMFS datastore, to ensure that log files will be available when a VMware ESXi host has physically failed to allow for a root cause analysis. This can be configured through the vSphere Client as follows:

  1. In the vSphere Client inventory, left-click the host.
  2. Click the Configuration tab.
  3. Click Advanced Settings under Software.
  4. Select Syslog in the tree control.
  5. In the Syslog.Local.DatastorePath text box, enter the datastore path to the file where syslog will log messages. If no path is specified, the default path is /var/log/messages. In addition, if pointing at a datastore, ensure that the directory has been created previously.

The datastore path format is /vmfs/volumes/<datastore>/<folder>/filename

Syslog Server

There are several choices available to use as a syslog server. The two most popular ones seem to be

Both come with a choice of a free or a paying version.

For vSphere 5.0 see http://www.boche.net/blog/index.php/2011/07/23/configure-a-vcenter-5-0-integrated-syslog-server/