LINternUX

As a system administrator, I have a variety of performance tools that I use to identify problem related to system performance. In this article, I have tried to demonstrate the usage of the “vmstat” command and also explain the meaning of each column that it produce. I inserted some tips regarding the interpretation of theses columns, I wanted to make this article a reference for me and hopefully for you too.  So the next time you use the command ” vmstat”,  you can come back at this article and analyze the output of your problematic system.

First let’s look at different way we can invoke the “vmstat” command.  To monitor the virtual memory activity on your system,  it is preferable to use “vmstat” with a delay. A delay is a number of seconds between updates. If you don’t supply a delay “vmstat” reports the average values since the last reboot and quit.  With the delay, you can also specify a count, which indicate how many updates you want to see before “vmstat” quits. If you don’t specify a count, the count defaults is infinity and you will have to press CTRL-C to stop “vmstat”.  In our example, we asked  “vmstat” to run with four updates with a delay of five seconds between update.

vmstat figure 1

procs section
The procs section indicate the number of running processes (r) and blocked (b) processes at the time of the reporting. The scheduler determines when processes are run. Each process is assigned a priority and a slot in the process table. Processes must be present in memory before they are run. If a process is scheduled to run but a memory page for part of that process is not in real memory, that process is blocked and placed in the run queue. Processes ready to run are placed in the run queue, “vmstat” reports on processes that are in the run queue and those that are blocked.

• (r) – Represent the number of runnable processes in the run queue. The value should be a sigle-digit on a healthy server, but can occasionally get higher. From my experience when it reach number greater than 20 on some big and busy system you should have a look on the system and see what process is taking most of the cpu (use the top command).
• (b) – This value represent the number of processes scheduled to be executed but blocked, waiting for the virtual memory to page the part of that process which is on disk into real memory. This number should also be a single digit on a healthy system.

memory section
The memory section report the amount of memory swapped out (swpd), free memory (free), buffer cache for i/o data structure (buff) and cache memory for file read
from disk (cache) in kilobytes.

Swapping, strictly speaking, refers to writing portions of memory (4K page) to disk.  A page fault occurs when the virtual memory manager attempts to access a memory page that is not in memory.  Real memory that is not used is place in the free list. The virtual memory manager is responsible for maintaining the free list.

• (swpd) – Represent the amount of virtual memory used (Kb). In our example we are using 1.48Gb (1176140/1024=1148Mb) of our swap space.
• (free) -  This represent the amount of real memory that is not used at the moment. In our example,  we have 943Mb (966,124/1024=943Mb) of unused memory.
• (buff) – The “buffers” entry indicate how much of the memory in use is currently being used for disk buffering. In our example, we have 76MB (78444/1024=76Mb) of memory allocated for disks buffering.
• (cache) -  When extra physical memory is not in use, the kernel attempts to put it to work as a disk buffer cache. The disk buffer stores recently accessed disk data in memory; if the same data is needed again it can be quickly retrieved from the cache, improving performance. The buffer grows and shrinks dynamically to use the memory available, although priority is given to using the memory for paging. Thus, all the memory you have is put to good use.  In our example above we have 2.1Gb (2211108/1024=2159Mb) used for disks caching.

swap section
The “swap” section indicate the amount of memory swapped in (si) from disk and swapped out (so) to disk, in kilobytes per second.  All processes use memory, but all processes doesn’t need all of it memory at all time,  Linux take advantage of this fact. The kernel frees up some physical memory by writing some or all of a process memory to disk until it is needed again. When pages are written to disk, the event is called a page-out, when pages returned to physical memory, this event is called a page-in.  Using swap space is not inherently bad, but intense paging activity can become problematic. When the kernel reach a point where it’s actually spending more time managing activity than running the applications, you will see a system performance degradation, this is when adding more memory to the system have to be considered.

• (si) -  Amount of memory swapped in from disk (/s).  Swap-in/Page-in are common, normal and may not cause concern. Example, when an application first starts up, its executable image and data are page-in, this a normal process.
• (so) – Amount of memory swapped in from disk (/s).  Swapping-out/Paging-out however, could be a sign of trouble. When the kernel detects that memory is running low, it attempts to free up memory by paging out. This may happens for a short period of time, but if see that it is becoming constants, then this may indicate that you are short on memory on the system. This is also causing a performance decrease on the server, because the system is spending a lot of it time writing (page-out) and reading (page-in) memory pages to disk.  The time that the kernel is doing the disk I/O  needed to perform page-out/page-in is not given to the user process. This is an important column to look at.

io section
Return the number of blocks read in (bi) from the devices and blocks written out (bo) to the devices in kilobytes per second. Pay particular attention to these two fields when running i/o intensive applications. The statistics shown here, are mostly disk I/O, but it can be any block devices on the systems (Optical disk, Tapes).

• (bi) -  Blocks received from a block device (blocks/s).
• (bo) -  Blocks sent to a block device (blocks/s).

system section
Report the number of interrupts (in) and context switches (cs) per second.

• (in) – The number of interrupts per second, including the clock.
• (cs) – The number of context switches per second. Cpu time is given to each process in time slices. When a process has used up its time slice, it must relinquish the cpu to another process. The cpu must save the working environment for the next process to be executed. This is known as context switch.

cpu section
Report the percentage of total CPU time in terms of user (us), system (sy), true idleness (id), and waiting for i/o completion (wa). Perhaps the most common metric is CPU utilization. If “wa” is large, examine the i/o subsystem – you may conclude, for example, that more i/o controller and disks are needed to reduce the i/o wait time, you may want to split the I/O on multiple disks. If swapping is occurring a lot, then you may want to add memory to reduce that i/o wait percentage. If the idle time (id) is consistently 0 and the system time (sy) is double the user time (us), the system is facing a shortage of cpu resource.

• (us) – Percent of time the cpu is operating in user mode. When is user mode, a process executes in its application code and does not require kernel resources.
• (sy) – Percent of the time the cpu is operating in kernel mode.. When is system mode, the process is making calls for accessing the kernel.
• (id) – Percent of time the cpu is idle not processes available for execution and no pending i/o.
• (wa) – Percent of time the cpu is idle with no processes available for execution but with pending i/o requests.

Conclusion

It isn’t necessarily bad for system to use some of its swap page, but if you realize that your system performance is decreasing, running low on memory and that there is a lot of paging going on, adding more memory can be a solution. With the column “si” and “so” you can even have an idea on how much memory is needed to reduce or eliminate the paging.

I hope that you appreciated this article and that we will come back to the site.  I have many more interresting article in mind, so subscribe to my RSS feed and hope to see you come back soon .