Do you need to monitor your Linux server’s performance? Most Linux distributions come equipped with many built-in monitoring tools. These tools allow you to retrieve information about system activities, and can be used to find possible causes for your server’s performance issues.
The commands discussed in this article are some of the most basic commands when it comes to system analysis and debugging server issues, such as discovering disk, CPU, memory and network bottlenecks.
1 - top – The Process Activity Command
The top command provides a dynamic, real-time view of the running system (i.e. actual process activity). By default, it displays the most CPU-intensive tasks running on the server and updates the list every five seconds.
Commonly Used Hot Keys
Most Linux distributions come equipped with many built-in monitoring tools.
There are several useful hot keys used with the top command:
-
t — toggles summary information off and on.
-
m — toggles memory information off and on.
-
A — sorts the display by top consumers of various system resources. This is useful for quickly identifying performance-hungry tasks.
-
f — enters an interactive configuration screen for top. It’s helpful for configuring top for a specific task.
-
o — enables you to interactively select the order of the displayed fields.
-
r — issues the renice command.
-
k — issues the kill command.
-
z — toggles between color and monochrome.
2 - vmstat – System Activity, Hardware and System Information
The vmstat command reports virtual memory statistics: processes, memory, paging, block IO, traps and cpu activity.
Syntax:
# vmstat 3
Sample output:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 2540988 522188 5130400 0 0 2 32 4 2 4 1 96 0 0
1 0 0 2540988 522188 5130400 0 0 0 720 1199 665 1 0 99 0 0
0 0 0 2540956 522188 5130400 0 0 0 0 1151 1569 4 1 95 0 0
0 0 0 2540956 522188 5130500 0 0 0 6 1117 439 1 0 99 0 0
0 0 0 2540940 522188 5130512 0 0 0 536 1189 932 1 0 98 0 0
0 0 0 2538444 522188 5130588 0 0 0 0 1187 1417 4 1 96 0 0
0 0 0 2490060 522188 5130640 0 0 0 18 1253 1123 5 1 94 0 0
Display memory utilization slabinfo with the following command:
# vmstat -m
And you can retrieve information about active and inactive memory pages with:
# vmstat -a
3 - w – Logged In Users
The w command displays information about the currently logged in users and their processes. Its syntax is:
# w [user]
Running this command gives you output similar to the following:
17:58:47 up 5 days, 20:28, 2 users, load average: 0.36, 0.26, 0.24
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root pts/0 10.1.3.145 14:55 5.00s 0.04s 0.02s vim /etc/resolv.conf
root pts/1 10.1.3.145 17:43 0.00s 0.03s 0.00s w
4 - uptime – System Uptime
The uptime command not only displays the amount of time the server has been running, but the current time, how many users are currently logged on and the system load average for the past 1, 5, and 15 minutes.
# uptime
Its output will look similar to:
11:57:40 up 96 days, 17:23, 0 users, load average: 1.17, 1.15, 1.14
5 - ps – The Processes
The ps command reports a snapshot of the current processes. To select all processes, use the -A or -e option.
# ps -A
Running this command on your system will provide similar results to the following:
PID TTY TIME CMD
6538 ? 00:00:00 anytermd
6543 pts/4 00:00:00 bash
6855 pts/4 00:00:00 ps
To show long format output, add “l” at the end:
# ps -Al
Turn on extra-full mode to show the command line arguments passed to the processes by adding “F”:
# ps -AlF
Add an “H” to see threads (LWP and NLWP):
# ps -AlFH
To see threads after processes, add “m”:
# ps -AlLm
Or print a process tree:
# ps -ejH
# ps axjf
# pstree
Display only the process IDs of lighttpd:
# ps -C lighttpd -o pid=
Or:
# pgrep lighttpd
You can also find the top ten processes using the most CPU with:
# ps -auxf | sort -nr -k 3 | head -10
6 - free – Memory Usage
The free command displays the total amount of physical and swap memory in the system, as well as the buffers used by the kernel.
# free
This simple command’s output looks like:
total used free shared buffers cached
Mem: 12302896 9739664 2563232 0 523124 5154740
-/+ buffers/cache: 4061800 8241096
Swap: 1052248 0 1052248
7 - iostat – Average CPU Load and Disk Activity
The iostat command reports CPU and input/output statistics for devices, partitions and network filesystems (NFS).
# iostat
And the obligatory sample output:
Linux 3.4.5-hardened-v3 (tryit) 12/14/12 _i686_ (1 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
2.47 6.52 2.54 1.57 0.00 86.90
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
8 - sar – Collect and Report System Activity
The sar command collects, reports and saves system activity information. To see the network counter, enter:
# sar -n DEV | more
To display the network counters from the 24th:
# sar -n DEV -f /var/log/sa/sa24 | more
You can also display real time usage using sar:
# sar 4 5
Sample output:
Linux 3.4.5-hardened-v3 (tryit) 12/14/12 _i686_ (1 CPU)
12:45:25 CPU %user %nice %system %iowait %steal %idle
12:45:29 all 0.43 19.70 1.07 0.00 0.00 78.80
12:45:33 all 0.43 19.35 1.51 1.51 0.00 77.20
12:45:37 all 0.43 18.49 1.94 0.00 0.00 79.14
12:45:41 all 0.43 19.02 2.14 0.00 0.00 78.42
12:45:45 all 0.65 18.49 2.37 0.00 0.00 78.49
Average: all 0.47 19.01 1.80 0.30 0.00 78.41
9 - mpstat – Multi-processor Usage
The mpstat command displays each available processor’s activities, with processor 0 being the first. Use the following command to display the average CPU utilization per processor:
# mpstat -P ALL
And this is the output:
Linux 3.4.5-hardened-v3 (tryit) 12/14/12 _i686_ (1 CPU)
12:47:46 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
12:47:46 all 2.47 6.52 1.48 1.57 0.00 1.06 0.00 0.00 86.90
12:47:46 0 2.47 6.52 1.48 1.57 0.00 1.06 0.00 0.00 86.90
10 - pmap – Process Memory Usage
The pmap command reports the memory map of a process. Use this command to find the causes of memory bottlenecks.
# pmap -d PID
To display process memory information for pid # 47394, enter:
# pmap -d 47394
Sample output:
47394: /usr/bin/php-cgi
Address Kbytes Mode Offset Device Mapping
0000000000400000 2584 r-x-- 0000000000000000 008:00002 php-cgi
0000000000886000 140 rw--- 0000000000286000 008:00002 php-cgi
00000000008a9000 52 rw--- 00000000008a9000 000:00000 [ anon ]
0000000000aa8000 76 rw--- 00000000002a8000 008:00002 php-cgi
000000000f678000 1980 rw--- 000000000f678000 000:00000 [ anon ]
000000314a600000 112 r-x-- 0000000000000000 008:00002 ld-2.5.so
000000314a81b000 4 r---- 000000000001b000 008:00002 ld-2.5.so
000000314a81c000 4 rw--- 000000000001c000 008:00002 ld-2.5.so
000000314aa00000 1328 r-x-- 0000000000000000 008:00002 libc-2.5.so
000000314ab4c000 2048 ----- 000000000014c000 008:00002 libc-2.5.so
.....
......
..
00002af8d48fd000 4 rw--- 0000000000006000 008:00002 xsl.so
00002af8d490c000 40 r-x-- 0000000000000000 008:00002 libnss_files-2.5.so
00002af8d4916000 2044 ----- 000000000000a000 008:00002 libnss_files-2.5.so
00002af8d4b15000 4 r---- 0000000000009000 008:00002 libnss_files-2.5.so
00002af8d4b16000 4 rw--- 000000000000a000 008:00002 libnss_files-2.5.so
00002af8d4b17000 768000 rw-s- 0000000000000000 000:00009 zero (deleted)
00007fffc95fe000 84 rw--- 00007ffffffea000 000:00000 [ stack ]
ffffffffff600000 8192 ----- 0000000000000000 000:00000 [ anon ]
mapped: 933712K writeable/private: 4304K shared: 768000K
The last line is very important:
mapped: 933712K total amount of memory mapped to files
writeable/private: 4304K the amount of private address space
shared: 768000K the amount of address space this process is sharing with others
11 - netstat – Network Statistics
The netstat command displays both incoming and outgoing network connections, routing tables and a number of network interface statistics. It is available on Unix, Unix-like systems, and Windows NT-based operating systems.
# netstat
12 - iptraf – Real-time Network Statistics
The iptraf command is an colorful, interactive IP LAN monitor. It is an ncurses-based IP LAN monitor that generates various network statistics including TCP info, UDP counts, ICMP and OSPF information, Ethernet load info, node stats, IP checksum errors and more. It can provide the following info in easy to read format:
- Network traffic statistics by TCP connection.
- IP traffic statistics by network interface.
- Network traffic statistics by protocol.
- Network traffic statistics by TCP/UDP port and packet size.
- Network traffic statistics by Layer2 address.
13 - tcpdump – Detailed Network Traffic Analysis
The tcpdump is a simple command that dumps a network’s traffic. You need, however, a good understanding of the TCP/IP protocol in order to use this tool. For example, to display traffic info about DNS, enter:
# tcpdump -i eth1 'udp port 53'
To display all IPv4 HTTP packets from port 80 (i.e. print only packets that contain data; not, for example, SYN abd FIN packets and ACK-only packets), enter:
# tcpdump 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
To display all HTTP sessions to 192.168.1.5:
# tcpdump -ni eth0 'dst 192.168.1.5 and tcp and port http'
14 - strace – System Calls
The strace command traces system calls and signals. This is useful for debugging the webserver and other server problems.
The following command runs strace against /bin/foo and captures its output to output.txt:
# strace -o output.txt /bin/foo
15 - /proc – Various Kernel Statistics
The /proc file system provides detailed information about various hardware devices and other Linux kernel information. Common /proc examples are:
# cat /proc/cpuinfo
# cat /proc/meminfo
# cat /proc/zoneinfo
# cat /proc/mounts
Conclusion
And there you have it: fifteen useful commands that let you monitor different aspects of your Linux system. Naturally, these commands are only a small subset of the many that Linux provides. But for day to day operations, they are usually enough.
Have a favorite command? Let’s keep the conversation going within the comments area.
