webpointmorpheus Linux Info
Performance and Problems

     Link3      Link4      Link5      Notes      Documents in this Series

©2005 - material compiled by Bob Carnaghi, www.webpointmorpheus.com

Redhat Linux Documentation

talk to me bob

System Troubleshooting

This section lists items that are common problem areas of the Linux Operating System. The problem areas are listed, and further defined with common corrective measures. Note the schematic breakdown of the topics of this section, as listed in the list below. For additional assistance, check the man pages, online documentation, FAQ's, etc. that pertain to the specific problem.

1. System Monitoring
2. Proactive Maintenance
3. Reactive Maintenance
4. • Hardware Problems
   • Software Problems
   • Application Problems
   • Operating System Problems
5. Documentation

System Monitoring

This area is a series of vigilant steps that are performed by a system administrator which include examining or reviewing log files and running performance utilities. These practices help establish the presence of potential problems before they occur and create reduced system productivity or failure.

Definitions

1. baseline - a series or collection of statistice taken from a system during a no- or light-load situation under normal circumstances that will serve as a reference of comparision for the performance of the system during various production loads and situations.
2. bus mastering - the practice of including a mini-processor on peripheral hardware that relieves the CPU of a certain amount of processing load.
3. jabbering - a syndrome where an aged and malfunctioning piece of hardware sends large amounts of unnecessary information to the CPU thereby slowing system performance.
4. page file - a hard drive or partitioned section of a hard drive that serves as a temporary storage space for data when the system RAM becomes filled or overtaxed.
5. paging - the process where the CPU writes data to the page file instead of to the system RAM.

Best General Practices

Listed below are several general practices that should be considered when initializing hardware of an overall Linux system. These considerations are not exhaustive, and should be considered in tandem with overall good system configuration practices.

1. Abundant system RAM - decreases the need for swap file use, improves general system speed and response.
2. Replace slower hard drives, use disk striping RAID for faster hard drive access time.
3. SCSI is generally faster than IDE./li>
4. CD-ROM is slower than hard drive - don't place them both on the same IDE controller
5. Keep and maintain the kernel trim and lean - eliminate unnecessary modules or recompile a lighter version.
6. Establish baseline system statistics for reference during troubleshooting.
7. Watch and become familiar with the contents of the /proc directory.

Commands and Utilities

System performance utilities are listed below. Note that the flags provided to the commands can alter the return, sometimes dramatically.

1. sysstat - System Statistics package that will monitor the system vrom the /proc directory and system devices.
2. mpstat - Multiple Processor Statistics utility. See below for return codes. No flags will give average values since last boot. Use flags (such as mpstat 1 10) to establish or alter the return values for interval sampling (as given, 1 second interval, 10 times.)
3. iostat - Input/Output Statistics. Shows information for data written to disk.
4. vmstat - Reports virtual memory statistics.
5. sar - System Activity Reporter. An enhanced utility that combines many of the features of the *stat package, and writes them to log files. Check the /var/log/sa, and the /var/log/sa/* directories. sar has a full array of useful flags and options, see the man pages for details.
6. free - Displays memory and its utilization in KB for both RAM and swap.
7. top - Displays a full array of process statistics dynamically.
8. ps - Gives a shapshot of the current processes on the system.

*stat headers
Header	Definition
%user	Refers to user initiated programs and daemons.*
%nice	Refers to processes and daemons initiated with nondefault values.*
%system	Refers to the amount of time the system spent maintaining itself such that it could execute user programs and daemons.*
%iowait	Refers to the amount of time the CPU stood idle while there existed outstanding I/O requests.**
%irq	Refers to the percentage of time the CPU uses to respond to normal interrupts.**
%soft	Refers to the percentage of time the CPU uses to respond to interrupts that span multiple CPUs.**
%idle	Refers to the amount of time the CPU was not processing requests.***
intr/s	The number of interrupts per second received by the CPU from peripheral devices.****
Notes: *Watch for a high %system value compared to the %user and %nice values. This condition would indicate too many resource-intensive programs. **Watch the values for %iowait, %irq, and %soft. A rapid increase over time indicates a situation such that the CPU is not keeping pace with the number of requests sent from software. ***%idle should be on average greater than 25%. Lesser values in short bursts are acceptable, but sustained values less than 25% indicate a need for faster or additional CPUs. ****The production value is best compared to a baseline value. When this comparision shows excessive returns, a jabbering hardware condition may exist.

Proactive Maintenance

Proactive system maintenance is defined as those measures taken to identify, reduce, or eliminate problems before they reduce system performance. System documentation is best kept separate from the files which constitute the system itself in case of filesystem or hard disk failure.

1. System backups
2. Identify and isolate potential problems

Reactive Maintenance

examine log files, examine /proc file system, run information utilities (ps or mount)

NOTE: the tail -f 'name-of-log-file' will give an updated version of the log file as items are written. Use this method for immediate troubleshooting of an application or service in progress.

Tips to identify and correct problems

1. Prioritize problems
2. Find and solve the root cause of the problem - know the system hardware and configuration, what is the empirical cause of a problem?

Resolving System Problems

Problems will typically fall into one of two broad categories. The categories will further subdivide, but identifying the first general category will channel corrective measures in the right direction.

1. Hardware Problems
2. Software Problems
3. • Application Software
   • Operating System Software

Hardware Type Problems

paragraph

1. damaged, defective, or decayed hardware
2. improper hardware or software configuration
3. SCSI drives properly terminated
4. Video card and monitor settings configured properly
5. Hardware Compatibility List (HCL)
6. IRQ/IO address conflict
7. • Both devices will not work. Check boot errors with dmesg or /var/log/boot.log or /var/log/messages
   • Look for non PnP devices to claim resources that are assigned to another device. The device may be configured manually, or there may be a chip on the device itself.
8. Check the /proc directory for hardware conflicts
9. Device drivers
10. • Hardware and drivers are typically checked at boot time by the kudzu program
    • Drivers are configured manually by loading the appropriate kernel module or recompiling the kernel
11. Hard drive failure
12. • Is it hardware or software RAID?
    • RAID level 1 or 5 will permit data to be regenerated. Check the configuration of the RAID utility.
13. Restoring non critical directories after hard drive failure
14. 1. Turn the power off to the computer. Remove the AC power cable from the back of the unit. Open the cover. Remove and replace the failed hard drive unit.
    2. Reboot the system.
    3. Use fdisk to create partitions as necessary on the new hard drive.
    4. Use the command mkfs to create filesystems on the new partitions.
    5. Using a back-up utility, restore the original data.
    6. Enter the appropriate data into the /etc/fstab system configuration file.
15. Restoring critical system directories after hard drive failure
16. 1. Turn the power off to the computer. Remove the AC power cable from the back of the unit. Open the cover. Remove and replace the failed hard drive unit.
    2. Reinstall the operating system. Mount file systems from additional hard drives as necessary.
    3. Using a back-up utility, restore the original data.

Software Type Problems

Once a problem has been determined to be in the software category there are two general types that are possible. The question then becomes whether the software problem pertains to application software, or to operating system software.

1. Application Software Problems
• Often caused by missing program library files, by process limitations, and/or by conflicting applications.
• Software dependency problems - package managers test dependency during install and uninstall procedures. Source code typically checks dependency during pre-compile, and refuses the makefile command if the dependency requirement is not met. Run the command rpm -V to check and verify package dependencies.
• Use the ldd command to check a specific program for dependent library files.
• Typical locations for library files are /lib or /usr/lib. Also refer to /etc/ld.so.conf and /etc/ld.so.cache are updated at install time with the ldconfig utility.
• Typically utilities and software will install with a install.sh script. If this script is not available, shared libraries will need to be manually copied to the above locations, and the configuration files manually updated.
• Linux processes can be a source of problematic behavior. Child or zombie processes can consume excessive resources to the point of rendering the system sluggish or locked. Look for a process PID to identify that process. Kill the parent process to kill all subsequent child processes.
• Linux processes that need, open, or use many or excessive filehandles can cause problems with system resources and operation. The system limit of 1024 filehandles as well as the number of possible processes can be configured with the ulimit command. see the man pages for details.
• Does the specific application have a dedicated log file? Check /var/log to explore this option. Also check for the existance of /var/log/*application* for additional log files.
• Problems can arise as a process attempts to access resources, or from different processes attempting to access the same resources at the same time. This can often be resolved by restarting the process using a SIGHUP. Attempt to start the process in Single User Mode to observe the process in a simpler form.
• Keep software up to date with patches, updates, upgrades, etc. for best performance. Always verify software compatibility with other software, especially when the processes run at the same time.
2. Operating System Software Problems
• Often caused by missing or corrupt boot loaders, file systems, serial devices, etc.
• Boot loaders can be problematic when loading the operting system. LILO bootloader can be simplified in the linear mode instead of compact. Look in /etc/lilo.conf for configuration.
• Create a boot disk with the mkbootdisk command to ensure that the system can be booted for repair in an emergency.
• Filesystem corruption is the cause of many problems. If the  /  root filesystem becomes corrupt, the system will become unstable. Use the command fsck to check and repair a filesystem. Do not restore data to a corrupt filesystem. To recreate a filesystem that is damaged or corrupted beyond repair, perform the steps as listed below.
• Repairing a filesystem on a non-critical drive:
1. Unmount the filesystem.
2. Run the fsck -f command on the filesystem.
3. If the fsck command fails to repair the system, recreate the filsystem with the mkfs command.
4. Restore the data to the filesystem from a back-up utility.
• To repair a filesystem on a system-critical drive:
1. Place the installation CD into the CD drive and boot the system. Ensure that the system boots to the CD.
2. At the welcome screen, or when prompted, type linux rescue and press Enter. Unless necessary, do not enable network support.
3. If prompted, and if desired, the rescue process will look for and mount existing installations. Choose this step if necessary. If restoring a filesystem is the only intended outcome, skip this process.
4. Use the mkfs command on the CD-ROM Linux system to create the new  /  root filesystem.
5. Use a back-up utility on the CD-ROM (tar, dump, or cpio) to restore the original data to the recreated  /  root filesystem.
6. Type exit to end the shell and reboot the system.
• A complete bootable Linux operating system is available on CD by using the Knoppix or BBC Linux utilities. These are available by download, and provide a complete system with repair utilities for working through system problems that exist on the hard drive. See the distribution section for more info.
• Serial devices are common sources of problems. Serial port configuration for most devices is done automatically, however some devices do not respone to PnP. When conflicts occur, the setserial command can be used to adjust device parameters. Some options are listed below, or see the man pages for further info.
1. port -n - Set theI/O address to n for a serial device.
2. irq -n - Sets the IRQ to 'n' for the serial device.
3. auto_irq - Attempts to automatically detect the IRQ setting for a serial device.
4. spd_normal - Assign 38.4KB/s speed to the serial port.
5. spd_hi - Assign 56Kb/s speed to the serial port.
6. spd_vhi - Assign 115KB/s speed to the serial port.

Documentation

A general rule of over-documentation will serve better in a time of emergency than one of under-documentation.

1. what are good documentation guidelines?
2. what should be documented?

title

paragraph

1. list

1. Introduction and History
2. Installation, Advanced Installation, and Usage
3. The Linux Kernel and the Boot Process
4. Filesystems - Management & Administration
5. The BASH and Other Shells
6. System Initialization and the X Environment
7. Linux Processes
8. Linux Administration, Peripherals, and Hardware
9. Software Installation and Management
10. Backups and Log Files
11. Performance and Problems
12. Network Configuration
13. Security
14. Key Linux Commands
15. Essential Linux Definitions