Performance Tuning

From Jtwiki
Jump to: navigation, search

Contents

General

Troubleshooting Tools
sar
provided by the sysstat package
http://pagesperso-orange.fr/sebastien.godard/faq.html
http://ksar.atomique.net/download.html
iostat
provided by the sysstat package
vmstat
provided by the sysstat package
tcpdump
http://www.tcpdump.org/#documentation
oProfile
http://oprofile.sourceforge.net/about/
SystemTap
http://sourceware.org/systemtap/
gdb
http://www.gnu.org/software/gdb/documentation/
cron
http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/sysadmin-guide/ch-autotasks.html
Tuning Parameters
/etc/sysctl.conf
/etc/security/limits.conf
NIC bonding
http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding
Benchmarking
bonnie++
http://download.fedora.redhat.com/pub/epel/5/i386/bonnie++-1.03a-6.el5.i386.rpm
http://download.fedora.redhat.com/pub/epel/5/x86_64/bonnie++-1.03a-6.el5.x86_64.rpm
Oracle stored procedures
apache benchmark - ab

Disk IO

Traditionally, it’s common to assume that the closer to 100% utilization a device is, the more saturated it is. This might be true when the system device corresponds to a single physical disk. However, with devices representing a LUN of a modern storage box, the story might be completely different. Rather than looking at device utilization, there is another way to estimate how loaded a device is. Look at the non-existent column I mentioned above — qutim — the average time a request is spending in the queue. If it’s insignificant, compare it to svctim — the IO device is not saturated. When it becomes comparable to svctim and goes above it, then requests are queued longer and a major part of response time is actually time spent waiting in the queue. The figure in the await column should be as close to that in the svctim column as possible. If await goes much above svctim, watch out! The IO device is probably overloaded.

sar disktop

#!/bin/bash

# Check particular hosts for their disk performance metrics
#       note1: run this on a trusted host
#       note2: bonnie++ is a better tool for benchmarking IO

HOSTS="ut40xl016 ut40xl045 ut40xl017 ut40xl046 ut40xl018 ut40xl047"

echo "host,topread,topwrite,toputil"

# Loop through hosts
for h in `echo $HOSTS | sort -n`; do
        #echo $h

        # Get list of sar data files
        SAFILES=`ssh root@$h 'ls /var/log/sa/sa[0-9][0-9]'`

        # Loop through sar data files and find top READ WRITE AND UTILIZATION
        TOPREAD=0
        TOPWRITE=0
        TOPUTIL=0
        for f in `echo $SAFILES`; do
                #echo $f
                READ=`ssh root@$h "sar -d -f $f" |  fgrep -v DEV | awk '{print $5}' | sort -n  | tail -n1 | cut -d\. -f1`
                WRITE=`ssh root@$h "sar -d -f $f" |  fgrep -v DEV | awk '{print $6}' | sort -n  | tail -n1 | cut -d\. -f1`
                UTIL=`ssh root@$h "sar -d -f $f" |  fgrep -v DEV | awk '{print $11}' | sort -n  | tail -n1 | sed 's/\...//g'`

                if [ "$READ" -gt  "$TOPREAD" ]; then
                        TOPREAD=`echo $READ`
                        #echo "TOP READ is :" $TOPREAD
                fi 

                if [ "$WRITE" -gt "$TOPWRITE" ]; then
                        TOPWRITE=`echo $WRITE`
                        #echo "TOP WRITE is :" $TOPWRITE
                fi


                if [ "$UTIL" -gt "$TOPUTIL" ]; then
                        TOPUTIL=`echo $UTIL`
                        #echo "TOP UTIL is :" $TOPUTIL
                fi
        done

        echo "$h,$TOPREAD,$TOPWRITE,$TOPUTIL"
done

Network

   # don't cache ssthresh from previous connection
   net.ipv4.tcp_no_metrics_save = 1
   # recommended to increase this for 10G NICS
   net.core.netdev_max_backlog = 30000 
  • sysctl net.ipv4.tcp_available_congestion_control
/sbin/modprobe tcp_htcp 
/sbin/modprobe tcp_cubic
  • For long fast paths, we highly recommend using cubic or htcp.
  • Warning on Large MTUs: If you have configured your Linux host to use 9K MTUs, but the connection is using 1500 byte packets, then you actually need 9/1.5 = 6 times more buffer space in order to fill the pipe. In fact some device drivers only allocate memory in power of two sizes, so you may even need 16/1.5 = 11 times more buffer space!
  • warning for both 2.4 and 2.6: for very large BDP paths where the TCP window is > 20 MB, you are likely to hit the Linux SACK implementation problem. If Linux has too many packets in flight when it gets a SACK event, it takes too long to located the SACKed packet, and you get a TCP timeout and CWND goes back to 1 packet. Restricting the TCP buffer size to about 12 MB seems to avoid this problem, but clearly limits your total throughput. Another solution is to disable SACK.

jtanner@trainwreck:~$ ifconfig -a | fgrep -e eth -e br -e que
br0       Link encap:Ethernet  HWaddr 40:61:86:BE:7D:D0  
          collisions:0 txqueuelen:0 
eth0      Link encap:Ethernet  HWaddr 00:10:4B:1F:95:71  
          collisions:0 txqueuelen:1000 
eth1      Link encap:Ethernet  HWaddr 40:61:86:BE:7D:D0  
          collisions:0 txqueuelen:1000 
          collisions:0 txqueuelen:0 
virbr0    Link encap:Ethernet  HWaddr 1E:16:34:68:69:02  
          collisions:0 txqueuelen:0 
          collisions:0 txqueuelen:500 
jtanner@trainwreck:~$ tc -s qdisc show dev eth0
qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 41873920 bytes 349870 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

jtanner@trainwreck:~$ tc -s qdisc show dev eth1
qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 417024970 bytes 479786 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
jtanner@trainwreck:~$ cat /proc/net/softnet_stat
001def9c 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
000dad07 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
-------- --------
   \        \_ drop count
    \_ packet count
jtanner@trainwreck:~$ cat /proc/net/snmp | grep '^Ip:' | cut -f17 -d' ' 
ReasmFails
0
jtanner@trainwreck:~$ ps axo pid,comm,util | grep softirq
    4 ksoftirqd/0      0
    7 ksoftirqd/1      0

Applications

Oracle

vmstat

vmstat is the first place to check performance issues.

    • If the wa (time waiting for I/O) column is high, this is usually an indication that the storage subsystem is overloaded. See recipes 8-9 and 8-10 for identifying the sources of I/O contention.
    • If b (processes sleeping) is consistently greater than 0, then you may not have enough CPU processing power. See recipe 8-2 for identifying Oracle processes and SQL state-ments consuming the most CPU.
    • If so (memory swapped out to disk) and si (memory swapped in from disk) are consistently greater than 0, you may have a memory bottleneck. See recipe 8-5 for details on identifying Oracle processes and SQL statements consuming the most memory.

Enumerating high cpu oracle pids

Grab the pid number

ps -e -o pcpu,pid,user,tty,args | grep -i oracle | sort -n -k 1 -r | head

Query the database for more information

SET LINESIZE 80 HEADING OFF FEEDBACK OFF
SELECT
  RPAD('USERNAME : ' || s.username, 80) ||
  RPAD('OSUSER    : ' || s.osuser, 80) ||
  RPAD('PROGRAM : ' || s.program, 80) ||
  RPAD('SPID      : ' || p.spid, 80) ||
  RPAD('SID       : ' || s.sid, 80) ||
  RPAD('SERIAL# : ' || s.serial#, 80) ||
  RPAD('MACHINE : ' || s.machine, 80) ||
  RPAD('TERMINAL : ' || s.terminal, 80)
FROM v$session s,
     v$process p
WHERE s.paddr = p.addr
AND    p.spid = '&PID_FROM_OS';

Enumerating high memory oracle pids

Grab the pid number

ps -e -o pmem,pid,user,tty,args | grep -i oracle | sort -n -k 1 -r | head

Query the database for more information

SET LINESIZE 80 HEADING OFF FEEDBACK OFF
SELECT
  RPAD('USERNAME : ' || s.username, 80) ||
  RPAD('OSUSER    : ' || s.osuser, 80) ||
  RPAD('PROGRAM : ' || s.program, 80) ||
  RPAD('SPID      : ' || p.spid, 80) ||
  RPAD('SID       : ' || s.sid, 80) ||
  RPAD('SERIAL# : ' || s.serial#, 80) ||
  RPAD('MACHINE : ' || s.machine, 80) ||
  RPAD('TERMINAL : ' || s.terminal, 80) ||
  RPAD('SQL TEXT : ' || q.sql_text, 80)
FROM v$session s
    ,v$process p
    ,v$sql      q
WHERE s.paddr           = p.addr
AND    p.spid           = '&PID_FROM_OS'
AND    s.sql_address    = q.address(+)
AND    s.sql_hash_value = q.hash_value(+);

Enumerating high disk utilization

  • Look for devices with abnormally high blocks read or written per second.
  • If any device is near 100 percent utilization, that’s a strong indicator I/O is a bottleneck.

If the bottlenecked disks are used by Oracle, then you can query the data dictionary to identify sessions with high I/O activity. The following query is useful for determining which SQL statements generate the most read/write activity:

SELECT *
FROM
(SELECT
  parsing_schema_name
 ,direct_writes
 ,SUBSTR(sql_text,1,75)
 ,disk_reads
FROM v$sql
ORDER BY disk_reads DESC)
WHERE rownum < 20;

Determining which objects produce the heaviest I/O activity in the database:

SELECT *
FROM
(SELECT
  s.statistic_name
 ,s.owner
 ,s.object_type
 ,s.object_name
 ,s.value
  FROM v$segment_statistics s
  WHERE s.statistic_name IN
     ('physical reads', 'physical writes', 'logical reads',
      'physical reads direct', 'physical writes direct')
ORDER BY s.value DESC)
WHERE rownum < 20;

Enumerating network utilization

netstat -ptc

Proto Recv-Q Send-Q Local Address Foreign Address State              PID/Program name
tcp        0      0 rmug.com:62386 rmug.com:1521      ESTABLISHED    22864/ora_pmon_RMDB
tcp        0      0 rmug.com:53930 rmug.com:1521      ESTABLISHED    6091/sqlplus
tcp        0      0 rmug.com:1521 rmug.com:53930      ESTABLISHED    6093/oracleRMDB1
tcp        0      0 rmug.com:1521 rmug.com:62386      ESTABLISHED    10718/tnslsnr

If the Send-Q (bytes not acknowledged by remote host) column has an unusually high value for a process, this may indicate an overloaded network. The useful aspect about the previous output is that you can determine the operating system process ID (PID) associated with a network connection.

Tomcat

   1. Decide what needs to be measured.
   2. Decide how to measure.
   3. Measure.
   4. Understand the implications of what you learned.
   5. Modify the configuration in ways that are expected to improve the measurements.
   6. Measure and compare with previous measurements.
   7. Go back to step 4. 

Headline text

Profiling resource usage with cron and ps

[root@t5400 ~]# crontab -l
*/2	*	*	*	* 	ps aux > /root/ps.output/ps.`date \+\%Y-\%m-\%d_\%H_\%M_\%S`.out

To archive and clean out the /root/ps.output directory every night do something like this:

Put this in /root/ps.archive.sh

#!/bin/bash

if [ -f /root/ps.output-archive.tar.bz2 ]
then
 /bin/rm -f /root/ps.output-archive.tar.bz2
fi

/bin/tar -cjf /root/ps.output-archive.tar.bz2 /root/ps.output
  • chmod +x /root/ps.archive.sh
  • crontab -e
  • 10 0 * * * /root/ps.archive.sh > /dev/null 2>&1

So with that cron entry and script at 10 past midnight every night it will tar up what is in /root/ps.output/ and remove the stuff in there. It should leave enough if you have a hang to determine the problem.

Personal tools