James Tanner

legacy-wiki

Rhnpush-5.2

February 6, 2011

Recovered from the older tannerjc.net wiki snapshot dated January 23, 2016.

Rhnpush-code-hang

  • httpd strace
  • strace -e open -ftttvo /tmp/strace.out /usr/sbin/httpd -f /etc/rhn/satellite-httpd/conf/httpd.conf
  • raw
[Tue Feb 01 16:37:31 2011] [notice] caught SIGTERM, shutting down
[Tue Feb 01 16:37:33 2011] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Feb 01 16:37:33 2011] [notice] Digest: generating secret for digest authentication ...
[Tue Feb 01 16:37:33 2011] [notice] Digest: done
[Tue Feb 01 16:37:33 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Tue Feb 01 16:37:33 2011] [notice] Apache/2.2.3 (Red Hat) configured -- resuming normal operations
  • rhnpush
  • rhnpush -vvv -u satadmin -p redhat –server=localhost ImageMagick-6.2.8.0-4.el5_5.2.x86_64.rpm
[root@sat52 ~]# rhnpush -vvv -u satadmin -p redhat --server=localhost ImageMagick-6.2.8.0-4.el5_5.2.x86_64.rpm
Connecting to http://localhost/APP
url is http://localhost/PACKAGE-PUSH
Result codes: 200 OK
Computing md5sum and package Info .This may take sometime ...
Package ImageMagick-6.2.8.0-4.el5_5.2.x86_64.rpm Not Found on RHN Server -- Uploading
Uploading package ImageMagick-6.2.8.0-4.el5_5.2.x86_64.rpm
Using POST request
  • rhn_upload_package_push.log
  • raw

Apache tuning

[root@sat52 ~]# ps -ylC httpd --sort:rss | awk '{print $8}' | fgrep -v RSS | awk '{ s += $1 } END {print s}'
138356
[root@sat52 ~]# echo 138356 / 1024 | bc
135
  • Total process size in MB
[root@sat52 ~]# free -m | fgrep cache: | awk '{print $3}'
799
  • MaxClients calculation
  • MaxClients ≈ (RAM - size_all_other_processes)/(size_apache_process)
  • MaxClients = (1469 - 799) / (135)
[root@sat52 ~]# echo (1469 - 799) / (135) | bc
4
  • RSS total
[root@sat52 ~]# ps auwxx --sort rss | awk '{print $6}' | fgrep -v RSS | awk '{s += $1} END {print s}' | xargs -I {} echo {} / 1024 | bc
2142
  • http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html blockquote As to the load spikes which can occur, what this comes down to is the startup costs of loading the Python web application being run. Here the problem is that Apache will create additional child server processes to meet demand. Because Python web applications these days generally have a lot of dependencies and need to load a lot of code they will not start up quickly. That startup is costly actually serves to multiply the severity of the problem, because although the additional processes are starting up, if they take too long, Apache will decide that it still doesn’t have enough processes and will start creating more. In the worst case this can snowball until you have completely swamped your machine. /blockquote blockquote The solution here is not to create only a minimal number of servers when Apache starts, but create closer to what would be the maximum number of processes you would expect to require to handle the load. That way the processes always exist and are ready to handle requests and you will not end up in a situation where Apache needs to suddenly create a huge number of processes. /blockquote

Apache debugging

[root@g73f14vm ~]# pstack 1364
#0  0x00007ff681057700 in __accept_nocancel () from /lib64/libpthread.so.0
#1  0x00007ff681283955 in apr_socket_accept () from /usr/lib64/libapr-1.so.0
#2  0x00007ff6827d0d4d in unixd_accept ()
#3  0x00007ff6827cee96 in child_main ()
#4  0x00007ff6827cf20a in make_child ()
#5  0x00007ff6827cf2c7 in startup_children ()
#6  0x00007ff6827cfc15 in ap_mpm_run ()
#7  0x00007ff6827a77be in main ()
  • gdb
[root@g73f14vm ~]# cat commands
bt
where
[root@g73f14vm ~]# gdb httpd 1364 -batch -x commands
[Thread debugging using libthread_db enabled]
0x00007ff681057700 in __accept_nocancel () from /lib64/libpthread.so.0
#0  0x00007ff681057700 in __accept_nocancel () from /lib64/libpthread.so.0
#1  0x00007ff681283955 in apr_socket_accept (new=0x7fff99fa3998, sock=0x7ff683ac5ae0, connection_context=0x7ff683be4198) at network_io/unix/sockets.c:204
#2  0x00007ff6827d0d4d in unixd_accept (accepted=0x7fff99fa3a00, lr=value optimized out, ptrans=value optimized out) at /usr/src/debug/httpd-2.2.17/os/unix/unixd.c:506
#3  0x00007ff6827cee96 in child_main (child_num_arg=value optimized out) at /usr/src/debug/httpd-2.2.17/server/mpm/prefork/prefork.c:643
#4  0x00007ff6827cf20a in make_child (s=0x7ff683ace860, slot=7) at /usr/src/debug/httpd-2.2.17/server/mpm/prefork/prefork.c:763
#5  0x00007ff6827cf2c7 in startup_children (number_to_start=1) at /usr/src/debug/httpd-2.2.17/server/mpm/prefork/prefork.c:781
#6  0x00007ff6827cfc15 in ap_mpm_run (_pconf=value optimized out, plog=value optimized out, s=value optimized out) at /usr/src/debug/httpd-2.2.17/server/mpm/prefork/prefork.c:1002
#7  0x00007ff6827a77be in main (argc=1, argv=0x7fff99fa3e68) at /usr/src/debug/httpd-2.2.17/server/main.c:739
#0  0x00007ff681057700 in __accept_nocancel () from /lib64/libpthread.so.0
#1  0x00007ff681283955 in apr_socket_accept (new=0x7fff99fa3998, sock=0x7ff683ac5ae0, connection_context=0x7ff683be4198) at network_io/unix/sockets.c:204
#2  0x00007ff6827d0d4d in unixd_accept (accepted=0x7fff99fa3a00, lr=value optimized out, ptrans=value optimized out) at /usr/src/debug/httpd-2.2.17/os/unix/unixd.c:506
#3  0x00007ff6827cee96 in child_main (child_num_arg=value optimized out) at /usr/src/debug/httpd-2.2.17/server/mpm/prefork/prefork.c:643
#4  0x00007ff6827cf20a in make_child (s=0x7ff683ace860, slot=7) at /usr/src/debug/httpd-2.2.17/server/mpm/prefork/prefork.c:763
#5  0x00007ff6827cf2c7 in startup_children (number_to_start=1) at /usr/src/debug/httpd-2.2.17/server/mpm/prefork/prefork.c:781
#6  0x00007ff6827cfc15 in ap_mpm_run (_pconf=value optimized out, plog=value optimized out, s=value optimized out) at /usr/src/debug/httpd-2.2.17/server/mpm/prefork/prefork.c:1002
#7  0x00007ff6827a77be in main (argc=1, argv=0x7fff99fa3e68) at /usr/src/debug/httpd-2.2.17/server/main.c:739
  • gcore
[root@g73f14vm ~]# gcore 1364
[Thread debugging using libthread_db enabled]
0x00007ff681057700 in __accept_nocancel () from /lib64/libpthread.so.0
Saved corefile core.1364

[root@g73f14vm ~]# gdb httpd core.1364 -batch -x commands
[New Thread 1364]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/bb/59d9741d29c8d4ef59c5aa92df45b808bb0eb9
Core was generated by `/usr/sbin/httpd'.
#0  0x00007ff681057700 in __accept_nocancel () from /lib64/libpthread-2.12.90.so
#0  0x00007ff681057700 in __accept_nocancel () from /lib64/libpthread-2.12.90.so
#1  0x00007ff681283955 in apr_socket_accept (new=0x7fff99fa3998, sock=0x7ff683ac5ae0, connection_context=0x7ff683be4198) at network_io/unix/sockets.c:204
#2  0x00007ff6827d0d4d in unixd_accept (accepted=0x7fff99fa3a00, lr=value optimized out, ptrans=value optimized out) at /usr/src/debug/httpd-2.2.17/os/unix/unixd.c:506
#3  0x00007ff6827cee96 in child_main (child_num_arg=value optimized out) at /usr/src/debug/httpd-2.2.17/server/mpm/prefork/prefork.c:643
#4  0x00007ff6827cf20a in make_child (s=0x7ff683ace860, slot=7) at /usr/src/debug/httpd-2.2.17/server/mpm/prefork/prefork.c:763
#5  0x00007ff6827cf2c7 in startup_children (number_to_start=1) at /usr/src/debug/httpd-2.2.17/server/mpm/prefork/prefork.c:781
#6  0x00007ff6827cfc15 in ap_mpm_run (_pconf=value optimized out, plog=value optimized out, s=value optimized out) at /usr/src/debug/httpd-2.2.17/server/mpm/prefork/prefork.c:1002
#7  0x00007ff6827a77be in main (argc=1, argv=0x7fff99fa3e68) at /usr/src/debug/httpd-2.2.17/server/main.c:739
#0  0x00007ff681057700 in __accept_nocancel () from /lib64/libpthread-2.12.90.so
#1  0x00007ff681283955 in apr_socket_accept (new=0x7fff99fa3998, sock=0x7ff683ac5ae0, connection_context=0x7ff683be4198) at network_io/unix/sockets.c:204
#2  0x00007ff6827d0d4d in unixd_accept (accepted=0x7fff99fa3a00, lr=value optimized out, ptrans=value optimized out) at /usr/src/debug/httpd-2.2.17/os/unix/unixd.c:506
#3  0x00007ff6827cee96 in child_main (child_num_arg=value optimized out) at /usr/src/debug/httpd-2.2.17/server/mpm/prefork/prefork.c:643
#4  0x00007ff6827cf20a in make_child (s=0x7ff683ace860, slot=7) at /usr/src/debug/httpd-2.2.17/server/mpm/prefork/prefork.c:763
#5  0x00007ff6827cf2c7 in startup_children (number_to_start=1) at /usr/src/debug/httpd-2.2.17/server/mpm/prefork/prefork.c:781
#6  0x00007ff6827cfc15 in ap_mpm_run (_pconf=value optimized out, plog=value optimized out, s=value optimized out) at /usr/src/debug/httpd-2.2.17/server/mpm/prefork/prefork.c:1002
#7  0x00007ff6827a77be in main (argc=1, argv=0x7fff99fa3e68) at /usr/src/debug/httpd-2.2.17/server/main.c:739