James Tanner

legacy-wiki

General Linux

June 18, 2013

Recovered from the older tannerjc.net wiki snapshot dated January 23, 2016.

autofs

http://people.redhat.com/jmoyer/

/etc/syslog.conf

  • daemon.* /var/log/debug /etc/sysconfig/autofs
  • DEFAULT_LOGGING=debug

dhcp

  • release ip lease: dhclient -r
  • obtain new lease: dhclient

bonding

[root@dhcp243-226 ~]# cat /etc/modprobe.conf | tail -n 2
alias bond0 bonding
options bond0 mode=active-backup
  • balance-rr [mode 0] bond w/ 2 vmware virtual NICs (this is without setting any options)…
[root@dhcp243-94 ~]# cat /proc/net/bonding/bond0  | egrep -v ^\$
Ethernet Channel Bonding Driver: v2.6.3-rh (June 8, 2005)
Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:0c:29:9a:d9:59
Slave Interface: eth2
MII Status: up
Link Failure Count: 0

[root@dhcp243-94 ~]# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 00:0C:29:9A:D9:59
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1

[root@dhcp243-94 ~]# ifconfig eth2
eth2      Link encap:Ethernet  HWaddr 00:0C:29:9A:D9:59
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1

[root@dhcp243-94 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.11.242.0     0.0.0.0         255.255.254.0   U     0      0        0 bond0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 bond0
0.0.0.0         10.11.243.254   0.0.0.0         UG    0      0        0 bond0
  • Active-backup [mode1] ..
[root@dhcp243-226 ~]# cat /proc/net/bonding/bond0  | egrep -v ^\$
Ethernet Channel Bonding Driver: v2.6.3-rh (June 8, 2005)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth1
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:0c:29:9a:d9:59
Slave Interface: eth2
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:0c:29:9a:d9:63

[root@dhcp243-226 ~]# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 00:0C:29:9A:D9:59
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1

[root@dhcp243-226 ~]# ifconfig eth2
eth2      Link encap:Ethernet  HWaddr 00:0C:29:9A:D9:59
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1

[root@dhcp243-226 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.11.242.0     0.0.0.0         255.255.254.0   U     0      0        0 bond0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 bond0
0.0.0.0         10.11.243.254   0.0.0.0         UG    0      0        0 bond0

Boot Proces

cracklib: disabling dictionary checks

This doesn’t actually disable dictionary checking, but it does empty the dictionary so that other requirements and policies can be made:

#EMPTYING THE DICTIONARY
mkdir /usr/lib/old.cracklib
mv /usr/lib/crack* old.cracklib/
echo words  /root/words.txt
mkdict /root/words.txt | packer /usr/lib/cracklib_dict

#RESTORING
rm /usr/lib/crack*
cd /usr/lib
cp /usr/lib/old.cracklib/* .

creating a password

  • openssl can create a string of characters useful as a password:
[root@server102 ~]# openssl rand -base64 6
lipIILym
[root@server102 ~]# openssl rand -base64 6
Z1o1rIeb
[root@server102 ~]# openssl rand -base64 6
GPUWyLSR
[root@server102 ~]# openssl rand -base64 6
5Al/ZNzW
[root@server102 ~]#

CIFS/smbfs troubleshooting

In order to troubleshoot CIFS errors, refer to this document:

http://pserver.samba.org/samba/ftp/cifs-cvs/linux-cifs-client-guide.pdf

The pseudo-file /proc/fs/cifs/DebugData shows information on the status of cifs mounts, sessions, and active network requests’''

  • cat /proc/fs/cifs/DebugData

The dmesg (Linux message log) will show certain serious errors, but enabling additional cifs debugging flags can increase the debug output.

  • echo 1 /proc/fs/cifs/DebugData
  • echo 3 /proc/fs/cifs/cifsFYI

One of the most powerful tools for analyzing networking or network filesystem (cifs client or SMB/CIFS server) problems is Wireshark

tcpdump can capture this data also:

http://kbase.redhat.com/faq/docs/DOC-6391

Finding Kernel Function That Caused D Process State

If there are too many processes in the D state and you can’t find out what is happening, issue the following command:

  • objdump -Dr /boot/vmlinux-uname -r /tmp/kernel.dump

and then get the process list:

  • ps axfwln
# ps axfwln

F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND

100 0 20418 20417 17 0 2588 684 - R ? 0:00 ps axfwln

100 0 1 0 8 0 1388 524 145186 S ? 0:00 init

040 0 8670 1 9 0 1448 960 145186 S ? 0:00 syslogd -m 0

040 0 8713 1 10 0 1616 1140 11ea02 S ? 0:00 crond

Look for a number under the WCHAN column for the process in question. Then you should open /tmp/kernel.dump in an editor, find that number in the first column and then scroll backward to the first function name, which can look like this:

c011e910 sys_nanosleep:

Then you can tell if the process lives or is blocked into the found function.

Filesystem repair

dumpe2fs and e2fsck …

http://kbase.redhat.com/faq/docs/DOC-1866

Finding other superblocks ..

[root@rhel5box rc.d]# dumpe2fs /dev/mapper/VolGroup00-LogVol00 | grep Backup
dumpe2fs 1.39 (29-May-2006)
  Backup superblock at 32768, Group descriptors at 32769-32769
  Backup superblock at 98304, Group descriptors at 98305-98305
  Backup superblock at 163840, Group descriptors at 163841-163841
  Backup superblock at 229376, Group descriptors at 229377-229377
  Backup superblock at 294912, Group descriptors at 294913-294913
  Backup superblock at 819200, Group descriptors at 819201-819201
  Backup superblock at 884736, Group descriptors at 884737-884737
  Backup superblock at 1605632, Group descriptors at 1605633-1605633

Kernel

[root@rhel4box ~]# cat /proc/sys/kernel/printk
6	4	1	7
[root@rhel4box ~]# echo 8   /proc/sys/kernel/printk
[root@rhel4box ~]# cat /proc/sys/kernel/printk
8	4	1	7
  • modfino - Info about kernel modules and their available parameters.
[root@rhel5box Documentation]# modinfo qla2xxx
parm:           ql2xlogintimeout:Login timeout value in seconds. (int)
parm:           qlport_down_retry:Maximum number of command retries to a port that returns a PORT-DOWN status. (int)
parm:           ql2xplogiabsentdevice:Option to enable PLOGI to devices that are not present after a Fabric scan.  This is needed for several broken switches. Default is 0 - no PLOGI. 1 - perfom PLOGI. (int)
parm:           ql2xloginretrycount:Specify an alternate value for the NVRAM login retry count. (int)
parm:           ql2xallocfwdump:Option to enable allocation of memory for a firmware dump during HBA initialization.  Memory allocation requirements vary by ISP type.  Default is 1 - allocate memory. (int)
parm:           ql2xextended_error_logging:Option to enable extended error logging, Default is 0 - no logging. 1 - log errors. (int)
parm:           ql2xfdmienable:Enables FDMI registratons Default is 0 - no FDMI. 1 - perfom FDMI. (int)
parm:           ql2xmaxqdepth:Maximum queue depth to report for target devices. (int)
parm:           ql2xqfullrampup:Number of seconds to wait to begin to ramp-up the queue depth for a device after a queue-full condition has been detected.  Default is 120 seconds. (int)
parm:           ql2xenablemsix:Set to enable MSI-X interrupt mechanism. (int)
module_sig:     883f35048fee4afeb9f68d87b0c276112be2809f5ccccda4ed161ad43ffb19bdf81121194a99f3809f698f7594b880432f81e16784c7dc1c73d9db95
[root@rhel5box Documentation]#
  • seeing what options/parameters were passed at boot time ….
[root@rhel5box tmp]# cat /proc/cmdline
ro root=/dev/VolGroup00/LogVol00 rhgb quiet crashkernel=128M@16M

LVM

  • All partitions must be removed from a disk before running pvcreate on the raw device.
[root@rhel5-target ~]# pvcreate /dev/sda
  Device /dev/sda not found (or ignored by filtering).
  • Change pvdisplay to show aliases instead of device paths: set in lvm.conf: preferred_names = [ ^/dev/mapper/ ]

mail

sending a log file as email with mailx

cat rhn-satellite-5-upgrade-scenario-1a.txt | mailx -s document tanner@tannerjc.net

Memory

http://www.pythian.com/news/245/the-mysterious-world-of-shmmax-and-shmall/

Like Soviet central planners there are two important settings. Like most people I knew about shmmax, but it is sly, it is not the maximum amount of memory which can be allocated, it is the maximum size of any shared memory chunk. Shmmax is how big a bite you want per bite from free memory. The real godfather, the wizard behind the curtain is shmall. Its value determines the maximum amount of memory that ALL shared memory can take. Just to make it fun, the actual setting is derived… the maximum amount of memory = shmall * pagesize where pagesize = getconf PAGE_SIZE and shmall = cat /proc/sys/kernel/shmall

Making shmall larger than free RAM is a recipe for paging hell and much gnashing of teeth. Oracle recommends half the RAM, we pushed the envelope and chose 75% as 8 gigabytes of free for OS and cache is just wasteful. Especially given Oracle is already caching hot blocks in its memory.

NMI Events

http://lkml.indiana.edu/hypermail/linux/kernel/0212.0/1264.html

From: Alan Cox (alan@lxorguk.ukuu.org.uk)

On Fri, 2002-12-06 at 14:55, Greg Boyce wrote:
 I work in a company with a large number of Linux machine deployed all
 around the country, and in some of the machines we've been seeing the
 following error:

 Uhhuh. NMI received. Dazed and confused, but trying to continue
 You probably have a hardware problem with your RAM chips

There are several causes of an NMI depending on the system - hardware
failures is one, some systems do it for things like PCI errors, a few
boxes you see them on power management events (notably old 486's)

 Due to the number of machines and their locations, running memtest86 on
 them isn't exactly feasible.

Then buy better ram ;)

 Is there anything besides failing hardware that could be the cause of this
 error? Also, how serious is this error? Some of the machines reporting
 this error have had problems with programs crashing, while others seem to
 run fine.

Take a sample set of machines which have been crashing and run memtest86
on a couple. That should tell you if it is RAM. From a sample you can
then figure out how to handle the rest (things that come to mind if
memtest86 fails on the test machines include replacing the ram in a few
more then taking the old ram back to test)
/pre.

== RPC ==
* RPC based services have program number
* RPC program numbers in /etc/rpc

[root@station2 misc]# grep nfs /etc/rpc
nfs		100003	nfsprog
rpcnfs		100116	na.rpcnfs
nfs_acl		100227
pcnfsd		150001	pcnfs
bwnfsd		545580417
  • RPC based services register with portmap
  • portmap service keeps track of RPC services
  • portmap tracks what tcp/udp port a particular RPC services has randomly chosen
[root@station2 misc]# rpcinfo -p digweed
   program vers proto   port
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  54939  status
    100024    1   tcp  50603  status
    100021    1   udp  55185  nlockmgr
    100021    3   udp  55185  nlockmgr
    100021    4   udp  55185  nlockmgr
    100021    1   tcp  41318  nlockmgr
    100021    3   tcp  41318  nlockmgr
    100021    4   tcp  41318  nlockmgr
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    2   tcp   2049  nfs_acl
    100227    3   tcp   2049  nfs_acl
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100227    2   udp   2049  nfs_acl
    100227    3   udp   2049  nfs_acl
    100005    1   udp  51814  mountd
    100005    1   tcp  59686  mountd
    100005    2   udp  51814  mountd
    100005    2   tcp  59686  mountd
    100005    3   udp  51814  mountd
    100005    3   tcp  59686  mountd

NFS

NFS debugging

http://stromberg.dnsalias.org/~strombrg/NFS-troubleshooting-2.html

# RPC debugging:

    * echo 2048  /proc/sys/sunrpc/rpc_debug
    * grep . /proc/net/rpc/*/content
    * ls -l /proc/fs/nfsd
    * cat /proc/fs/nfs/exports

# NFS debugging:

    * # turn on linux nfs debug
    * echo 1  /proc/sys/sunrpc/nfs_debug
    * # turn off linux nfs debug
    * echo 0  /proc/sys/sunrpc/nfs_debug

TCPDUMP:

Note that NFS requests are very large and much of the detail won’t be printed unless snaplen is increased. Try using ‘-s 192’ to watch NFS traffic.

[root@rhel5box ~]# tcpdump -vvv -s 192 -w tcpdump.out  host opensolaris

STRACE:

[root@rhel5box ~]# strace -fftvo /root/dd.test dd if=/dev/zero of=/stage/irsa-staff/isg_test1 bs=1024k count=1024

NFS: breaking

Block the guest from accessing the NFS share (to simulate netlink down)…

[root@rhel5box ~]# ifconfig eth0 | fgrep inet addr
          inet addr:10.11.243.75  Bcast:10.11.243.255  Mask:255.255.254.0
[root@rhel5box ~]# iptables -A INPUT -s 10.11.243.94 -p udp -j REJECT
[root@rhel5box ~]# iptables -A INPUT -s 10.11.243.94 -p tcp -j REJECT

Jul 15 10:23:23 rhel4box kernel: NFS: find_dirent() returns -523
Jul 15 10:23:23 rhel4box kernel: NFS: find_dirent_page() returns -523
Jul 15 10:23:23 rhel4box kernel: NFS: readdir_search_pagecache() returned -523
Jul 15 10:34:09 rhel4box kernel: nfs: server 10.11.243.75 not responding, still trying

It took 11 minutes before the not responding message was finally entered into /var/log/messages.

Stop portmap on the server …

[root@dhcp242-239 /]# mount 10.11.243.75:/share /test
mount: mount to NFS server '10.11.243.75' failed: System Error: Connection refused.
...
[root@dhcp242-239 /]# mount 10.11.243.75:/share /test
mount: mount to NFS server '10.11.243.75' failed: RPC Error: Program not registered.
  • mounting and unmounting will fail if portmap is stopped
  • If still mounted, the ports will still be mapped and transfers will complete.

Making export non-writable with chmod 400 …

Sep  4 16:46:42 dhcp242-239 kernel: NFS: nfs_update_inode(0:18/1179651 ct=1 info=0x6)
Sep  4 16:46:42 dhcp242-239 kernel: NFS: permission(0:18/1179651), mask=0x1, res=-13

NFS Tuning

http://publib.boulder.ibm.com/infocenter/systems/index.jsp?topic=/com.ibm.aix.prftungd/doc/prftungd/nfs_tuning_client.htm

In general, when implemented appropriately, tuning NFS-specific options can help with issues like the following:

* Decrease the load on the network and on the NFS server
* Work around network problems and client memory usage

http://www.princeton.edu/~unix/Solaris/troubleshoot/nfs.html

http://www.princeton.edu/~unix/Solaris/troubleshoot/nfsstat.html

  • If attribute caching does not make sense (for example, with a mail spool), mount the filesystem with the noac option. If nfsstat reports a high getattr level, actimeo may need to be increased (if the attributes do not change too often).
  • nfsstat reports on most NFS-related statistics. The nfsstat page includes information on tuning suggestions for different types of problems that can be revealed with nfsstat.

NFS mount protocol behavior

  • RHEL3 mount
[jtanner@t5400 Download]$ tshark -r tcpdump.out.3
  1   0.000000 10.11.242.61 - 10.11.242.58 Portmap V2 DUMP Call
  2   0.001252 10.11.242.58 - 10.11.242.61 Portmap V2 DUMP Reply (Call In 1)[Packet size limited during capture]
  3   0.001337 10.11.242.61 - 10.11.242.58 MOUNT V3 MNT Call /export
  4   0.023292 10.11.242.58 - 10.11.242.61 MOUNT V3 MNT Reply (Call In 3)
  5   0.023482 10.11.242.61 - 10.11.242.58 NFS V3 GETATTR Call, FH:0x00fd7cfd
  6   0.023888 10.11.242.58 - 10.11.242.61 NFS V3 GETATTR Reply (Call In 5)  Directory mode:0777 uid:0 gid:0
  7   0.023947 10.11.242.61 - 10.11.242.58 NFS V3 FSINFO Call, FH:0x00fd7cfd
  8   0.024353 10.11.242.58 - 10.11.242.61 NFS V3 FSINFO Reply (Call In 7)
  • RHEL3 mount -o nfsvers=3
[jtanner@t5400 Download]$ tshark -r tcpdump.out.4
  1   0.000000 10.11.242.61 - 10.11.242.58 Portmap V2 DUMP Call
  2   0.000999 10.11.242.58 - 10.11.242.61 Portmap V2 DUMP Reply (Call In 1)[Packet size limited during capture]
  3   0.001115 10.11.242.61 - 10.11.242.58 MOUNT V3 MNT Call /export
  4   0.019354 10.11.242.58 - 10.11.242.61 MOUNT V3 MNT Reply (Call In 3)
  5   0.019627 10.11.242.61 - 10.11.242.58 NFS V3 GETATTR Call, FH:0x00fd7cfd
  6   0.020876 10.11.242.58 - 10.11.242.61 NFS V3 GETATTR Reply (Call In 5)  Directory mode:0777 uid:0 gid:0
  7   0.020929 10.11.242.61 - 10.11.242.58 NFS V3 FSINFO Call, FH:0x00fd7cfd
  8   0.021497 10.11.242.58 - 10.11.242.61 NFS V3 FSINFO Reply (Call In 7)
  • RHEL3 mount -o nfsvers=2
[jtanner@t5400 Download]$ tshark -r tcpdump.out.5
  1   0.000000 10.11.242.61 - 10.11.242.58 Portmap V2 DUMP Call
  2   0.000922 10.11.242.58 - 10.11.242.61 Portmap V2 DUMP Reply (Call In 1)[Packet size limited during capture]
  3   0.001016 10.11.242.61 - 10.11.242.58 MOUNT V2 MNT Call /export
  4   0.014026 10.11.242.58 - 10.11.242.61 MOUNT V2 MNT Reply (Call In 3)
  5   0.014216 10.11.242.61 - 10.11.242.58 NFS V2 GETATTR Call[Packet size limited during capture]
  6   0.014952 10.11.242.58 - 10.11.242.61 NFS V2 GETATTR Reply (Call In 5)
  7   0.015005 10.11.242.61 - 10.11.242.58 NFS V2 STATFS Call[Packet size limited during capture]
  8   0.015437 10.11.242.58 - 10.11.242.61 NFS V2 STATFS Reply (Call In 7)

Export contents of loop mounted ISOs over NFS

If you export a loop mounted ISO remote machines sometimes won’t be able to see that iso9660 file system. The solution is to add the ‘crossmnt’ export option to /etc/exports

/opt/repos 192.168.0.0/255.255.0.0(rw,crossmnt,no_root_squash)

   crossmnt
          This option is similar to nohide but it makes it possible for clients to move from  the
          filesystem  marked  with  crossmnt  to exported filesystems mounted on it.  Thus when a
          child filesystem B is mounted on a parent A, setting crossmnt on A has  the  same
          effect as setting nohide on B.

NTP sync example

Oct  5 14:01:06 dhcp242-239 ntpd[27236]: ntpd exiting on signal 15
Oct  5 14:01:06 dhcp242-239 ntpd[27317]: ntpd 4.2.2p1@1.1570-o Thu May 14 13:00:57 UTC 2009 (1)
Oct  5 14:01:06 dhcp242-239 ntpd[27318]: precision = 4.000 usec
Oct  5 14:01:06 dhcp242-239 ntpd[27318]: Listening on interface wildcard, 0.0.0.0#123 Disabled
Oct  5 14:01:06 dhcp242-239 ntpd[27318]: Listening on interface wildcard, ::#123 Disabled
Oct  5 14:01:06 dhcp242-239 ntpd[27318]: Listening on interface lo, ::1#123 Enabled
Oct  5 14:01:06 dhcp242-239 ntpd[27318]: Listening on interface eth0, fe80::20c:29ff:fe6b:9ae3#123 Enabled
Oct  5 14:01:06 dhcp242-239 ntpd[27318]: Listening on interface lo, 127.0.0.1#123 Enabled
Oct  5 14:01:06 dhcp242-239 ntpd[27318]: Listening on interface eth0, 10.11.242.239#123 Enabled
Oct  5 14:01:06 dhcp242-239 ntpd[27318]: kernel time sync status 0040
Oct  5 14:01:06 dhcp242-239 ntpd[27318]: frequency initialized 0.000 PPM from /var/lib/ntp/drift
Oct  5 14:04:18 dhcp242-239 ntpd[27318]: synchronized to 66.187.233.4, stratum 1
Oct  5 14:21:29 dhcp242-239 ntpd[27318]: time reset +1031.566271 s
Oct  5 14:21:29 dhcp242-239 ntpd[27318]: kernel time sync enabled 0001
Oct  5 14:25:05 dhcp242-239 ntpd[27318]: synchronized to 66.187.233.4, stratum 1

[root@dhcp242-239 ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*clock1.redhat.c .CDMA.           1 u  116  128  377    0.509   -1.360   9.070

Nvidia driver issues

Feb 20 09:35:58 workbox kernel: NVRM: RmInitAdapter failed! (0x26:0xffffffff:1119)
Feb 20 09:35:58 workbox kernel: NVRM: rm_init_adapter(0) failed

[root@workbox ~]# cat /boot/grub/grub.conf | fgrep vmalloc
	kernel /vmlinuz-2.6.30.10-105.2.23.fc11.i586 ro root=/dev/mapper/vg_workbox-lv_root rhgb quiet vmalloc=192M

rescue initrd notes

Rescue mode’s default mounts …

/proc on /proc type proc (rw)
/dev on /dev type tmpfs (rw)
/dev/pts on /dev/pts type devpts (rw)
/sys on /sys type sysfs (rw)
none on /tmp type ramfs (rw)
none on /tmp/ramfs type ramfs (rw)
/proc/bus/usb on /proc/bus/usb type usbfs (rw)
/tmp/loop0 on /mnt/runtime type squashfs (ro)
/selinux on /selinux type selinuxfs (rw)
/dev/myvg/rootvol on /mnt/sysimage type ext3 (rw,data=ordered)
/dev/hda1 on /mnt/sysimage/boot type ext3 (rw,data=ordered)
/tmp/sysfs on /mnt/sysimage/sys type sysfs (rw)
/tmp/proc on /mnt/sysimage/proc type proc (rw)
/dev on /mnt/sysimage/dev type tmpfs (rw)
/dev/pts on /mnt/sysimage/dev/pts type devpts (rw)
/selinux on /mnt/sysimage/selinux type selinuxfs (rw)
  • Chroot environment should have full access to LVM information and commands like pvscan/vgscan/lvscan should return the correct information.

Manually mounting necessary filesystems

    1  mkdir /mnt/sysimage
    2  lvm pvscan
    3  lvm vgscan
    4  lvm lvscan
    6  lvm vgchange -a y myvg
    7  mount /dev/myvg/rootvol /mnt/sysimage/
    8  mount /dev/hda1 /mnt/sysimage/boot/
   12  mount /sys /mnt/sysimage/sys/
   13  mount /proc /mnt/sysimage/proc
   14  mount /dev /mnt/sysimage/dev
   15  mount /dev/pts /mnt/sysimage/dev/pts
   16  chroot /mnt/sysimage

Adding addtional virtual terminals while in rescue or single user mode

  • openvt -cX /bin/bash
  • openvt -c2 /bin/login

Samba / Winbind

[root@dbox-rhel5 sysconfig]# cat /etc/sysconfig/samba
# Options to smbd
SMBDOPTIONS=-D -d3
# Options to nmbd
NMBDOPTIONS=-D -d3
# Options for winbindd
WINBINDOPTIONS=-d3

strace

UDEV

Writing rules:

http://www.reactivated.net/writing_udev_rules.html#example-usbhdd

vnc

html5/ajax client: http://guacamole.sourceforge.net/

http://kanaka.github.com/noVNC/

vsftpd debugging

Make these changes to your /etc/vsftpd/vsftpd.conf and restart the service …

[root@rhel5box vsftpd]# diff vsftpd.conf.bak vsftpd.conf
54c54
 xferlog_std_format=YES
---
 #xferlog_std_format=YES
116a117,121

 #DEBUGGING
 log_ftp_protocol=YES
 anon_upload_enable=YES

XEN and kdump

title Red Hat Enterprise Linux Server (2.6.18-128.1.10.el5xen)
        kernel /xen.gz-2.6.18-128.1.10.el5      crashkernel=128M@16M
        root (hd0,0)
        module /vmlinuz-2.6.18-128.1.10.el5xen ro root=/dev/VolGroup00/LogVol00
        module /new-initrd-2.6.18-128.1.10.el5xen.img

A non-xen PAE kernel -must- be installed.

        self.origCrashKernel =
        self.runningKernel = os.popen(/bin/uname -r).read().strip()
        self.xenKernel = False
            self.xenKernel = True
        if self.xenKernel and self.arch == 'ia64':
            self.showErrorMessage(_(Sorry, ia64 xen kernels do not support kdump
        self.xenKdumpKernel = kernel
        if self.arch == i686 and self.xenKernel:
                        self.xenKdumpKernel = kernel-PAE
            self.origCrashKernel = %dM@%dM % (kdumpMem, kdumpOffset)
        if self.arch in kernelKdumpArches:
        if self.xenKernel and self.kdumpEnabled:
                               (self.xenKdumpKernel, self.xenKdumpKernel))
            origKdumpMem = int(self.origCrashKernel.split(@)[0][:-1])
        if self.xenKernel:
            crashKernel = %iM@%iM % (self.kdumpMem, offset)
                        % (self.bootloader, kpath, self.runningKernel,
            if self.origCrashKernel:
                        % (self.bootloader, kpath, self.runningKernel,
                           args, self.origCrashKernel
        if self.xenKernel and self.kdumpEnabled:

            self.showMessage(_(WARNING: xen kdump support requires a
                               non-xen %s RPM to perform actual crash
                               dump capture. Please be sure you have
                               the non-xen %s RPM of the same version
                               as your xen kernel installed.) %
                               (self.xenKdumpKernel, self.xenKdumpKernel))

Join FLAC files into a single wave file

Download/install shntool

[john@jb-xen History of Our World Part 1]$ shntool join *.flac
Joining [01 Mr. Kirk's Nightmare.flac] (2:26.25) -- [joined.wav] (65:22.00) : 100% OK
Joining [02 Keep The Fire Burning.flac] (2:14.50) -- [joined.wav] (65:22.00) : 100% OK
Joining [03 Vengeance.flac] (1:46.60) -- [joined.wav] (65:22.00) : 100% OK
Joining [04 Deep In Your Eyes.flac] (3:01.07) -- [joined.wav] (65:22.00) : 100% OK
Joining [05 Drumtrip II.flac] (2:27.70) -- [joined.wav] (65:22.00) : 100% OK
Joining [06 Total Ecstasy (Remix).flac] (2:26.10) -- [joined.wav] (65:22.00) : 100% OK
Joining [07 Peace Maker.flac] (3:08.25) -- [joined.wav] (65:22.00) : 100% OK
Joining [08 Mystic Stepper (Feel Better).flac] (2:19.13) -- [joined.wav] (65:22.00) : 100% OK
Joining [09 Renegade Snares.flac] (5:41.00) -- [joined.wav] (65:22.00) : 100% OK
Joining [10 You Got Me Burnin' (Ray Keith  Nookie Remix).flac] (2:36.20) -- [joined.wav] (65:22.00) : 100% OK
Joining [11 Journey From The Light.flac] (2:58.47) -- [joined.wav] (65:22.00) : 100% OK
Joining [12 More Time.flac] (1:54.43) -- [joined.wav] (65:22.00) : 100% OK
Joining [13 Gonna Be Alright.flac] (2:00.65) -- [joined.wav] (65:22.00) : 100% OK
Joining [14 Case Closed.flac] (2:19.22) -- [joined.wav] (65:22.00) : 100% OK
Joining [15 The Slammer (Nookie Remix).flac] (1:29.30) -- [joined.wav] (65:22.00) : 100% OK
Joining [16 What The....flac] (2:25.23) -- [joined.wav] (65:22.00) : 100% OK
Joining [17 Roots 'n Future.flac] (3:02.30) -- [joined.wav] (65:22.00) : 100% OK
Joining [18 Euphoria (Nino's Dream).flac] (2:22.00) -- [joined.wav] (65:22.00) : 100% OK
Joining [19 Darkage.flac] (1:51.25) -- [joined.wav] (65:22.00) : 100% OK
Joining [20 Paragone (10-Inch Remix).flac] (3:48.22) -- [joined.wav] (65:22.00) : 100% OK
Joining [21 Dark Stranger.flac] (2:58.73) -- [joined.wav] (65:22.00) : 100% OK
Joining [22 Incredible Base (Slipmatt Remix).flac] (3:07.42) -- [joined.wav] (65:22.00) : 100% OK
Joining [23 Mixrace Outahand.flac] (1:47.73) -- [joined.wav] (65:22.00) : 100% OK
Joining [24 Let Me Be Your Fantasy (Ray Keith Remix).flac] (2:58.70) -- [joined.wav] (65:22.00) : 100% OK
Joining [25 A New Breed Of Ravers.flac] (2:08.55) -- [joined.wav] (65:22.00) : 100% OK
No padding needed.
[john@jb-xen History of Our World Part 1]$ ls -lh joined.wav
-rw-rw-r-- 1 john john 660M May 10 10:12 joined.wav

SSH

kill a hung ssh client

enter ~ .

in other words:

hit enter, shift+tilde, period

port forward over ssh

ssh -o GSSAPIAuthentication=no -o StrictHostKeyChecking=no -L 5900:localhost:5900 user@server.network.com

now you can connect to localhost:5900 on your workstation and get into 5900 on server.network.com

Misc JUNK

Svn is a revision control repository, usually used for source code. What makes it really powerful is that you can easily have a history of everything that has changed in a file and when. On my systems, I keep /etc in svn, plus bind’s zone files, plus all the non-image web content, and the Network Documentation folder.

Second trick, rsync. I use it to backup my home directory to another box. Very nice when you go through a hard drive/year.

Screen -x was my next pick, but somebody already mentioned it.

echo ProtocolKeepAlives 120 /.ssh/config No more dropped ssh sessions because of stupid nat boxes.

su -u Username -s .. become Username, but keep the current shell. Good for diagnosing permissions problems when the user has a /bin/false shell. (named/www-user/backup/etc).

A little awk goes a long way. Not the big-bad-I-am-a-programming-language-awk, but the smaller-friendlier extract one or two columns of text from something awk. ex. awk ‘{print $2}’ prints the thing in the second column. Add -F the field separator tool and it gets really useful. Better example. Here is a postfix log line. Nov 5 16:27:19 pdc postfix/smtpd[13601]: 92B3F499C25F: client=exprod5mx254.postini.com[64.18.0.49] Here is the awk to extract just the message id. awk -F’: ’ ‘{print $2}’

And here is the I didn’t get this message your mailserver must have eaten it disprover. It searches the maillog for every message from or to a given address and extracts the full email transaction for that message id.

grep -i user@domain.com /var/log/maillog | grep smtpd | awk -F’: ’ ‘{print $2}’ | sort -un temp.fil grep maillog -f temp.fil

Next trick, back ticks. `` Back ticks substitute the output of a command within a command. Ex. Name a file after the date. echo hi date +%Y%M%d.txt

On the subject of dates. date -d’yesterday’ or ’last week’ or ‘-4 hours’ can be handy.

Last one. Loopback nat with Iptables, so you can access local hosts by their external ip. (Instead of setting up split dns.) iptables -I POSTROUTING -s $local_network -d $local_network -j SNAT –to $lan_interface

My local network is 192.168.0.0/24 and the netfilter lan ip is 192.168.0.1, so that becomes… iptables -t nat -A POSTROUTING -d 192.168.0.0/24 -s 192.168.0.0/24 -j SNAT –to 192.168.0.1

Troubleshooting

FAQs

http://www.reddit.com/r/linux/comments/gklx2/help_prepping_for_linux_admin_interview/

colorize terminal output

xrdp

http://ubuntuforums.org/showthread.php?t=392184

  • iptables -F
  • service iptables save
  • yum install tightvnc tightvnc-server xrdp
  • chkconfig xrdp on
  • service xrdp start
[root@g73 xrdp]# pwd
/etc/xrdp
[root@g73 xrdp]# cat startwm.sh
#!/bin/bash
x11vnc -display :0 -localhost
sleep 5
vncviewer localhost:0 -fullscreen
  • yum groupinstall KDE Plasma Workspaces