legacy-wiki
Networking
Recovered from the older tannerjc.net wiki snapshot dated January 23, 2016.
General Troubleshooting Process
-
check ethtool for link detection
-
check dmesg / messages for interface related errors
-
dropping packet: http://kbase.redhat.com/faq/docs/DOC-8721
-
check ifconfig output for errors, drops, or overruns
-
check route -n for proper gateway
-
check /etc/resolv.conf and /etc/hosts for proper syntax and valid entries
-
check for blocking or default drops in iptables: iptables -xvn -L
-
check dns resolution with dig or nslookup
-
check that service is attached to the appropriate ports and addresses:
-
lsof -i :21
-
netstat -tunp
-
check for socket exhaustion: netstat -paunt
-
increase the size of syn backlog queue (net.ipv4.tcp_max_syn_backlog)
-
check ethtool output for the interface
-
rx_fw_discards: http://kbase.redhat.com/faq/docs/DOC-22653
-
check netstat -s
-
capture packets with tcpdump: http://www.wireshark.org/docs/wsug_html_chunked/index.html
-
disable relative sequence numbers for TCP when comparing dumps from multiple machines
-
Edit Preferences Protocols TCP check Relative Sequence Numbers and window Scaling
-
drops
-
missing packets
-
out of order
-
slow response from destination
-
truncated packets
-
packet loss
-
retransmissions without acks: http://kbase.redhat.com/faq/docs/DOC-22403
-
Check nic module options
Benchmarking
- http://www.scl.ameslab.gov/Projects/old/ClusterCookbook/nprun.html
- plot netpipe-file using 4:2
- plot netpipe-file using 1:2
Tuning
-
THIS SECTION IS INCORRECT! **
-
Calculate BDP (bandwidth delay product)
100 Mebibits | 220 | Byte | 3 s
BDP = -------------+------+--------+------- = 39321600 Bytes
s | Mebi | 8 bits |
- http://www.ibm.com/developerworks/linux/library/l-hisock.html
- (wire speed) * 2^20 * (RTT to target) / (8 bits)
- (100 MBps) * 2^20 * (50 sec) / 8 = 655360000
Example w/ Verizon 3G
[root@x200 ~]# ping yahoo.com
PING yahoo.com (67.195.160.76) 56(84) bytes of data.
64 bytes from ir1.fp.vip.ac4.yahoo.com (67.195.160.76): icmp_seq=5 ttl=52 time=158 ms
-
wire speed = 2Mpbs http://www.corenetworkz.com/2010/02/what-is-normal-internet-speed-using.html
-
BDP = (2 * (2^20) / 8) * 158 = 262144 * .158 = 41418.752 bytes
-
Calculate socket buffer
-
socket buffer = BDP / #sockets
-
Check if the NIC module has tunable DMA buffers by 25%
-
RxDescriptors=1024 TxDescriptors=512
-
Increase the tcp advanced window scale
-
net.ipv4.tcp_adv_win_scale=N
-
default N=2: up to (65535 * 22) ~= 256 KB
-
maximum N=14: up to (65535 * 214) ~= 1 GB
- Check for packet fragmentation and tune buffer to compensate
[jtanner@x200 ~]$ cat /proc/net/snmp | grep '^Ip:' | cut -f17 -d' '
ReasmFails
0
[jtanner@x200 ~]$ ps axo pid,comm,util | grep softirq
4 ksoftirqd/0 0
3102 ksoftirqd/1 0
blockquote If ksoftirqd consumes more than a tiny percentage of CPU time, then that is an indication that the system is under heavy softirq load. See the ksoftirqd(9) man page for more information. /blockquote
-
Two approaches for improving interrupt handling
-
Interrupt coalescing: process multiple frames with each interrupt
-
Polling: process queues based on a timer-driven interrupt
-
NAPI devices: /usr/share/doc/kernel-doc-*/Documentation/networking/NAPI_HOWTO.txt
NAT
- echo 1 /proc/sys/net/ipv4/ip_forward
- iptables –table nat –append POSTROUTING –out-interface eth0 -j MASQUERADE
- iptables –append FORWARD –in-interface eth1 -j ACCEPT
Bridging
F18
- chkconfig NetworkManager off
- chkconfig network on
- reboot
- virsh iface-bridge eth0 br0
- if error, check https://bugzilla.redhat.com/show_bug.cgi?id=911806