legacy-wiki
Lab network tuning
Recovered from the older tannerjc.net wiki snapshot dated January 23, 2016.
Plan
create two VMs
koan --server=192.168.2.58 --virt --profile=rhel5u4configremote:1:RedHatdeArgentinaSA --virt-bridge=virbr2 --virt-name=netperf-client
koan --server=192.168.2.58 --virt --profile=rhel5u4configremote:1:RedHatdeArgentinaSA --virt-bridge=virbr2 --virt-name=netperf-server
install netperf
- yum -y install gcc
- wget ftp://ftp.netperf.org/netperf/netperf-2.4.5.tar.bz2
- tar xvf netperf-2.4.5.tar.bz2
- cd netperf-2.4.5
- ./configure –prefix=/opt/netperf-2.4.5
- make
- make install
- scp -r /opt/netperf-2.4.5 root@10.0.0.115:/opt/.
run generic netperf test
- Server
[root@netperfs netperf-2.4.5]# /opt/netperf-2.4.5/bin/netserver
Starting netserver at port 12865
Starting netserver at hostname 0.0.0.0 port 12865 and family AF_UNSPEC
[root@netperfs netperf-2.4.5]# ps aux | fgrep netserver
root 8695 0.0 0.0 8608 312 ? Ss 21:57 0:00 /opt/netperf-2.4.5/bin/netserver
root 8697 0.0 0.1 61120 632 pts/0 R+ 21:57 0:00 fgrep netserver
[root@netperfs netperf-2.4.5]# lsof -i | fgrep netserver
netserver 8695 root 4u IPv4 15698 TCP *:12865 (LISTEN)
-
Client w/ single 100MB nic
-
10^6bits/sec = 122.070312 kilobytes – http://www.google.com/search?q=10^6+bits
-
122.070312 kilobytes * 229.65 = 27.3764132 megabytes – http://www.google.com/search?hl=enq=122.070312+kilobytes+*+229.65
-
BULK DATA TRANSFER PERFORMANCE [root@netperfc ~]# /opt/netperf-2.4.5/bin/netperf -H 10.0.0.114 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.0.114 (10.0.0.114) port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 10.02 229.65
## UDP STREAM PERFORMANCE
[root@netperfc ~]# /opt/netperf-2.4.5/bin/netperf -H 10.0.0.114 -t UDP_STREAM -- -m 1024
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.0.114 (10.0.0.114) port 0 AF_INET
Socket Message Elapsed Messages
Size Size Time Okay Errors Throughput
bytes bytes secs # # 10^6bits/sec
129024 1024 10.00 374922 0 307.10
129024 10.00 90230 73.91
## TCP Request/Response Performance
[root@netperfc ~]# /opt/netperf-2.4.5/bin/netperf -H 10.0.0.114 -t TCP_RR
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.0.114 (10.0.0.114) port 0 AF_INET
Local /Remote
Socket Size Request Resp. Elapsed Trans.
Send Recv Size Size Time Rate
bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 10.00 2920.92
16384 87380
## UDP Request/Response Performance
[root@netperfc ~]# /opt/netperf-2.4.5/bin/netperf -H 10.0.0.114 -t UDP_RR
UDP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.0.114 (10.0.0.114) port 0 AF_INET
Local /Remote
Socket Size Request Resp. Elapsed Trans.
Send Recv Size Size Time Rate
bytes Bytes bytes bytes secs. per sec
129024 129024 1 1 10.00 2649.07
129024 129024
tune and rerun test
calculate bdp
[root@netperfc ~]# ping -c 10 10.0.0.114
PING 10.0.0.114 (10.0.0.114) 56(84) bytes of data.
64 bytes from 10.0.0.114: icmp_seq=1 ttl=64 time=0.201 ms
64 bytes from 10.0.0.114: icmp_seq=2 ttl=64 time=1.20 ms
64 bytes from 10.0.0.114: icmp_seq=3 ttl=64 time=0.771 ms
64 bytes from 10.0.0.114: icmp_seq=4 ttl=64 time=0.279 ms
64 bytes from 10.0.0.114: icmp_seq=5 ttl=64 time=0.957 ms
64 bytes from 10.0.0.114: icmp_seq=6 ttl=64 time=2.16 ms
64 bytes from 10.0.0.114: icmp_seq=7 ttl=64 time=0.814 ms
64 bytes from 10.0.0.114: icmp_seq=8 ttl=64 time=1.02 ms
64 bytes from 10.0.0.114: icmp_seq=9 ttl=64 time=0.883 ms
64 bytes from 10.0.0.114: icmp_seq=10 ttl=64 time=3.23 ms
--- 10.0.0.114 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9001ms
rtt min/avg/max/mdev = 0.201/1.153/3.234/0.860 ms
# BDP calculation
100 Mb | 0.001153 | Byte | 2^20
BDP = -------------+-----------+--------+------------ = 15112.6016 Bytes
s | Mebi | 8 bits |
calculate socket buffer
[root@netperfc ~]# lsof -i | wc -l
17
tune windows
run generic netperf test
Apache
[root@netperfc ~]# time wget http://10.0.0.114/bdp/bigfile
--2010-10-06 02:09:29-- http://10.0.0.114/bdp/bigfile
Connecting to 10.0.0.114:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [text/plain]
Saving to: `bigfile'
100%[===========================================================================================================] 104,857,600 27.6M/s in 3.8s
2010-10-06 02:09:33 (26.2 MB/s) - `bigfile' saved [104857600/104857600]
real 0m3.838s
user 0m0.051s
sys 0m2.858s
[root@netperfc ~]# cat /proc/sys/net/ipv4/tcp_rmem
4096 87380 1048576
[root@netperfc ~]# cat /proc/sys/net/core/rmem_max
129024
[root@netperfc ~]# echo '15113 15113 15113' /proc/sys/net/ipv4/tcp_rmem
[root@netperfc ~]# echo '15113' /proc/sys/net/core/rmem_max
[root@netperfc ~]# time wget http://10.0.0.114/bdp/bigfile
--2010-10-06 02:12:03-- http://10.0.0.114/bdp/bigfile
Connecting to 10.0.0.114:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [text/plain]
Saving to: `bigfile.1'
100%[===========================================================================================================] 104,857,600 14.3M/s in 7.4s
2010-10-06 02:12:11 (13.6 MB/s) - `bigfile.1' saved [104857600/104857600]
real 0m7.399s
user 0m0.164s
sys 0m3.939s
[root@netperfc ~]# echo 258048 /proc/sys/net/core/rmem_max
[root@netperfc ~]# time wget http://10.0.0.114/bdp/bigfile
--2010-10-06 02:19:26-- http://10.0.0.114/bdp/bigfile
Connecting to 10.0.0.114:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [text/plain]
Saving to: `bigfile'
100%[===========================================================================================================] 104,857,600 27.6M/s in 3.6s
2010-10-06 02:19:29 (27.4 MB/s) - `bigfile' saved [104857600/104857600]
real 0m3.664s
user 0m0.042s
sys 0m2.951s
[root@netperfc ~]# echo 387072 /proc/sys/net/core/rmem_max
[root@netperfc ~]# time wget http://10.0.0.114/bdp/bigfile
--2010-10-06 02:23:03-- http://10.0.0.114/bdp/bigfile
Connecting to 10.0.0.114:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [text/plain]
Saving to: `bigfile.1'
100%[===========================================================================================================] 104,857,600 26.6M/s in 3.9s
2010-10-06 02:23:07 (25.9 MB/s) - `bigfile.1' saved [104857600/104857600]
real 0m3.882s
user 0m0.061s
sys 0m3.151s
[root@netperfc ~]# echo 258048 /proc/sys/net/core/rmem_max
[root@netperfc ~]# echo '258048 258048 258048' /proc/sys/net/ipv4/tcp_rmem
[root@netperfc ~]# time wget http://10.0.0.114/bdp/bigfile
--2010-10-06 02:24:35-- http://10.0.0.114/bdp/bigfile
Connecting to 10.0.0.114:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [text/plain]
Saving to: `bigfile.2'
100%[===========================================================================================================] 104,857,600 23.9M/s in 4.3s
2010-10-06 02:24:39 (23.2 MB/s) - `bigfile.2' saved [104857600/104857600]
real 0m4.321s
user 0m0.056s
sys 0m3.053s
[root@netperfc ~]# echo 0 /proc/sys/net/ipv4/tcp_window_scaling
[root@netperfc ~]# time wget http://10.0.0.114/bdp/bigfile
--2010-10-06 02:26:27-- http://10.0.0.114/bdp/bigfile
Connecting to 10.0.0.114:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [text/plain]
Saving to: `bigfile.3'
100%[===========================================================================================================] 104,857,600 19.4M/s in 5.0s
2010-10-06 02:26:32 (20.2 MB/s) - `bigfile.3' saved [104857600/104857600]
real 0m4.965s
user 0m0.068s
sys 0m3.231s
[root@netperfc ~]# echo 1 /proc/sys/net/ipv4/tcp_window_scaling
[root@netperfc ~]# time wget http://10.0.0.114/bdp/bigfile
--2010-10-06 02:26:56-- http://10.0.0.114/bdp/bigfile
Connecting to 10.0.0.114:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [text/plain]
Saving to: `bigfile.4'
100%[===========================================================================================================] 104,857,600 22.3M/s in 4.4s
2010-10-06 02:27:00 (22.6 MB/s) - `bigfile.4' saved [104857600/104857600]
real 0m4.439s
user 0m0.071s
sys 0m2.969s
[root@netperfc ~]# echo 1 /proc/sys/net/ipv4/tcp_window_scaling
[root@netperfc ~]# echo '120904 120904 120904' /proc/sys/net/ipv4/tcp_rmem
[root@netperfc ~]# echo 120904 /proc/sys/net/core/rmem_max
[root@netperfc ~]# time wget http://10.0.0.114/bdp/bigfile
--2010-10-06 02:30:51-- http://10.0.0.114/bdp/bigfile
Connecting to 10.0.0.114:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [text/plain]
Saving to: `bigfile.5'
100%[===========================================================================================================] 104,857,600 23.3M/s in 4.3s
2010-10-06 02:30:56 (23.5 MB/s) - `bigfile.5' saved [104857600/104857600]
real 0m4.273s
user 0m0.056s
sys 0m2.896s
[root@netperfc ~]# echo 0 /proc/sys/net/ipv4/tcp_window_scaling
[root@netperfc ~]# time wget http://10.0.0.114/bdp/bigfile
--2010-10-06 02:31:57-- http://10.0.0.114/bdp/bigfile
Connecting to 10.0.0.114:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [text/plain]
Saving to: `bigfile.7'
100%[===========================================================================================================] 104,857,600 19.2M/s in 4.9s
2010-10-06 02:32:02 (20.6 MB/s) - `bigfile.7' saved [104857600/104857600]
real 0m4.862s
user 0m0.079s
sys 0m2.851s
[root@netperfc ~]# sysctl -a | fgrep scale
net.ipv4.tcp_adv_win_scale = 2
[root@netperfc ~]# sysctl -w net.ipv4.tcp_adv_win_scale=14
net.ipv4.tcp_adv_win_scale = 14
[root@netperfc ~]# time wget http://10.0.0.114/bdp/bigfile
--2010-10-06 02:44:08-- http://10.0.0.114/bdp/bigfile
Connecting to 10.0.0.114:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [text/plain]
Saving to: `bigfile.8'
100%[===========================================================================================================] 104,857,600 15.7M/s in 6.0s
2010-10-06 02:44:14 (16.6 MB/s) - `bigfile.8' saved [104857600/104857600]
real 0m6.038s
user 0m0.083s
sys 0m4.370s