4 Netbench is a terrible benchmark, but it's an "industry standard" and
5 it's what is used in the press to rate windows fileservers like Samba
8 The big problem with netbench for the open source community is that
9 very few people who work on open source software have the facilities
10 to run it properly. You need a lab with between 60 and 150 windows PC,
11 all on switched fast ethernet and a really grunty server (say a quad
12 xeon with 1GB ram and hardware raid). Then you need some way to nurse
13 all those machines along so they will run a very fussy benchmark suite
14 without crashing. Not easy and very expensive. Only one person in the
15 open source community that I know of has access to such a lab (Jeremy
18 In order for the development methodologies of the open source
19 community to work we need to be able to run this benchmark in an
20 environment that a bunch of us have access to. We need the source to
21 the benchmark so we can see what it does. We need to be able to split
22 it into pieces to look for individual bottlenecks. In short, we need
23 to open up netbench to the masses.
25 To do this I have written three tools, dbench, tbench and
26 smbtorture. All three read a load description file called client.txt
27 that was derived from a network sniffer dump of a real netbench
28 run. client.txt is about 4MB and describes the 90 thousand operations
29 that a netbench client does in a typical netbench run. They parse
30 client.txt and use it to produce the same load without having to buy a
31 huge lab. They can simulate any number of simultaneous clients.
33 client.txt must either be in the working directory, or specified on
34 the command line with the -c option.
39 dbench produces only the filesystem load. It does all the same IO
40 calls that the smbd server in Samba would produce when confronted with
41 a netbench run. It does no networking calls.
43 You can get dbench from ftp://samba.org/pub/tridge/dbench/
44 You run it as "dbench N" where N is the number of clients to
45 simulate. It gives out three numbers like this (this is from a 144
46 client run on a quad xeon box):
48 Throughput 40.6701 MB/sec (NB=50.8376 MB/sec 406.701 MBit/sec)
50 the first is the true throughput as seen by dbench. The second and
51 third numbers are "netbench scaled" numbers that give the throughput
52 that would be seen by Win9X boxes after taking into account the client
53 file cacheing performed by oplocks. They are given in both MB/sec and
54 MBit/sec as different netbench reports use different scales.
59 tbench produces only the TCP and process load. It does the same socket
60 calls that smbd would do under a netbench load. It does no filesystem
61 calls. The idea behind tbench is to eliminate smbd from the netbench
62 test, as though the smbd code could be made infinately fast. The
63 throughput results of tbench tell us how fast a netbench run could go
64 if we eliminated all filesystem IO and SMB packet processing. tbench
65 is built as part of the dbench package.
67 To run tbench first run tbench_srv on the server. Then run "tbench N SERVER"
68 on the client. N is the number of clients to simulate and SERVER is
69 the hostname of the server. The results are given in the same format
75 smbtorture is the stress tester from the Samba suite. I've recently
76 added a stress test that allows you to simulate the full netbench
77 benchmark, including all network traffic and filesystem IO.
79 To run smbtorture you first need to install Samba version 2.0.X. The
80 binary distrubtions at ftp://ftp.samba.org/pub/samba/bin-pkgs/redhat/
81 would do fine if you don't want to compile it yourself. Then setup a
82 netbench share on the fastest disk you have, making sure you have at
83 least 25MB free per simulated client. The simplest smb.conf would look
94 Then you need smbtorture. You can either grab a precompiled i386
95 Redhat5.2 binary from ftp://samba.org/pub/tridge/dbench/smbtorture.gz
96 or you can follow the instructions at http://samba.org/cvs.html to
97 download the SAMBA_2_0 branch of the Samba cvs tree and use
98 "make smbtorture" to build it.
100 Finally, you'll need client.txt from
101 ftp://samba.org/pub/tridge/dbench/dbench.tgz in the same directory
102 that you run smbtorture from.
104 To run it you do this:
106 smbtorture //localhost/netbench -U% -N 32 NBW95
108 that will run a 32 client load. You can, of course, also run
109 smbtorture against a dfferent SMB server (such as a NT server) to give
110 comparitive results for any client load that the server can handle.
112 Even better is to run smbtorture on one machine and smbd on another,
113 connected by a very fast network (such as gigabit ethernet). That will
114 stop the smbtorture code itself from loading the server and will also
115 test the network driver instead of the loopback driver.
120 To give you an idea of what to expect here are some results on a quad
121 xeon machine running Linux 2.2.9 with 1GB of memory (it has 4GB but
122 Linux only uses 1GB by default). The machine also has a 45GB hardware
123 raid system and a Alteon AceNIC gigabit network card.
125 The results below are in netbench MB/sec (the NB= number in the
126 result). Multiply by 8 to get the MBit/sec numbers that Mindcraft used
127 in their graphs. The first column is the number of simulated clients,
128 the second is the result.
141 With tbench on loopback I get:
149 With tbench running across the gigabit network (using a dual
150 processor Origin200 as the client) I get:
158 With smbtorture running over loopback I get:
166 With smbtorture running across the gigabit network (using a dual
167 processor Origin200 as the client) I get:
175 With smbtorture running across the gigabit network but with smbd
176 modified so that write_file() and read_file() are null operations
177 (which eliminates the file IO) I get:
186 The above results show that, at least for this hardware configuration,
187 the problem isn't the filesystem code or the raid drivers. More tests
188 will be needed to find out exactly what the problem is but it looks
189 like a TCP scaling problem.
194 Hopefully Jeremy will be able to run smbtorture against NT on the same
195 hardwre sometime in the next week so we have direct numbers for
196 comparison, but from looking at the mindcraft numbers under netbench
197 we would expect NT to get about the following:
205 so we do well by comparison with small client loads but fall behind
206 quite a lot with large loads. Note that the numbers in the mindcraft
207 report for Linux/Samba are quite a long way behind what I give above
208 because mindcraft did a hopeless job of tuning Linux/Samba.
210 comparison with netbench
211 ------------------------
213 An initial comparison with real netbench results shows that smbtorture
214 does produce very similar throughput numbers. They aren't exactly the
215 same but they are similar enough for us to target our tuning efforts
216 and expect to see improvements reflected in real netbench runs. When
217 we find something that looks promising we can get Jeremy to run a real
224 the tbench results really pointed at the problem being the Linux TCP
225 stack. I made a quick (and very unsafe!) hack to Samba and the Linux
226 kernel to see if I could remove the lock_kernel() in sys_sendto() and
227 sys_recvfrom() for smbd processes by passing a MSG_NOLOCK flag in
228 send() and recv(). That gave an enormous improvement in the loopback
237 and in the loopback smbtorture results I also saw a big improvement:
245 that's a 50% improvement. I suspect the numbers will be higher with a
246 real netbench run as it won't have the overhead of running 80
247 smbtorture clients on the same box as the server.
253 One question some people may ask is whether the above represents a
254 realistic load on a fileserver. It doesn't. Nearly 90% of the
255 read/write IO operations in netbench are writes whereas in a "normal"
256 office load reads dominate. Also, the load is *much* higher than a
257 normal office PC would put on a server. There aren't many office PCs
258 that write over a hundred megabytes of data to a server in a few
259 minutes, unless maybe they are copying a CD.
261 That doesn't mean the benchmark is useless, it just means you
262 shouldn't use this for purchasing decisions unless you really
263 understand the results and how they relate to your environment.
269 smbtorture and dbench are released under the terms of the GNU Public
270 License version 3 or later.
git.samba.org / tridge / dbench.git / blob