utilities » weighttp

Hmmm... I checked the libev version with 'ldd weighttp' and libev.so.4 is used on both machines:

UBUNTU 10.10 (2x faster):
linux-vdso.so.1 => (0x00007fffb5bff000)
libev.so.4 => /usr/local/lib/libev.so.4 (0x00007f384d8ed000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007f384d6d0000)
libc.so.6 => /lib/libc.so.6 (0x00007f384d34c000)
libm.so.6 => /lib/libm.so.6 (0x00007f384d0c9000)
/lib64/ld-linux-x86-64.so.2 (0x00007f384db17000)

UBUNTU 12.04 (2x slower):
linux-vdso.so.1 => (0x00007fff465ff000)
libev.so.4 => /usr/lib/libev.so.4 (0x00007f66ad978000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f66ad75b000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f66ad39b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f66ad09f000)
/lib64/ld-linux-x86-64.so.2 (0x00007f66adbb5000)

What, in your experience, could be lead to so different results?

If that's not libev, is it the OS (kernel, libc)?

on 10.10 you have manually installed libev to /usr/local while 12.04 you use the system copy.

Can you explain your idea further?

On the 10.10 OS:
ll /usr/local/lib/libev.so.4
lrwxrwxrwx 1 root root 14 Apr 3 2012 /usr/local/lib/libev.so.4 -> libev.so.4.0.0

On the 12.04 OS:
ll /usr/lib/libev.so.4
lrwxrwxrwx 1 root root 14 Apr 3 2012 /usr/lib/libev.so.4 -> libev.so.4.0.0

Is the detrimental effect on weighttp caused by an higher shared library loading time?

Could it be that much slower for the dynamic linker to resolve the shared library at startup due to the larger number of libraries under /usr/lib (as compared to /usr/local/lib)?

If so, why would it have any effect during a long benchmark?
(I mean, once the library has been loaded a long time ago)

1. you need to find out which libev version you have on your 10.10
2. copy your libev.so.* from your 10.10 /usr/local/lib to the 12.04 /usr/local/lib and see if that fixes your problems. verify with ldd that it will use that version. just because you run different libev versions doesnt mean that is the problem.
3. maybe check the ubuntu source archive if they have patches that might cause the slow down.

These are quite different files, not sure which version is used by Ubuntu 12:

UNBUNTU 10.10 (libev 4.04 - found source code folder):

ll /usr/local/lib/libev.*
-rw-r--r-- 1 root root 240562 2011-10-05 06:32 /usr/local/lib/libev.a
-rwxr-xr-x 1 root root    926 2011-10-05 06:32 /usr/local/lib/libev.la*
lrwxrwxrwx 1 root root     14 2011-10-05 06:32 /usr/local/lib/libev.so -> libev.so.4.0.0*
lrwxrwxrwx 1 root root     14 2011-10-05 06:32 /usr/local/lib/libev.so.4 -> libev.so.4.0.0*
-rwxr-xr-x 1 root root 173681 2011-10-05 06:32 /usr/local/lib/libev.so.4.0.0*

ll /usr/lib/libev.*
-rw-r--r-- 1 root root 67706 Apr  3  2012 /usr/lib/libev.a
lrwxrwxrwx 1 root root    14 Apr  3  2012 /usr/lib/libev.so -> libev.so.4.0.0
lrwxrwxrwx 1 root root    14 Sep 19  2012 /usr/lib/libev.so.3 -> libev.so.4.0.0
-rw-r--r-- 1 root root 51512 Apr  3  2012 /usr/lib/libev.so.3.0.0
lrwxrwxrwx 1 root root    14 Apr  3  2012 /usr/lib/libev.so.4 -> libev.so.4.0.0
-rw-r--r-- 1 root root 51512 Apr  3  2012 /usr/lib/libev.so.4.0.0

Note that I just did an apt-get libev remove/install on Ubuntu 12.04 and the version I was using is (the same and) the latest version available in the repository.

But while running weighttp on Ubuntu 12.04 with the 173KB libev:

lsof -p 12826
weighttp 12826 -  mem  REG  8,33 173681 32769599 /usr/lib/libev.so.4

weighttp is still 2x slower than on Ubuntu 10.10...

Any (other) idea?

so it wasnt libev's fault. are those machines the same hardware? especially the same network cards?

you will have to do some more debugging to find where it is slow.

These are 2 Mac Pro units with the same specs:

CPU: 1 x 6-Core CPU Intel(R) Xeon(R) CPU W3680 @ 3.33GHz
RAM: 8 GB
NIC: Intel Corporation 82574L Gigabit (2 of them)
NIC: Broadcom Corporation BCM4322 802.11a/b/g/n Wireless LAN

The only difference is the OS version (Ubuntu 12.04 machines are connected to the Internet, Ubuntu 10.10 are setup and never updated).

Having excluded weighttp and libev Ubuntu 12.04 remains the next candidate for the 2x slow-down.

Anybody having seen that already?

Do you have recommandations for a specific Linux distribution and version for weighttp benchmarks?

Given that you installed stuff in /usr/local your setup is not clean (and maybe you sometimes installed self compiled stuff into /usr?). How can we know what you changed apart from that? custom/other kernel (apparmor, selinux?...), other custom libs, ...

Measure stuff to find the real reason for the slowdown - run libev benchmarks, network benchmarks, ...

Is weighttp using more CPU time (/usr/bin/time!) or does it just wait more?

- "Given that you installed stuff in /usr/local..."
- "How do we know what you changed, (apparmor, selinux?...), custom libs"

The manually-installed libraries are on the well-performing machine so that's not the source of the problem.

And the slow machine has installed everything from the repositories (since it is connected to the Internet).

Both do not use any OS-based firewall, apparmor, selinux, etc.

- "run libev benchmarks, network benchmarks"
- "Is weighttp using more CPU time (/usr/bin/time!) or does it just wait more?"

On Ubuntu 12.04 the Web servers are slowed-down by using more CPU time to process less data.

weighttp is likely to suffer from the same symptoms.

Testing Ubuntu 12.04 with a new fresh installation will tell if that's any of the installed packages - or if that's Ubuntu 12.04 causing this slow-down.

Thanks anyway for your time and suggestions.

I have an experimental branch at https://github.com/gstrauss/weighttp/tree/rewrite which is a nearly complete rewrite of weighttp. Among other things, libev has been replaced with the very portable, traditional poll(). For most usage other than a huge number of concurrent connections, traditional poll() should perform quite well.