Project

General

Profile

Bug #2518

Loss of performance (x2)

Added by Pierre2013 about 4 years ago. Updated almost 2 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Start date:
2013-10-07
Due date:
% Done:

0%

Estimated time:

Description

We have used weighttp over the years since 2011, in thousands of benchmarks for many Web servers.

We rely on weighttp to detect and correct problems so we value it enormously.

Recently, I have installed the latest version of weighttp on a new machine and to my dismay it was much slower than on an older machine with the same specs.

I have copied the old weighttp on the new machine (and vice-versa) without any change in the results, reaching the conclusion that weighttp was not the problem.

The problem is rather likely to come from the libev dependency.

This is raising the question of weighttp's relevance if libev goes nuts (which it seems to have done since 2011).

I know it will be a pain to cut the dependency with libev but this may be the price to pay for weighttp to stay relevant.

Sincerely,

Pierre.

History

#1

Updated by Pierre2013 about 4 years ago

Hmmm... I checked the libev version with 'ldd weighttp' and libev.so.4 is used on both machines:

UBUNTU 10.10 (2x faster):
linux-vdso.so.1 => (0x00007fffb5bff000)
libev.so.4 => /usr/local/lib/libev.so.4 (0x00007f384d8ed000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007f384d6d0000)
libc.so.6 => /lib/libc.so.6 (0x00007f384d34c000)
libm.so.6 => /lib/libm.so.6 (0x00007f384d0c9000)
/lib64/ld-linux-x86-64.so.2 (0x00007f384db17000)

UBUNTU 12.04 (2x slower):
linux-vdso.so.1 => (0x00007fff465ff000)
libev.so.4 => /usr/lib/libev.so.4 (0x00007f66ad978000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f66ad75b000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f66ad39b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f66ad09f000)
/lib64/ld-linux-x86-64.so.2 (0x00007f66adbb5000)

What, in your experience, could be lead to so different results?

#2

Updated by Pierre2013 about 4 years ago

If that's not libev, is it the OS (kernel, libc)?

#3

Updated by darix about 4 years ago

on 10.10 you have manually installed libev to /usr/local while 12.04 you use the system copy.

#4

Updated by Pierre2013 about 4 years ago

Can you explain your idea further?

On the 10.10 OS:
ll /usr/local/lib/libev.so.4
lrwxrwxrwx 1 root root 14 Apr 3 2012 /usr/local/lib/libev.so.4 -> libev.so.4.0.0

On the 12.04 OS:
ll /usr/lib/libev.so.4
lrwxrwxrwx 1 root root 14 Apr 3 2012 /usr/lib/libev.so.4 -> libev.so.4.0.0

Is the detrimental effect on weighttp caused by an higher shared library loading time?

Could it be that much slower for the dynamic linker to resolve the shared library at startup due to the larger number of libraries under /usr/lib (as compared to /usr/local/lib)?

If so, why would it have any effect during a long benchmark?
(I mean, once the library has been loaded a long time ago)

#5

Updated by darix about 4 years ago

  1. you need to find out which libev version you have on your 10.10
  2. copy your libev.so.* from your 10.10 /usr/local/lib to the 12.04 /usr/local/lib and see if that fixes your problems. verify with ldd that it will use that version. just because you run different libev versions doesnt mean that is the problem.
  3. maybe check the ubuntu source archive if they have patches that might cause the slow down.
#6

Updated by Pierre2013 about 4 years ago

These are quite different files, not sure which version is used by Ubuntu 12:

UNBUNTU 10.10 (libev 4.04 - found source code folder):

ll /usr/local/lib/libev.*
-rw-r--r-- 1 root root 240562 2011-10-05 06:32 /usr/local/lib/libev.a
-rwxr-xr-x 1 root root    926 2011-10-05 06:32 /usr/local/lib/libev.la*
lrwxrwxrwx 1 root root     14 2011-10-05 06:32 /usr/local/lib/libev.so -> libev.so.4.0.0*
lrwxrwxrwx 1 root root     14 2011-10-05 06:32 /usr/local/lib/libev.so.4 -> libev.so.4.0.0*
-rwxr-xr-x 1 root root 173681 2011-10-05 06:32 /usr/local/lib/libev.so.4.0.0*


UBUNTU 12.04 (unknown libev version fetched with apt-get install):
ll /usr/lib/libev.*
-rw-r--r-- 1 root root 67706 Apr  3  2012 /usr/lib/libev.a
lrwxrwxrwx 1 root root    14 Apr  3  2012 /usr/lib/libev.so -> libev.so.4.0.0
lrwxrwxrwx 1 root root    14 Sep 19  2012 /usr/lib/libev.so.3 -> libev.so.4.0.0
-rw-r--r-- 1 root root 51512 Apr  3  2012 /usr/lib/libev.so.3.0.0
lrwxrwxrwx 1 root root    14 Apr  3  2012 /usr/lib/libev.so.4 -> libev.so.4.0.0
-rw-r--r-- 1 root root 51512 Apr  3  2012 /usr/lib/libev.so.4.0.0

Note that I just did an apt-get libev remove/install on Ubuntu 12.04 and the version I was using is (the same and) the latest version available in the repository.

But while running weighttp on Ubuntu 12.04 with the 173KB libev:

lsof -p 12826
weighttp 12826 -  mem  REG  8,33 173681 32769599 /usr/lib/libev.so.4

weighttp is still 2x slower than on Ubuntu 10.10...

Any (other) idea?

#7

Updated by darix about 4 years ago

so it wasnt libev's fault. are those machines the same hardware? especially the same network cards?

you will have to do some more debugging to find where it is slow.

#8

Updated by Pierre2013 about 4 years ago

These are 2 Mac Pro units with the same specs:

CPU: 1 x 6-Core CPU Intel(R) Xeon(R) CPU W3680 @ 3.33GHz
RAM: 8 GB
NIC: Intel Corporation 82574L Gigabit (2 of them)
NIC: Broadcom Corporation BCM4322 802.11a/b/g/n Wireless LAN

The only difference is the OS version (Ubuntu 12.04 machines are connected to the Internet, Ubuntu 10.10 are setup and never updated).

Having excluded weighttp and libev Ubuntu 12.04 remains the next candidate for the 2x slow-down.

Anybody having seen that already?

Do you have recommandations for a specific Linux distribution and version for weighttp benchmarks?

#9

Updated by stbuehler about 4 years ago

Given that you installed stuff in /usr/local your setup is not clean (and maybe you sometimes installed self compiled stuff into /usr?). How can we know what you changed apart from that? custom/other kernel (apparmor, selinux?...), other custom libs, ...

Measure stuff to find the real reason for the slowdown - run libev benchmarks, network benchmarks, ...

Is weighttp using more CPU time (/usr/bin/time!) or does it just wait more?

#10

Updated by Pierre2013 about 4 years ago

- "Given that you installed stuff in /usr/local..."
- "How do we know what you changed, (apparmor, selinux?...), custom libs"

The manually-installed libraries are on the well-performing machine so that's not the source of the problem.

And the slow machine has installed everything from the repositories (since it is connected to the Internet).

Both do not use any OS-based firewall, apparmor, selinux, etc.

- "run libev benchmarks, network benchmarks"
- "Is weighttp using more CPU time (/usr/bin/time!) or does it just wait more?"

On Ubuntu 12.04 the Web servers are slowed-down by using more CPU time to process less data.

weighttp is likely to suffer from the same symptoms.

Testing Ubuntu 12.04 with a new fresh installation will tell if that's any of the installed packages - or if that's Ubuntu 12.04 causing this slow-down.

Thanks anyway for your time and suggestions.

#11

Updated by gstrauss almost 2 years ago

I have an experimental branch at https://github.com/gstrauss/weighttp/tree/rewrite which is a nearly complete rewrite of weighttp. Among other things, libev has been replaced with the very portable, traditional poll(). For most usage other than a huge number of concurrent connections, traditional poll() should perform quite well.

Also available in: Atom