Lighttpd

Is there a Wiki page that I'm missing that would provide guidance on how to migrate 1.4.55 plugins to 1.4.64?

Unfortunately not. We receive vanishingly little feedback about how people are using lighttpd. I really mean vanishingly little. Therefore, there is little information available about how much impact changing some internals might have versus changing other internals. I have seen very few open-source lighttpd modules (and I do not get to see the proprietary ones). Of the few open-source lighttpd modules I have seen, almost all have been very dated or terribly insecure (poorly written), and modern lighttpd releases have provided more secure and faster solutions. (I am writing only from my own experience as one of the lighttpd developers.)

lighttpd 1.4.56 adds support for HTTP/2 and some of the lighttpd internal concepts had to be extended and changed. Also, lighttpd performance improved in multiple ways, including configuration file processing. The plugin module boilerplate code which is run only at lighttpd startup became a bit more complex in order to be faster at runtime during request processing. You can review smaller modules such as src/mod_access.c for some simpler examples of how that code now looks.

If you care to share, what is your plugin doing?
Is there a specific reason it must be written as a lighttpd plugin?
What parts of your module are you having trouble porting to newer lighttpd versions? Are you able to share the code?

Can your customizations be written using mod_magnet with a few lines of lua code? Since lighttpd 1.4.60, lighttpd mod_magnet has been growing more powerful and is recommended for extending lighttpd. Using mod_magnet and lua is reasonably performant and is often safer than writing custom modules. Using the mod_magnet lua interfaces is generally more future-proof than coding against lighttpd internals.

Thanks for responding. Good to know that I didn't miss the new page somewhere.

As you guessed, my plug-in is a proprietary application and it uses a lot of lighttpd functionality to interface with a threaded high-performance backend, while keeping lighttpd as light as possible by disabling all of its modules and just using the HTTP framework that handles all requests, including chunked input, and responses.

I will need to investigate things like chunked input, which no longer has connection.request_content_queue and buffers/chunking seems to have changed as well, dropped HANDLER_WAIT_FOR_FD, which I use to synchronize back with the lighttpd thread when the threaded backend signals that it has the response, joblist_append dropping src, and so on, just to give a few examples.

It looks like it will be a fairly big exercise for me - last time I had to map out all lighttpd request processing calls to write the plug-in in question and was hoping that sticking to the public interface will protect me from massive changes. I guess will stay with v1.4.55 for now and when the time come will revisit the API that is available at that time.

Thanks for your work and insights. It's a fantastic web server.

I will need to investigate things like chunked input, which no longer has connection.request_content_queue and buffers/chunking seems to have changed as well, dropped HANDLER_WAIT_FOR_FD, which I use to synchronize back with the lighttpd thread when the threaded backend signals that it has the response, joblist_append dropping src, and so on, just to give a few examples.

Those are mostly still there, though some have been renamed. Changes were made to better split connection handling from request handling since HTTP/2 can have multiple active HTTP/2 streams handling requests in parallel on a single connection. For your requests handling, instead of con->request_content_queue, you want to look at r->reqbody_queue which will contain the demuxed request body for the stream (HTTP/2) or chunked-decoded request body (for HTTP/1.1 with Transfer-Encoding: chunked).

It sounds like you are misusing HANDLER_WAIT_FOR_FD, which was intended as a flag for running out of file descriptor resources. You probably should be using HANDLER_WAIT_FOR_EVENT instead.

As to joblist_append(), again, with HTTP/2 support that needed to change. Rescheduling the connection * in the joblist will cycle through the active HTTP/2 streams (or the active HTTP/1.1 request) attached to the connection.

Overall, I have a hunch that you need to modify your plugin to operate on request_st *, return HANDLER_WAIT_FOR_EVENT when waiting for your backend threads, and call joblist_append(r->con) when any backend event is ready.

It looks like it will be a fairly big exercise for me - last time I had to map out all lighttpd request processing calls to write the plug-in in question and was hoping that sticking to the public interface will protect me from massive changes.

I tried to make it clear in 2019 in https://redmine.lighttpd.net/boards/3/topics/8682 that lighttpd headers are not installed, so your comment about public interfaces was, is, and will continue to be little more than wishful thinking.

While there may be some performance improvement having additional threads within the lighttpd process, a somewhat lightweight backend communication protocol such as SCGI, FastCGI, AJP13, or even HTTP reverse proxy with mod_proxy -- and connected over a fast, local unix domain socket -- would isolate your backend from internal lighttpd changes. For high-performance applications with multiple data streams, HTTP/2 framing and flow-control can make a big difference in application performance (and your application currently does not support HTTP/2 if using lighttpd 1.4.55). The next version of lighttpd (lighttpd 1.4.65) adds support for Websockets over HTTP/2 and also adds support for the upcoming RFC standard for PRIORITY_UPDATE frames to manage HTTP/2 stream priorities (replacing the overly complex and resource hungry HTTP/2 PRIORITY frame management which lighttpd and many other servers chose not to implement).

It sounds like you are misusing HANDLER_WAIT_FOR_FD, which was intended as a flag for running out of file descriptor resources. You probably should be using HANDLER_WAIT_FOR_EVENT instead.

I use it as a response continuation signal and it resumes the request processing cycle when there's data for a descriptor I register. I don't have the code in front of me, but will post more tomorrow, just in case you'd be kind enough to comment on this particular use of the event and if you see a better way of doing it.

HANDLER_WAIT_FOR_EVENT didn't work for this at the time, as I recall, but maybe I'm misremembering. All I need is to resume a request, which awaits continuation in the lighttpd's event loop, when I have data ready in my threaded backend, and HANDLER_WAIT_FOR_FD is working well for this.

so your comment about public interfaces was, is, and will continue to be little more than wishful thinking.

I meant this in the programmatic sense - what's available via the plug-in interface and not relying on the code I fished out wading through the source. I'm aware of how it is distributed and maintained and I'm happy about it. I even tried to switch to the packaged version - CentOS 8 Streams has v1.4.55 available as a package, but it runs about 20-30% slower than the one I build from the source, so I'll keep using it as I am now.

While there may be some performance improvement having additional threads within the lighttpd process, a somewhat lightweight backend communication protocol ... connected over a fast, local unix domain socket -- would isolate your backend from internal lighttpd changes.

We are talking about different kind of fast. I squeeze out multiple 10-15 ms responses in parallel, all hitting the database. No lightweight protocol or shared memory will beat in-process plug-in API speed, all other things being equal. lighttpd is doing very well in managing HTTP parsing/writing and I/O, so as long as the plug-in API is available, I'll try to adapt to it.

I use it as a response continuation signal and it resumes the request processing cycle when there's data for a descriptor I register. I don't have the code in front of me, but will post more tomorrow, just in case you'd be kind enough to comment on this particular use of the event and if you see a better way of doing it.

As I said above, you should return HANDLER_WAIT_FOR_EVENT when waiting for a backend event, and should joblist_append(r->con) when the backend fd has an event for the associated r.

To try to put it another way: if you register your own fd and fdevent handlers with the lighttpd event loop, then when you create that fd, you should keep track of which connection(s) that fd is serving, and should joblist_append() those connection(s) when there is data on your fd that you want those connection(s) to process. This pattern is used by many of lighttpd's dynamic backends. When a backend fd returns data, the registered fd event handler in lighttpd calls the equivalent of joblist_append(r->con) for the associated r. Then, lighttpd can read/processes that data and write into the associated r->write_queue, or wait for the event loop to process r to then read from the associated backend fd.

Most dynamic backend modules/plugins keep track of associated r and con in a handler_ctx hctx specific to each module/plugin. Most dynamic backends reuse the code in gw_backend.[ch] ("gateway backend") to manage the backend socket events.

As I mentioned above, I think you need to modify your plugin to operate on request_st * instead of connection *. request_st * was split out of connection * in lighttpd 1.4.56 for HTTP/2 support.

We are talking about different kind of fast. I squeeze out multiple 10-15 ms responses in parallel, all hitting the database. No lightweight protocol or shared memory will beat in-process plug-in API speed, all other things being equal. lighttpd is doing very well in managing HTTP parsing/writing and I/O, so as long as the plug-in API is available, I'll try to adapt to it.

I am glad to hear that you have measurements, and so this is not theoretical to you. I do not have your inside numbers and I have no insight into your actual bottlenecks, so my response is general and may not apply to your use cases: lighttpd proxying over a unix domain socket to another lighttpd instance can serve static requests more than 100x faster than 10-15ms, so I would not immediately dismiss lightweight protocols used over local unix domain sockets unless you have tried them and they do not meet your requirements. e.g. pumping large amounts of data over unix domain sockets will, of course, be slower than avoiding excess copying by accessing the data in the same process.

CentOS 8 Streams has v1.4.55 available as a package, but it runs about 20-30% slower than the one I build from the source

I am not surprised. Would you share the compiler flags that you use? A sample is fine. -O2? -Os? etc.

I do use the API similar to how you describe it, only using the API for my current version. I register a handler with fdevent_register and fdevent_fdnode_event_add and call joblist_append in the callback.

I misspoke about HANDLER_WAIT_FOR_FD and had to look at the code to confirm. It is indeed returned from the URI handler when pipe2, which I use to communicate with the backend, runs out of descriptors, so with this code lighttpd calls it again when more descriptors are available.

so I would not immediately dismiss lightweight protocols used over local unix domain sockets

Not trying to minimize their capabilities, just saying that as fast as they are, nothing beats the speed of a direct function call, so all other things aside, and provided that there is no synchronization being performed, a direct call will always be faster than a local communication, no matter how fast it is.

Anyway, not trying to argue - apologies if it sounded as if I was. Just thinking aloud, really. Performance is high on the list of things for me on this project.

There's nothing special for my compiler flags - I just use -O3. Compiling with libev showed better performance in my initial tests, but the last preliminary test I ran against 1.4.55 shows that the default epol works just as fast, if not faster, so I need to run more thorough tests to finalize the event library.

I was a bit puzzled that the packaged lighttpd ran visibly slower - everything else was the same, configuration-wise. If I have time, I will run it in VTune and maybe it will shed some light on this. If I get anything out of it, I will let you know.

Thanks again for your insights. It's much appreciated.

If you're running out of fds, the recommended approach is to increase the ulimit -Hn for number of fds and configure large server.max-fds in lighttpd.conf, or else to tune the maximum number of lighttpd connections (server.max-connections) to be served down so that you do not run out of fds when under load.

Performance is high on the list of things for me on this project.

Real-world performance is end-to-end. Microbenchmarks can show one thing marginally faster than another, but in a real-world test with parallel load, the opposite may be true. lighttpd using libev using epoll on Linux should not be faster than lighttpd using epoll directly, without the unnecessary libev layer, so this sounds like an issue with your microbenchmark. On a system where most fds are active a large percentage of the time, lighttpd using poll can be faster than lighttpd using epoll, but such load on a large number of connections is true more often in microbenchmarks than it is in practice. (For a small number of connections, the differences between the polling mechanisms on actual response times is negligible.)

Similarly, I measure lighttpd performance in microseconds, so your app taking 10's of milliseconds suggests to me that local unix domain socket communication might not be the biggest bottleneck unless you are pushing large amounts of data.

The point I am trying to make is that if local unix domain socket communication is not the biggest bottleneck, then running your application separately from lighttpd will be much easier to maintain than building your application into lighttpd. The communication overhead for running a separate application needs to be measured for your application to determine whether or not it is a reasonable configuration for you.

Keeping a connection pool of open database connections can also be a big win in a database-backed application, if the connection pool is managed properly.

There's nothing special for my compiler flags - I just use -O3. Compiling with libev showed better performance in my initial tests, but the last preliminary test I ran against 1.4.55 shows that the default epol works just as fast, if not faster, so I need to run more thorough tests to finalize the event library.

I was a bit puzzled that the packaged lighttpd ran visibly slower - everything else was the same, configuration-wise. If I have time, I will run it in VTune and maybe it will shed some light on this. If I get anything out of it, I will let you know.

Most distros use the default of -O2. Some embedded systems use -Os. Most distros use gcc or clang.
If you're using icc and -O3, either might help explain the performance differences.

Sounds good. Thanks for your thoughts.