pixelserv-tls

pixelserv-tls started as a toy project to understand OpenSSL API by forking the original pixelserv. It is still a hobby but luckily ends up being something useful at the same time.

pixelserv-tls inherits fundamental functionality from pixelserv that is enhancing browsing experience with faster empty advert responses. pixelserv-tls adds HTTPS and HTTP/1.1 support which on their own are major steps forward when compared its ancestor. The HTTPS feature to my understanding is so far unique on this planet. :)

Over the past two years, the internal of pixelserv-tls has been gradually upgraded to better technologies and hence have morphed into a very different state from its ancestor. For example, pixelserv-tls builds on top of pthread and implements persistent connection, a mandatory requirement by HTTP/1.1 specification. Both features boost performance and efficiency. See the benchmark.

For usage details, check out the wiki and in particular the FAQ.

Release 2.2.0 (2018-10-9)

Changes (since 2.1.2)

* NEW support TLS 1.3.
* NEW support 0-RTT in TLS 1.3.
* NEW TLS 1.3 requires OpenSSL version >= 1.1.1
* NEW TLS 1.3 support autodetected and enabled at compile time. No special configuration needed.
* NEW counters v13, v12, v10 - breakdown of slh & slc requests into TLS versions.
  # of requests in TLS 1.3, TLS 1.2 and TLS 1.0 respectively.
* NEW indicator of TLS 1.3 support on servstats page.
  `no_tls1_3` when compiled against OpenSSL <= v1.1.0.
  `tls1_3` when compiled against OpenSSL >= 1.1.1
* NEW counter ucb - bad certificates as reported by clients.
* NEW counter ush - shutdown by clients after ServerHello.
* NEW log "shutdown after ServerHello" on LEVEL 2.
* NEW log "Handshake failed: socket i/o error" on LEVEL 2.
* NEW log "Handshake failed: reached max retries" on LEVEL 2.
* CHANGED enhance uniqueness of serial numbers for generated certificates.
* CHANGED much faster logging to syslog.
* CHANGED relax memory pool restriction for better multithread performance 
* CHANGED increase number of retries in TLS handshakes.
* CHANGED improve accuracy in processing time of requests recovered from initially failed handshakes.
* FIXED various complaints from musl libc on Alpine Linux (contributed by JohnNilsson on Github; issue #16)
* FIXED clean up compile warnings seen on newer platforms
* FIXED replace deprecated OpenSSL APIs

New feature highlights

TLS 1.3 support

IETF officially published RFC 8446 in Aug, 2018. One major benefit for pixelserv-tls is faster speed. TLS 1.3 is one round-trip less than previous standards. Hence between server and client only two round-trips are needed to complete one request. This particularly benefits deployment of pixelserv-tls over WAN or other slow tunnels.

0-RTT support

0-RTT further reduces the round-trip between server and client down to one. A client with 0-RTT support sends request in the very first packet of the TLS handshake process. pixelserv-tls will service such 0-RTT requests while completing the handshake at the same time.

TLS 1.3 support status on servstats page

When pixelserv-tls is compiled against OpenSSL >= 1.1.1, TLS 1.3 support is automatically detected and enabled at compile time. You should see tls1_3 on the servstats page of the running binary. When pixelserv-tls is compiled against OpenSSL <= 1.1.0, no TLS 1.3 support will be enabled at compile time. You'll see no_tls1_3 on the servstats page of such running binaries.

No special compile-time config is needed to enable TLS 1.3 support. One example that indicates a pixelserv-tls binary supporting TLS 1.3:

pixelserv-tls 2.2.0-rc.4 (compiled: Sep 23 2018 19:12:30 flags: tls1_3) options: 192.168.1.3 -A 344 -l 2 -c 350

slh & slc breakdown into TLS versions

Successful HTTPS requests are accounted for the TLS versions they used to communicate. The new counters v13, v12, v10 represent TLS v1.3, v1.2 and v1.0 respectively.

ucb, ush breakdown from slu

Two new categories for slu breakdown. ucb accounts for a client reports the server certificate is considered "bad certificate" by the client. ush accounts for a client abruptly shutds down a connection in the middle of TLS handshake (right after receiving ServerHello from pixelserv-tls). Both are considered tactics used by apps/systems to validate "authenticity" of the server i.e. if the server looks like not their own, stop sending any secret data.

Purge old generated certificates

If you upgrade from pixelserv-tls 2.1.2 or older to this release for the very first time, please purge all certificates except ca.crt and ca.key in CERT_PATH. New generated certificates will pick up one enhancement made in 2.2.0, and avoid possible clashes on serial numbers with old certificates.

Download

Regular pixelserv-tls (support TLS versions <= 1.2)

sh -c "$(wget -qO - https://kazoo.ga/pixelserv-tls/install-beta.sh)"

Statically linked pixelserv-tls (support TLS 1.3 and versions <= 1.2)

_binfavor=static sh -c "$(wget -qO - https://kazoo.ga/pixelserv-tls/install-beta.sh)"

2.2.0-rc.6 (2018-10-5)

Changes

  • NEW counters v13, v12, v10
  • CHANGE excluded compiler generated debug info from binaries

slh/slc breakdown on servstats page

Successful HTTPS requests are accounted for the TLS versions they used to communicate. The new counters v13, v12, v10 represent TLS v1.3, v1.2 and v1.0 respectively. pixelserv-tls is designed to support only these three versions.

2.2.0-rc.5 (2018-9-30)

Changes

  • NEW counter ucb that indicates a bad certificate reported by clients
  • CHANGE improvement in interoperability between 0rtt and non-0rtt clients

New ucb counter

A new slu break-down that indicates "bad certificate" as reported by clients. Exact reason for a certificate considered bad is not yet known. One common source of such error is Instagram client connecting to graph.instagram.com. It's believed to be one of the tactics Instagram verifies their server's real identity.

Improvement in 0rtt interoperability

We received a couple of reports of a hung process. For people getting the "hang", it's quite easy to reproduce but yet 100% reproducible case is rare. The pixelserv-tls process is actually not hung but take a varying amount of time getting itself out of a stuck state inside OpenSSL.

This version improves the handling of 0rtt and non-0rtt connections from client. The process will never get into a stuck state as seen in the previous build.

To work out this bug, we received lots of help from pixelserv-tls users. Pls see the announcement on SNBforum for credits.

2.2.0-rc.4 (2018-9-24)

Changes

  • NEW indicator of TLS 1.3 support status on servstats page
  • FIXED failed to log server name on unsuccessful handshakes. Garbage may be captured instead. When it happens it may lead to crash or a hung process.

Indicator of TLS 1.3 support on servstats page

Only pixelserv-tls compiled for OpenSSL 1.1.1 or above will support the final standard of TLS 1.3. You should see tls1_3 on the servstats page if your pixelserv-tls falls into this category. pixelserv-tls compiled for OpenSSL 1.1.0 or below won't support TLS 1.3. If your pixelserv-tls falls into this category, then you'll see no_tls1_3 on the servstats page. One example from servstats page that indicates pixelserv-tls supporting TLS 1.3:

pixelserv-tls 2.2.0-rc.4 (compiled: Sep 23 2018 19:12:30 flags: tls1_3) options: 192.168.1.3 -A 344 -l 2 -c 350

2.2.0-rc.3 (2018-9-22)

Changes

  • Added statically linked builds
  • No other functional changes

Statically linked builds

Beta versions for ARM routers running Entware now have a statically linked build that combines pixelserv-tls and OpenSSL 1.1.1 into a single binary. This eliminates the dependency of the OpenSSL library from Entware. Also since we use OpenSSL 1.1.1, this binary will support TLS 1.3. Additionally the memory optimization flag is enabled to provide better performance.

Dependent on your routers ARMv7 or ARMv8, the statically linked binary size is between 2MB to 3MB. This includes a copy of the brand new OpenSSL 1.1.1 library that in itself is a significant upgrade from OpenSSL 1.0.2.

Regular pixelserv-tls binaries (without OpenSSL) are also available. They work with the stock OpenSSL 1.0.2 library from Entware just like before. Note that this favour supports TLS <= 1.2 but not TLS 1.3.

The TLS 1.3 experience

To experience pixelserv-tls with TLS 1.3, you'll need a supported browser. Currently Firefox v63 beta and Chrome v70 DEV edition both support the final standard of TLS 1.3. You can get them from the previous links.

To check you're connected over TLS 1.3, in Firefox visit "https://pixelserv_ip/servstats". Click on the green padlock. Then ">" and next "More information". You should see TLS 1.3 is under "Technical Details". For Chrome, go into Developer's Tool window and then "Security" tab.

2.2.0-rc.2 (2018-9-15)

Changes

  • NEW support 0-RTT in TLS 1.3
  • CHANGED faster logging

Support 0-RTT in TLS 1.3

0-RTT is the magic in TLS 1.3 that enables HTTPS traffic finally to be as fast as plain HTTP. It also diminishes the value of HTTP persistent connection that we enabled in pixelserv-tls v2 as far as TLS 1.3 traffic is concerned.

It took me a few hours to understand the thing since it's marked as a bit controversial wherever you read an article on TLS 1.3. But then I realised it's not an issue in the scope of pixelserv-tls. Better yet I found the architecture refactoring in pixelserv-tls v2 allows 0-RTT functionality to be easily fit in. While it took me a few hours to understand, it only needs 30mins or so to code and test!

So here we go, we have rc.2 with a new feature. 0-RTT is better described in this blog post from Cloudflare.

2.2.0-rc.1 (2018-9-9)

Changes

  • NEW support TLSv1.3 (requires OpenSSL >= v1.1.1)
  • NEW bump version from 2.1.3 to 2.2.0 to indicate the significance of TLSv1.3.
  • CHANGED replaced deprecated OpenSSL APIs
  • CHANGED removed compile warnings
  • CHANGED integrated pull request #16

TLSv1.3 support

IETF officially published RFC 8446 in Aug, 2018. Essential features are better described in this blog post from Cloudflare. OpenSSL v1.1.1 (currently at pre-release 9) will support the official version of RFC 8446. Firefox will support the official TLSv1.3 in v63. Official support from Chrome and Safari shall follow soon.

For router users, you're unlikely to get OpenSSL v1.1.1 anytime soon. For example, Entware is still on v1.0.2, not even moved to v1.1.0 yet. However, PC or single-board computer users shall benefit from TLSv1.3 very soon.

One major benefit from pixelserv-tls perspective is faster speed. TLSv1.3 is one round-trip less between server and client than older standards. This significantly reduces delay in every new connection. Enjoy!

Browser quirks

Browsers are busy updating to support TLSv1.3. Perhaps because of that recent versions of Chrome show regression on TLSv1.2 session resumption. This implies Chrome users in general will see longer latency visiting HTTPS sites. From pixelserv-tls perspective, you shall see a higher tav.

Both latest Firefox and Safari are still good and same as before. So the issue is limited to recent versions of Chrome only.

2.1.3-test.4 (2018-7-17)

Changes

  • CHANGED relax memory pool restriction for better concurrent thread performance
  • CHANGED increase number of retries in TLS handshakes
  • CHANGED improve accuracy in processing time of requests recovered from initially failed handshakes.

Improvement Details

This test version is expected to improve performance in medium to heavy workload. The chance of "reached max retries" failures is reduced if not completely eliminated. Memory usage shall be similar to previous versions.

Also, tav shall more accurately reflect the reality, not screwed to longer but false processing time by failed requests of "reached max retries". For such requests, time is mostly spent on waiting. In previous versions, it's accounted as processing time. That's not correct.

2.1.3-test.3 (2018-7-10)

Changes

  • NEW counter ush - shutdown by clients after ServerHello.
  • NEW logging "shutdown after ServerHello" on LEVEL 2.

Shutdown by clients after ServerHello

A client initiates a handshake, receives a response from server and then shuts down the connection unilaterally. The most likely reason is a client finds out the certificate in the server's response not matching its hard-coded fingerprint. Instead of notifying the server of unknown cert or CA, the client shuts down the connection silently. It's considered suspicious client activity worth more attention.

2.1.3-test.2 (2018-7-2)

Changes

  • NEW logging "handshake failed: socket i/o error" on LEVEL 2.
  • NEW logging "handshake failed: reached max retries" on LEVEL 2.

2.1.3-test.1 (2018-6-26)

Changes

  • NEW improvement in uniqueness of serial numbers in generated certificates.

The issue

With a fast processor, previous versions will very likely generate two certificates with the same serial number. Firefox has historically treated such certificates as invalid. Hence, TLS handshake will fail, and boost your slu count significantly if the problematic certificates are frequently accessed.

Install

Use the same one-liner script to install.

Then delete all existing generated certificates with the following commands (or otherwise):

cd /opt/var/cache/pixelserv
mv ca.* ..
rm *
mv ../ca.* .

Release 2.1.2 (2018-6-20)

Changes

  • NEW support --enable-static for building static binary using GNU Autotools. (Issue #13)
  • CHANGED move other TLS handshake errors from log LEVEL 5 to 2.
  • FIXED missing client ip in logging TLS handshake errors in some situations.

Also don't forget to check out NEW features in v2.1.

Announcement (2018-5-24)

  • pixelserv-tls 2.1.1 Debian packages are available for all EdgeRouters (mipsel and 64-bit mips). Get it from GitHub. You may follow the previous blog post to install. Remember to substitute with v2.1.1 package file name.

  • An update to optimised OpenSSL library is available for Edgerouter X. This build turns on the flag OPENSSL_NO_BUF_FREELISTS which saves memory on many SSL connections that pixelserv-tls 2.1.1 usually maintains and serves throughout its lifetime. Get it from GitHub.

Announcement (2018-5-23)

  • pixelserv-tls 2.1.1 is available on Entware.
  • libopenssl 1.0.2o is also available. But it does not include this ticket. Hence, you may want to withhold upgrading libopenssl.

Release 2.1.1 (2018-4-15)

Changes

  • fixes 404 error on /ca.crt

Also don't forget to check out NEW features in v2.1.

Download

Pre-built binaries will be available from Entware in due time.

Early adopters can install v2.1.1 through the one-liner script, available for Entware ARMv7, ARMv8 64-bit and mipsel.

sh -c "$(wget -qO - https://kazoo.ga/pixelserv-tls/install-beta.sh)"

Release 2.1.0 (2018-4-11)

Below is the ChangeLog. Also don't forget to check out NEW features in v2.1.

Changes (since v2.0.1)

* NEW cache frequently used SSL certs in memory.
* NEW enable SSL session cache and resumptions.
* NEW prefetch SSL certs from disk on startup
* NEW save top 3/4 of mostly frequently used SSL certs to disk on exit.
* NEW counters `sct`, `sch`, `scm` and `scp` to register operations of caching SSL certs.
* NEW counters `sst`, `ssh`, `ssm` and `ssp` to register operations of cached sessions and resumptions.
* NEW option `-c` to specify cache size for SSL certs.
* NEW counters `uca` and `uce` to better classify TLS handshake failures
* NEW logging client ip and port and server name for `uca` and `uce` failures
* NEW URI `/ca.crt` to download and import users' Pixelserv CA on client devices.
* NEW crypto and disk benchmark for gauging performance of most timing sensitve routines in pixelserv-tls.
* NEW CLI option `-B` with optional argument to run crypto and disk benchmark.
* NEW preliminary support for Cross-Origin Resource Sharing (CORS).
* NEW support TLSv1.0 for wider compatibility with old clients.
* NEW port for administration. Optional but when used, it faciliates setup of firewall rules to restrict access to 'administrative' URIs such as '/log=' and '/servstats'.
* NEW ARMv8 64-bit build target (aka aarch64) in Makefile-XC. Proven to work in ASUS RT-AC86U.
* NEW ability to run without CA cert. In this mode pixelserv-tls will act as SNI servers.
* CHANGED improvement in TLS handshake handling. Added retries for better chance of success from troubled clients.
* CHANGED reduced socket latency inside service threads.
* CHANGED default HTTP_KEEPALIVE to 5 mins for boosting chance of reusing service threads and HTTP/1.1 persistent connections.
* CHANGED default CERT_PATH to /var/cache/pixelserv for non-Entware builds.
* CHANGED `slu` description to "other TLS handshake errors." Possible causes such as clients without CA cert, mismatch in TLS protocol version or other parameters.
* CHANGED removal of fork() code. Now pthread only for multiprocessing.
* CHANGED added support for glibc < 2.17 and OpenSSL 1.0.1
* FIXED race condition between reading CA's private key and dropping root
* FIXED crash on non-FQDN hostname (#9)

History of changes in Km cycle

2.1.0-rc.4 (2018-4-5)

  • NEW breaks down counter slu into uca and uce. See MAN page for how to make best use of the new counters
  • NEW logging on LEVEL 2 for clients registering in new counters uca and uce
  • CHANGED improves cert cache lookup; will work great with large "-c X". People with a quality OpenSSL library installed are encouraged to cache all frequently used certs.
  • CHANGED default "-c X" to 50 certs to prevent blowing up memory on systems with bad OpenSSL library.

2.1.0-rc.3 (2018-3-29)

  • NEW binary for EdgeRouter X
  • CHANGED improved memory efficiency

2.1.0-rc.2 (2018-3-20)

  • NEW ability to run without CA cert. See details below.
  • CHANGED removal of TLSv1.1 support

2.1.0-rc.1 (2018-3-17)

  • NEW support TLSv1.0 and TLSv1.1 for better backward compatibility with older clients when needed. Devices from the past decade are likely all supported with this change. Note that TLSv1.2 is most widely used at the moment and was supported by pixelserv-tls on day one.
  • CHANGED properly described slu as "other TLS handshake errors" (possible causes: clients not have CA cert installed, TLS version or other parameters mismatch)
  • FIXED one bug introduced in an earlier Km test version that may contribute to large portion of slu.

Km-test.7 (2018-3-14)

  • NEW support URI /ca.crt to download and import Pixelserv CA certificate (generated by yourself) on client devices
  • CHANGED improved SSL cache management. RAM usage should be much better under control.
  • CHANGED better define the meaning of counter sst
  • CHANGED improved handling of SSL connections. Increase chance of a successful connection for some troubled clients.
  • CHANGED added a bit more CORS support (Daily Mail alike shall be happier).
  • CHANGED split SSL request and SSL cache related counters into two tables on the servstats page.
  • FIXED missing new ssX counters from URI '/servstats.txt'

Km-test.6 (2018-3-13)

  • NEW counters sst, ssh, ssm, ssp to describe metrics of cached sessions
  • NEW crypto and disk benchmark, option -B
  • CHANGED improvement in SSL caching
  • CHANGED save up to 3/4 of most frequently used certs for prefetch
  • CHANGED timeout of cached sessions reduced to 1hr
  • CHANGED max allowed sessions in cache reduced to 2560.
  • CHANGED default HTTP_KEEPALIVE increased to 5mins

Km-test.5 (2018-3-10)

  • NEW preliminary CORS support
  • FIXED scm not properly initialised after prefetch on startup

Km-test.4 (2018-3-10)

  • NEW prefetch ssl cache from disk on startup
  • NEW save top two-third ssl cache to disk on exit
  • NEW benchmark loading cert from disk
  • NEW prefetch and purge logging on LEVEL 3
  • CHANGED improvement in check and purge of expired TLS sessions
  • CHANGED CERT_PATH default to /var/cache/pixelserv for non-Entware
  • FIXED scm double count

Km-test.3 (2018-3-8)

  • NEW optional port for administration (e.g. /servstats)(#5)
  • NEW ARMv8 64-bit build target
  • CHANGED improved ssl cache lookup
  • CHANGED reduced latency in child sockets
  • CHANGED reordered cipher preferences for less computation

Km-test.2 (2018-3-4)

  • Revised ssl caching code.

Km-test.1 (2018-3-3)

  • NEW cache frequently used SSL certs in memory and enable SSL session resume.
  • NEW option -c to specify ssl cache size.
  • NEW counters sct, sch, scm and scp
  • CHANGED removed fork() code
  • CHANGED added support for glibc < 2.17 and OpenSSL 1.0.1
  • FIXED crash on non-FQDN hostname (#9)

Announcement (2018-2-17)

Thanks to a user named liljaylj, binary package of pixelserv-tls is available on Arch Linux and its derivative distributions.

Release v2.0.1 (2017-12-17)

This is a bug fix release. The fastest and most efficient pixelserv-tls ever made. Enjoy v2 with your holidays.

Changes (since v2.0.0)

  • FIXED the 'stuck' issue
  • FIXED three crash bugs (#7 & #8)
  • FIXED incorrect logics of select() in the main event loop
  • FIXED a memory leak on failure to create a service thread for HTTPS
  • FIXED OpenSSL 1.1.x compatibility for Autotools (#6)
  • CHANGED more efficient check for next request in a service thread
  • CHANGED '-o SELECT_TIMEOUT' default to 10s
  • NEW backtrace on crash

More efficient check for next request

Replaced select() with poll() in service thread for checking next request. poll() unlike select() does not have a hard limit of 1024 file descriptors. Hence, maximum service won't be limited to 1024.

Also changed poll() to time out on -O KEEPALIVE_TIME instead of waking up every -o SELECT_TIMEOUT. It's more efficient and guarantees fastest response possible to the next available request. This may change the observation in v2.0.0 that 10s SELECT_TIMEOUT gives better overall speed than 1s. I didn't benchmark again. I believe SELECT_TIMEOUT no longer matters in terms of overall speed as well as counters such as 'tav' and 'tmx'.

The current implementation shall be an optimal solution. Please see the updated v2 benchmark where v2.0.1-rc4 competes against NXDOMAIN, fastest possible without pixelserv-tls.

I have plan to rewrite in asynchronous I/O in v3 timeframe. The paradigm will significantly increase concurrent connections e.g. >= 10k on ARM Cortex A9.

Download

Binaries will be available from Entware-ng in due time. Look for status change of this ticket. Early birds can download the last release candidate, v2.0.1-rc4 with install-beta.sh. It's functionally same as v2.0.1. Or build from source.

Instructions are available on Github's front-page.

v2.0.1 Development

v2.0.1-rc4 (2017-12-12)

  • fixed 'stuck' issue
  • more efficient check for next request in a service thread

v2.0.1-rc3 (2017-12-9)

  • added traces for debugging 'stuck' & crash issues
  • fixed two crash bugs (#7 & #8)

v2.0.1-rc2 (2017-12-7)

  • fixed incorrect logics of select() in the main event loop
  • fixed a memory leak on fail to create service thread for HTTPS connection
  • default select_timeout to 10s (rationale)

Release v2.0.0 (2017-11-29)

Changes (since version Kk):

* NEW support for HTTP/1.1 persistent connections
* NEW hugely increased scalability on concurrent connections & reduced memory requirement
* NEW logging facility. Six levels of granularities. No more chopped messages in syslog.
* NEW support for HTTP POST method & logging POST content (with -l 4)
* NEW added option '-O keepalive_time' to specify idle timeout of persistent connections
* NEW added option '-T max_threads' to specify maximum allowed service threads 
* NEW added option '-l level' to specify log level
* NEW added counters kcc, kmx, kvg, krq & clt related to persistent connections
* NEW added a MAN page as part of effort to bring pixelserv-tls more Linux
* NEW support for TCP Fast Open. Require Linux kernel >= 3.16.
* NEW support for GNU Autotools. Easier for native Linux systems to build and install.
* CHANGED default select_timeout to 1s
* CHANGED reviewed & updated log messages
* CHANGED descriptions of a few counters on servstats page
* CHANGED refactored SSL code & clean up connection handler
* CHANGED removed legacy build.sh from code repository
* CHANGED version naming scheme to MAJOR.MINOR.MICRO
* CHANGED enhanced ring buffer handling in certificate generations
* FIXED no matching cipher when servstats page is accessed using IP address
* FIXED measurement of POST processing time
* FIXED compilation failure on LEDGE due to missing TEMP\_RETRY\_FAILURE (thanks to a fix from [email protected])
* FIXED crash when the CA certificate is not available on startup.

The above list is from ChangeLog. A lot of things changed under the hood in v2 and brings very noticeable performance boost.

You may also scroll down this page and find additional details of some of the new features mentioned during development. The development cycle began with KL-test1 and ended with KL-test8d, spanning two months with multiple test versions and very helpful feedback from beta users (separately credited on this forum post).

Other Highlights with v2 release

  1. Tidy up the frontpage on Github and its Wiki. The frontpage now shows its simplicity. Details are moved to separate pages on the wiki.

  2. A new MAN page for pixelserv-tls. It'll be installed for full-blown Linux systems. Type man pixelserv-tls to remind yourself of CLI options. Or you can check out on wiki here.

  3. A new FAQ for pixelserv-tls. Yeah, finally a FAQ on wiki.

  4. A install-beta.sh script to pull the latest beta binary. You can see the one line command on the frontpage.

  5. Change in Naming Scheme to Versions Begin with v2 release, version naming reverts back to traditional MAJOR.MINOR.MICRO. See this FAQ to what happened to v1.x.

  6. A new way to build natively on AMD64 systems. Using GNU Autotools. Instructions are here on the frontpage.


OLDER CONTENTS CAN BE FOUND IN THE ARCHIVE.


comments powered by Disqus