pixelserv-tls

pixelserv-tls started as a toy project to understand OpenSSL API by forking the original pixelserv. It is still a hobby but luckily ends up being something useful at the same time.

pixelserv-tls inherits fundamental functionality from pixelserv that is enhancing browsing experience with faster empty advert responses. pixelserv-tls adds HTTPS and HTTP/1.1 support which on their own are major steps forward when compared its ancestor. The HTTPS feature to my understanding is so far unique on this planet. :)

Over the past two years, the internal of pixelserv-tls has been gradually upgraded to better technologies and hence have morphed into a very different state from its ancestor. For example, pixelserv-tls builds on top of pthread and implements persistent connection, a mandatory requirement by HTTP/1.1 specification. Both features boost performance and efficiency. See the benchmark.

For usage details, check out the wiki and in particular the FAQ.

2.1.3-test.4 (2018-7-17)

Changes

  • CHANGED relax memory pool restriction for better concurrent thread performance
  • CHANGED increase number of retries in TLS handshakes
  • CHANGED improve accuracy in processing time of requests recovered from initially failed handshakes.

Download

sh -c "$(wget -qO - https://kazoo.ga/pixelserv-tls/install-beta.sh)"  

Improvement Details

This test version is expected to improve performance in medium to heavy workload. The chance of "reached max retries" failures is reduced if not completely eliminated. Memory usage shall be similar to previous versions.

Also, tav shall more accurately reflect the reality, not screwed to longer but false processing time by failed requests of "reached max retries". For such requests, time is mostly spent on waiting. In previous versions, it's accounted as processing time. That's not correct.

2.1.3-test.3 (2018-7-10)

Changes

  • NEW counter ush - shutdown by clients after ServerHello.
  • NEW logging "shutdown after ServerHello" on LEVEL 2.

Shutdown by clients after ServerHello

A client initiates a handshake, receives a response from server and then shuts down the connection unilaterally. The most likely reason is a client finds out the certificate in the server's response not matching its hard-coded fingerprint. Instead of notifying the server of unknown cert or CA, the client shuts down the connection silently. It's considered suspicious client activity worth more attention.

2.1.3-test.2 (2018-7-2)

Changes

  • NEW logging "handshake failed: socket i/o error" on LEVEL 2.
  • NEW logging "handshake failed: reached max retries" on LEVEL 2.

2.1.3-test.1 (2018-6-26)

Changes

  • NEW improvement in uniqueness of serial numbers in generated certificates.

The issue

With a fast processor, previous versions will very likely generate two certificates with the same serial number. Firefox has historically treated such certificates as invalid. Hence, TLS handshake will fail, and boost your slu count significantly if the problematic certificates are frequently accessed.

Install

Use the same one-liner script to install.

Then delete all existing generated certificates with the following commands (or otherwise):

cd /opt/var/cache/pixelserv  
mv ca.* ..  
rm *  
mv ../ca.* .  

Release 2.1.2 (2018-6-20)

Changes

  • NEW support --enable-static for building static binary using GNU Autotools. (Issue #13)
  • CHANGED move other TLS handshake errors from log LEVEL 5 to 2.
  • FIXED missing client ip in logging TLS handshake errors in some situations.

Also don't forget to check out NEW features in v2.1.

Announcement (2018-5-24)

  • pixelserv-tls 2.1.1 Debian packages are available for all EdgeRouters (mipsel and 64-bit mips). Get it from GitHub. You may follow the previous blog post to install. Remember to substitute with v2.1.1 package file name.

  • An update to optimised OpenSSL library is available for Edgerouter X. This build turns on the flag OPENSSL_NO_BUF_FREELISTS which saves memory on many SSL connections that pixelserv-tls 2.1.1 usually maintains and serves throughout its lifetime. Get it from GitHub.

Announcement (2018-5-23)

  • pixelserv-tls 2.1.1 is available on Entware.
  • libopenssl 1.0.2o is also available. But it does not include this ticket. Hence, you may want to withhold upgrading libopenssl.

Release 2.1.1 (2018-4-15)

Changes

  • fixes 404 error on /ca.crt

Also don't forget to check out NEW features in v2.1.

Download

Pre-built binaries will be available from Entware in due time.

Early adopters can install v2.1.1 through the one-liner script, available for Entware ARMv7, ARMv8 64-bit and mipsel.

sh -c "$(wget -qO - https://kazoo.ga/pixelserv-tls/install-beta.sh)"  

Release 2.1.0 (2018-4-11)

Below is the ChangeLog. Also don't forget to check out NEW features in v2.1.

Changes (since v2.0.1)

* NEW cache frequently used SSL certs in memory.
* NEW enable SSL session cache and resumptions.
* NEW prefetch SSL certs from disk on startup
* NEW save top 3/4 of mostly frequently used SSL certs to disk on exit.
* NEW counters `sct`, `sch`, `scm` and `scp` to register operations of caching SSL certs.
* NEW counters `sst`, `ssh`, `ssm` and `ssp` to register operations of cached sessions and resumptions.
* NEW option `-c` to specify cache size for SSL certs.
* NEW counters `uca` and `uce` to better classify TLS handshake failures
* NEW logging client ip and port and server name for `uca` and `uce` failures
* NEW URI `/ca.crt` to download and import users' Pixelserv CA on client devices.
* NEW crypto and disk benchmark for gauging performance of most timing sensitve routines in pixelserv-tls.
* NEW CLI option `-B` with optional argument to run crypto and disk benchmark.
* NEW preliminary support for Cross-Origin Resource Sharing (CORS).
* NEW support TLSv1.0 for wider compatibility with old clients.
* NEW port for administration. Optional but when used, it faciliates setup of firewall rules to restrict access to 'administrative' URIs such as '/log=' and '/servstats'.
* NEW ARMv8 64-bit build target (aka aarch64) in Makefile-XC. Proven to work in ASUS RT-AC86U.
* NEW ability to run without CA cert. In this mode pixelserv-tls will act as SNI servers.
* CHANGED improvement in TLS handshake handling. Added retries for better chance of success from troubled clients.
* CHANGED reduced socket latency inside service threads.
* CHANGED default HTTP_KEEPALIVE to 5 mins for boosting chance of reusing service threads and HTTP/1.1 persistent connections.
* CHANGED default CERT_PATH to /var/cache/pixelserv for non-Entware builds.
* CHANGED `slu` description to "other TLS handshake errors." Possible causes such as clients without CA cert, mismatch in TLS protocol version or other parameters.
* CHANGED removal of fork() code. Now pthread only for multiprocessing.
* CHANGED added support for glibc < 2.17 and OpenSSL 1.0.1
* FIXED race condition between reading CA's private key and dropping root
* FIXED crash on non-FQDN hostname (#9)

History of changes in Km cycle

2.1.0-rc.4 (2018-4-5)

  • NEW breaks down counter slu into uca and uce. See MAN page for how to make best use of the new counters
  • NEW logging on LEVEL 2 for clients registering in new counters uca and uce
  • CHANGED improves cert cache lookup; will work great with large "-c X". People with a quality OpenSSL library installed are encouraged to cache all frequently used certs.
  • CHANGED default "-c X" to 50 certs to prevent blowing up memory on systems with bad OpenSSL library.

2.1.0-rc.3 (2018-3-29)

  • NEW binary for EdgeRouter X
  • CHANGED improved memory efficiency

2.1.0-rc.2 (2018-3-20)

  • NEW ability to run without CA cert. See details below.
  • CHANGED removal of TLSv1.1 support

2.1.0-rc.1 (2018-3-17)

  • NEW support TLSv1.0 and TLSv1.1 for better backward compatibility with older clients when needed. Devices from the past decade are likely all supported with this change. Note that TLSv1.2 is most widely used at the moment and was supported by pixelserv-tls on day one.
  • CHANGED properly described slu as "other TLS handshake errors" (possible causes: clients not have CA cert installed, TLS version or other parameters mismatch)
  • FIXED one bug introduced in an earlier Km test version that may contribute to large portion of slu.

Km-test.7 (2018-3-14)

  • NEW support URI /ca.crt to download and import Pixelserv CA certificate (generated by yourself) on client devices
  • CHANGED improved SSL cache management. RAM usage should be much better under control.
  • CHANGED better define the meaning of counter sst
  • CHANGED improved handling of SSL connections. Increase chance of a successful connection for some troubled clients.
  • CHANGED added a bit more CORS support (Daily Mail alike shall be happier).
  • CHANGED split SSL request and SSL cache related counters into two tables on the servstats page.
  • FIXED missing new ssX counters from URI '/servstats.txt'

Km-test.6 (2018-3-13)

  • NEW counters sst, ssh, ssm, ssp to describe metrics of cached sessions
  • NEW crypto and disk benchmark, option -B
  • CHANGED improvement in SSL caching
  • CHANGED save up to 3/4 of most frequently used certs for prefetch
  • CHANGED timeout of cached sessions reduced to 1hr
  • CHANGED max allowed sessions in cache reduced to 2560.
  • CHANGED default HTTP_KEEPALIVE increased to 5mins

Km-test.5 (2018-3-10)

  • NEW preliminary CORS support
  • FIXED scm not properly initialised after prefetch on startup

Km-test.4 (2018-3-10)

  • NEW prefetch ssl cache from disk on startup
  • NEW save top two-third ssl cache to disk on exit
  • NEW benchmark loading cert from disk
  • NEW prefetch and purge logging on LEVEL 3
  • CHANGED improvement in check and purge of expired TLS sessions
  • CHANGED CERT_PATH default to /var/cache/pixelserv for non-Entware
  • FIXED scm double count

Km-test.3 (2018-3-8)

  • NEW optional port for administration (e.g. /servstats)(#5)
  • NEW ARMv8 64-bit build target
  • CHANGED improved ssl cache lookup
  • CHANGED reduced latency in child sockets
  • CHANGED reordered cipher preferences for less computation

Km-test.2 (2018-3-4)

  • Revised ssl caching code.

Km-test.1 (2018-3-3)

  • NEW cache frequently used SSL certs in memory and enable SSL session resume.
  • NEW option -c to specify ssl cache size.
  • NEW counters sct, sch, scm and scp
  • CHANGED removed fork() code
  • CHANGED added support for glibc < 2.17 and OpenSSL 1.0.1
  • FIXED crash on non-FQDN hostname (#9)

Announcement (2018-2-17)

Thanks to a user named liljaylj, binary package of pixelserv-tls is available on Arch Linux and its derivative distributions.

Release v2.0.1 (2017-12-17)

This is a bug fix release. The fastest and most efficient pixelserv-tls ever made. Enjoy v2 with your holidays.

Changes (since v2.0.0)

  • FIXED the 'stuck' issue
  • FIXED three crash bugs (#7 & #8)
  • FIXED incorrect logics of select() in the main event loop
  • FIXED a memory leak on failure to create a service thread for HTTPS
  • FIXED OpenSSL 1.1.x compatibility for Autotools (#6)
  • CHANGED more efficient check for next request in a service thread
  • CHANGED '-o SELECT_TIMEOUT' default to 10s
  • NEW backtrace on crash

More efficient check for next request

Replaced select() with poll() in service thread for checking next request. poll() unlike select() does not have a hard limit of 1024 file descriptors. Hence, maximum service won't be limited to 1024.

Also changed poll() to time out on -O KEEPALIVE_TIME instead of waking up every -o SELECT_TIMEOUT. It's more efficient and guarantees fastest response possible to the next available request. This may change the observation in v2.0.0 that 10s SELECT_TIMEOUT gives better overall speed than 1s. I didn't benchmark again. I believe SELECT_TIMEOUT no longer matters in terms of overall speed as well as counters such as 'tav' and 'tmx'.

The current implementation shall be an optimal solution. Please see the updated v2 benchmark where v2.0.1-rc4 competes against NXDOMAIN, fastest possible without pixelserv-tls.

I have plan to rewrite in asynchronous I/O in v3 timeframe. The paradigm will significantly increase concurrent connections e.g. >= 10k on ARM Cortex A9.

Download

Binaries will be available from Entware-ng in due time. Look for status change of this ticket. Early birds can download the last release candidate, v2.0.1-rc4 with install-beta.sh. It's functionally same as v2.0.1. Or build from source.

Instructions are available on Github's front-page.

v2.0.1 Development

v2.0.1-rc4 (2017-12-12)

  • fixed 'stuck' issue
  • more efficient check for next request in a service thread

v2.0.1-rc3 (2017-12-9)

  • added traces for debugging 'stuck' & crash issues
  • fixed two crash bugs (#7 & #8)

v2.0.1-rc2 (2017-12-7)

  • fixed incorrect logics of select() in the main event loop
  • fixed a memory leak on fail to create service thread for HTTPS connection
  • default select_timeout to 10s (rationale)

Release v2.0.0 (2017-11-29)

Changes (since version Kk):

* NEW support for HTTP/1.1 persistent connections
* NEW hugely increased scalability on concurrent connections & reduced memory requirement
* NEW logging facility. Six levels of granularities. No more chopped messages in syslog.
* NEW support for HTTP POST method & logging POST content (with -l 4)
* NEW added option '-O keepalive_time' to specify idle timeout of persistent connections
* NEW added option '-T max_threads' to specify maximum allowed service threads 
* NEW added option '-l level' to specify log level
* NEW added counters kcc, kmx, kvg, krq & clt related to persistent connections
* NEW added a MAN page as part of effort to bring pixelserv-tls more Linux
* NEW support for TCP Fast Open. Require Linux kernel >= 3.16.
* NEW support for GNU Autotools. Easier for native Linux systems to build and install.
* CHANGED default select_timeout to 1s
* CHANGED reviewed & updated log messages
* CHANGED descriptions of a few counters on servstats page
* CHANGED refactored SSL code & clean up connection handler
* CHANGED removed legacy build.sh from code repository
* CHANGED version naming scheme to MAJOR.MINOR.MICRO
* CHANGED enhanced ring buffer handling in certificate generations
* FIXED no matching cipher when servstats page is accessed using IP address
* FIXED measurement of POST processing time
* FIXED compilation failure on LEDGE due to missing TEMP\_RETRY\_FAILURE (thanks to a fix from [email protected])
* FIXED crash when the CA certificate is not available on startup.

The above list is from ChangeLog. A lot of things changed under the hood in v2 and brings very noticeable performance boost.

You may also scroll down this page and find additional details of some of the new features mentioned during development. The development cycle began with KL-test1 and ended with KL-test8d, spanning two months with multiple test versions and very helpful feedback from beta users (separately credited on this forum post).

Other Highlights with v2 release

  1. Tidy up the frontpage on Github and its Wiki. The frontpage now shows its simplicity. Details are moved to separate pages on the wiki.

  2. A new MAN page for pixelserv-tls. It'll be installed for full-blown Linux systems. Type man pixelserv-tls to remind yourself of CLI options. Or you can check out on wiki here.

  3. A new FAQ for pixelserv-tls. Yeah, finally a FAQ on wiki.

  4. A install-beta.sh script to pull the latest beta binary. You can see the one line command on the frontpage.

  5. Change in Naming Scheme to Versions Begin with v2 release, version naming reverts back to traditional MAJOR.MINOR.MICRO. See this FAQ to what happened to v1.x.

  6. A new way to build natively on AMD64 systems. Using GNU Autotools. Instructions are here on the frontpage.


OLDER CONTENTS CAN BE FOUND IN THE ARCHIVE.


Author

Stephen Yip

Something about you know. Come and share.

comments powered by Disqus