Archive of /pixelserv-tls

Version KL-test8 (2017-11-17)

Changes:

  • Refactored SSL code and reduced memory requirement by more than 40% when compared to KL-test7
  • Fixed an issue of no matching ciphers when pixelserv-tls stats page is accessed over HTTPS and using IP address.
  • Fixed an error in automatically generated certificate when IP address is the hostname.
  • Fine tuned buffer management for socket message and improved memory efficiency.
  • More robust binary check for logging POST content (applicable only when -l 4 or above used
  • [KL-test8a] Fixed counter issues due to code refactoring.
  • [KL-test8b] Fixed glitches in 'tav' and 'tmx'
  • [KL-test8c] Fixed glitches in 'krq', kvg' and 'cls'. Refined the description of 'tmo' on servstats page.
  • [KL-test8d] Keep CLI option '-l' without level for backward compatibility.

Refactoring SSL code

KL-test8 focuses on bringing faster HTTPS responses. Revisited and refactored the SSL code in pixelserv-tls. Memory usage is reduced by more than 40% at the same amount of concurrent HTTPS connections.

RAM used on RT-AC56U vs # of concurrent HTTPS connections

# of HTTPS   KL-test8   KL-test7
       400   19.7MB     35.1MB
       800   38.9MB     68.5MB
      1200   58.1MB     91.7MB

Note that for SOHO environment, pixelserv-tls can rarely hit 100 concurrent HTTPS connections (or equivalently aka service threads) under normal use. The above figures stress the extreme conditions and memory requirement.

After this refactoring exercise, pixelserv-tls is readily prepared for further major improvement in efficiency in a future version.

Supported browsers for HTTPS connections

  • Android >= 4.4.2; Chrome >= 51; Firefox >= 49
  • IE 11 Win 10; Edge >= 13; Safari >= 9; Apple ATS 9 iOS 9
  • IE 11 Win 7,8.1; IE 11 Winphone 8.1; Opera >= 17; Safar 7 iOS 7.1

Download

Binaries for 64-bit AMD64, mipsel (Entware-ng), and armv7 (Entware-ng) can be downloaded here.

Version KL-test7 (2017-11-14)

Changes:

  • More efficient threading and higher capacity of concurrent connections
  • Changed default max threads from 400 to 1200
  • Changed timeout for POST content back to 5s
  • Fixed a major issue in processing POST content (see last section in this article)
  • Fixed a bug/more rigid check for binary POST content not to output to syslog

Higher capacity of concurrent connections

Tuned threading for more efficient use of resources, and allow more concurrent connections. Hence, default max of service threads is raised from 400 to 1200. On RT-AC56U with 256MB RAM, 2000 HTTPS concurrent connections are no problem.

Higher efficiency shall also reduce chance of crash on systems of very high load such as hundreds of pixelserv-tls instances with each accepting thousands of connections.

Download

Binaries for 64-bit AMD64, mipsel (Entware-ng), and armv7 (Entware-ng) can be downloaded here.

Version KL-test6 (2017-11-12)

Changes:

  • Do not output binary POST content to syslog

Logging HTTP POST content

Users have the option to enable access log with CLI option '-l 4'. That is log level INFO. All request URLs and POST contents will be output to syslog.

Advert networks and trackers are increasingly capturing more data from victims. Using HTTP POST method to transmit the content to their servers is a common practice.

Most websites POST with text i.e. Base64/UU encoded. Some websites claim they're sending text but actually transmitting binary data.

Test6 will not output such binary data to syslog even with '-l 4' as it'll upset some syslog implementations and spit the binary data over multiple log files.

Instead a message like this [-binary POST content not dumped-] will be logged.

Support for mipsel Debian on ER-X

After much effort to bring support to current ER-X, I conclude it's not the right time to spend more effort on Debian Wheezy. I'll wait for EdgeOS 2.0 release where the firmware will migrate to Debian Jessie, and pixelserv-tls will run smoothly.

Download

Binaries for 64-bit AMD64, mipsel (Entware-ng), and armv7 (Entware-ng) can be downloaded here.

Version KL-test5 (2017-11-5)

Changes:

  • New capability to log full content of POST message. Max 256K data per POST can be logged. Use the same INFO (4) level.
  • New control for maximum allowed service threads (default 400)
  • New CLI option '-T num' for user adjustable max number of threads
  • New counter 'clt' - dropped requests due to exceeding max threads
  • Changed 'select_timeout' default to 1s
  • Changed keep-alive for HTTP/1.1 connections default to 120s
  • Fixed chopped access log in some situations e.g. when URL is very long.
  • Fixed stale 'kcc' count and never drop down to single digit.

Enhanced access log for privacy check

No more chopped data in logging client requests. The new capability of logging POST content is very good for privacy check. Many websites and companies send hundreds kB of data as POST messages from you to them in background. This data could include mobile number and IMEI from your smartphone for example.

Be warned that INFO (4) level is very chatty. You may only want to enable when needed if you don't have a robust syslog such as syslog-ng configured.

For ARM & MIPS routers

Better performance could be achieved with setting 'ulimit -s 64' before starting pixelserv-tls. For example, in the Entware-ng init.d script, S80-pixelserv-tls, you could

#!/bin/sh

ENABLED=yes
PROCS=pixelserv-tls
ARGS=""
PREARGS=""
DESC=$PROCS
PATH=/opt/sbin:/opt/bin:/opt/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

ulimit -s 64
. /opt/etc/init.d/rc.func

Download KL-test5

Binaries for 64-bit AMD64, mipsel (Entware-ng), and armv7 (Entware-ng) can be downloaded here.

Update: Binary for mipsel Debian 7 aka Edgerouter-X is added and available from the same place. Note that it's an experimental build. Timing related counters don't work yet. Other features seem to be fully working.

Version KL-test4 (2017-10-28)

Changes:

  • Enhanced efficiency and robustness in reading messages from sockets.
  • Reduced initial size of message buffer to 4kB. Can grow on demand to as big as 128kB.
  • Added support for GNU Autotools. Easier for native Linux systems to build and install.
  • Renamed old Makefile to Makefile-XC (aka old Makefile for easier cross-compilation)
  • Added macro TEMP_RETRY_FAILURE and fixed compilation error on LEDE which uses musl libc. Thanks to laoshaw on github.
  • Fixed crash when the CA cert is unavailable on startup.
  • Fixed missing host name in clients' access logs.
  • Removed legacy build.sh script.

For Linux desktop/server users

To compile your own binary, now you can do so in the familiar way:

    autoreconf -i
    ./configure
    make (or make install)

Make sure you have libssl-dev package installed from your Linux distribution. Please refer to INSTALL for more details on compiling pixelserv-tls on different platforms.

Version KL-test3 (2017-10-23)

Changes:

  • Fixed crashes in processing POST.
  • Fixed growing number of service threads under some usage scenarios.
  • Changed tiers of a few logging messages.
  • Minor changes to descriptions of a few counters.

Note that increasing verbrosity of logging will introduce extra processing time. Noticeable on slow machines e.g. 800MHz ARM Cortex-A9. The default '-l 1' gives a very snappy browsing experience on 800MHz Cortex-A9 in my tests.

Version KL-test2 (2017-10-13)

Changes:

  • added new facility of tiered logging
  • reviewed/refined tiers of all logged messages
  • expanded the CLI option '-l' to '-l level'
  • expanded the function of dynamically switching log tiers through HTTP
  • expanded display of counter 'log' on servstats page
  • cleaned up further logics/codes in connection handler
  • fixed measurement of POST processing time

Notes on log tiers:

  • six tiers of logging: critical (0) error (1) warning (2) notice (3) info (4) debug (5)
  • from 0 to 5 in increasing verbrosity. Default is (1) if omitted in command line.

Version KL-test1 (2017-10-9)

Changes:

  • Support HTTP/1.1 persistent connections
  • Add option 'O' for specifying timeout of persistent connections
  • Add new counters kcc, kmx, kvg and krq related to service threads (of persistent connections). See a sample servstats page for details.
  • Add support for TCP Fast Open (require Linux kernel >= 3.16)
  • Support HTTP POST method for graceful processing
  • Double message buffer for more efficient processing of lengthy requests
  • Correct/enhance ring buffer handling in certificate generations

Announcement (2017-9-27)

  • Version Kk is also available from Entware-ng repository.

If you run into permission error on /opt/var/cache/pixelserv, run as root

$ chown nobody /opt/var/cache/pixelserv

Release of Version Kk (2017-9-18)

Changes (since Version Kj):

  • added support for HTTP OPTIONS method (issue reported by Popov on snbforums)
  • added experimental support for HSTS
  • added a new counter 'slc' that counts the number of clients who disconnect without sending any data. It usually indicates clients not have ca.crt installed.
  • added new counter 'cly' - client disconnect before response sent
  • fixed certificates generated for erroneous domains under extreme load
  • fixed crashes when non-SNI requests are received (reproducible case reported by Popov on snbforums. Thanks!)
  • fixed crashes when null requests are received while logging enabled
  • fixed delay in certificate generation
  • fixed potential duplication in certificate generation
  • fixed description of counter 'err' and 'cls'
  • updated README.md for macOS/iOS instructions, and sample servstats page.

Binaries for 64-bit AMD64, ASUSWRT mips, and patch for Entware-ng armv7 softfloat can be downloaded here. Enjoy and have fun!

Version Kk-test4 (2017-9-4)

Changes:

  • added new counter 'cly' - client disconnect before response sent
  • fixed description of counter 'err'

Version Kk-test3 (2017-9-1)

Changes:

  • added support for HTTP OPTIONS method [issue reported by [email protected]]
  • added a new counter 'slc' that counts the number of clients who disconnect without sending any data. It usually indicates clients not have ca.crt installed.
  • added experimental support for HSTS
  • fixed description of counter 'cls'

Version Kk-test2 (2017-8-28)

Change:

  • fixed certificates generated for erroneous domains under extreme load

Version Kk-test (2017-8-24)

Changes:

  • fixed crashes when non-SNI requests are received [reported by [email protected]]
  • fixed crashes when null requests are received while logging enabled
  • fixed delay in certificate generation
  • fixed potential duplication in certificate generation
comments powered by Disqus