Skip to content

Category: Linux

Ending an Era of OpenBSD: Or a Brief History of my Firewalls

For something approaching 20 years, I’ve used OpenBSD to firewall my network from the internet and provide basic network services (DHCP, DNS, NTP, VPN, etc.). Just recently I’ve decided to retire OpenBSD and stand alone computers from the role of firewalls for something smaller, lower power, and easier to manage and upgrade.

I’ve been steadily moving towards smaller and lower power systems for as long as I’ve been doing OpenBSD based firewalls. My first machines were nothing more than mid-tower desktops that I had upgraded away from. In 2000-2003 I made my first moves towards building something more specialized, when I switched from using old towers to building a specific micro-atx pizza box style machines; though still with standard Athlon XP CPUs and parts.

Lua String Compare Performance Testing (Nginx-Lua)

In another article I wrote about my ongoing attempt to move my server’s WordPress’s security plugin’s firewall functionality out of PHP and into the embedded lua environment in Nginx. While I’m certainly not nearly as the scale where the C10K problem is a real issue for me, I still do my best to insure that I’m doing things as efficiently as possible.

In my last post, I was looking at the performance degradation between doing no firewalling at all (just building the page in WordPress and serving it), and using the embedded Lua environment to do basic application firewalling tests.

In that article, I saw approximately 425 microsecond latency impact form the Lua processing compared to just building the page. Of course, that was still on the order of 2 orders of magnitude faster than doing the same work in PHP.

Part of the larger part of the actual processing that is being done, is looking for various strings in the myriad of data that’s pushed along as part of the various requests. Things like, know bad user agents, key bits used in SQL injection attacks, and various things like that.

Lua and Nginx both offer some options for searching strings. On the Lua side, there’s the built in string.find() (Lua5.1 docs) and associated functions. On the Nginx-Lua side of things there’s ngx.re.find() (lua-nginx-module docs) which allows calls into Nginx’s regex engine.

I’ve done a significant amount of digging trying to find performance informational about both of these methods, and I haven’t been able to find any. So I sat down and did my own testing.

Nginx-Lua Module: Access Control Performance Testing

I’ve been playing with the Lua engine in Nginx for a while. My primary intent is to offload most, if not all, of my WordPress security stuffy from running in the PHP environment to running in something that potentially won’t use as much in the way of resources. The first question I need to answer before I can reasonably consider doing this is what kind of of overhead doing extended processing in Nginx–Lua imposes in terms of performance.

To put some perspective on this, I’ve been running the WordPress security plug-in Word Fence for a while now. When I compare my production server (which has Wordfence enabled) and my development server (which doesn’t have word fence installed, but is otherwise running the same plugins and code base), I see on average a 10–20 ms increase page rendering times, and nearly 20 additional database queries per page.

The overhead from Wordfence isn’t creating a performance problem per say, however, shaving even 15 ms off a 50–60 ms page render time is an appreciable improvement. Additionally, less resources consumed by a bad actor means more resources are available for actual users.

In any even the question here is how much performance overhead does the Nginx-Lua module carry for doing some reasonable processing.

Setting up OpenVPN with Certificates

I did this a couple of years ago, with certificates that had a 1 year expiry date. Then my certs expired, and I’d forgotten what to do. So I figured it out again, and this time I’m writing it down.

There are two ways to setup client auth in OpenVPN, a shared secret and TLS certificates. TLS certificates are the preferred way if you can manage them, as they make it possible to revoke access to devices without having to change the shared secret for every other device.

To do this you need to setup a certificate authority and sign and issue your own certificates. Most OpenVPN guides tell you how to do this using OpenSSL and it’s associated long cryptic commands. I like my method better.

Lets Encrypt & Nginx: Www-root method and Subject Alt Names

Digital Ocean has a pretty good guide for setting up Lets Encrypt with Nginx on Ubuntu 14.04. However, their guide requires you to turn down your Nginx server while initially getting you Lets Encrypt TLS certificates, this of course is problematic for server/site operators who either need or want to continue to have service continuity while getting lets Encrypt Certificates. They also don’t explain how to use subject alternative names to handle multiple sub domains on the same server.

Lets Encrypt’s software requires that they be able to connect to your server to verify that the domain you’re attempting to register a certificate for you control. In the Digital Ocean guide, this process is handled by using the built-in web server in the lens encrypt package. However, Lets Encrypt does not need to operate in this manner to create new certificates, it can use the wwwroot/filesystem approach and your existing server configuration.

The process is very similar to Digital Oceans guide, but the order of operations are slightly different.

Fixing my slow Wordpress, Nginx, & WP-Supercache Setup

I probably should have caught this one a long time ago, but I didn’t. For a while now I’ve been complaining endlessly, at least in my internal monologue, about the poor performance I’ve been seeing from WP-Supercache on my VPS. Preloaded cache files simply shouldn’t take 1-1.5 seconds to serve up. They should be quick quick quick. Yet I was seeing such slow load times.

I’ve been struggling with the issue for quite sometime. I had changed from WP-Supercache to W3 Total Cache, added memcached, cached DB operations and so on and so forth trying to figure out why my pages were so slow to load.

What struck me, was that when I rolled back my W3 Total Cache implementation to no caching, responsiveness stayed about the same or got slightly better. As a non-logged in user, the inverse should have been true. Pages should have taken longer to load without a caching implementation than with one.

Then it hit me, Nginx config files are parsed in order.

Okay, let me take  a step to the side here a moment. On Dreamhost VPSes if you’re running Nginx the server looks for supplementary config files in ~/nginx/$domain/ for each of the virtual hosts it’s configured to serve. Knowing that Nginx config files are read in order, I organized mine using the ##-description.conf style. So I might have 10-rewrites.conf, 50-wordpress.conf, 60-supercache.conf.

On a lark the idea struck me that maybe I should try loading the supercache rules before I got to the regular Wordpress rules, which include the directives for passing requests for .php files back to the PHP back-end.

A quick rename of 45-supercache.conf to 30-supercache.conf, thus placing the supercache rules ahead of the Wordpress rules, and my non-logged in user (i.e. reading from the static HTML cache) page response times dropped form 1-1.5s to less than 400ms.

Suffice to say, all my grumbling about slow performance was entirely due to poor configuration on my part.

I’m sure there’s probably a note about this somewhere in one of the myriad of guides for running Wordpress and WP-Supercache on Nginx, but I missed it.

I’d still love to improve the response time for pages that are being processed with PHP, but I can live with it being a touch slower for me knowing that it’s a whole lot faster for everybody else.

Bash: Watching Aliases

If you’re trying to watch the output of an alias, you need to make watch an alias of watch as well.

For example, if you have an alias like say:

alias zfslist='zfs list -o name,volsize,used,avaiable,referenced,compressratio,mountpoint'

And want to watch the output over time as if you ran:

user@host:~/$ watch zfslist

Then you need to set up watch as an alias to watch, as follows.

alias watch='watch '

The space after watch is necessary to get bash to expand subsequent aliases after the first one.

Unclear Instructions: Setting Ubuntu/Unity Keyboard Shortcuts

I don’t know who wrote the text for the dialog for changing Keyboard shortcuts in Ubuntu 11.10, but wow could they have been more misleading.

The instructions read, “click the row and hold down the new keys,” in reality the only place that will allow you to change the short cut is to click the text directly under the black arrow in the above image. I spent the better part of 20 minutes trying to figure out why I couldn’t change a keyboard shortcut because the directions are utterly useless.

This should probably also be filed under: Won’t file a bug report as it’s too much bloody work, and now I won’t forget how to do it.

Wordpress on Nginx on Dreamhost

By the time this is done being written I’ll have been running Wordpress on Nginx on a Dreamhost VPS. Better yet, I’ll be doing it with a smaller more well defined resource foot print, with better response times, and faster page loads than I had with Apache. The tradeoff, some more upfront configuration.

I’ve covered the broad strokes about my motivation here, here, and here. In short, Nginx offers a more consistent dependable level resource usage while still having the capability of scaling to serve may users, reducing the possibility of crashing the VPS while under moment of heavy load.

This is long, and may end up being multiple parts, so if you’re interested follow the jump and keep reading.

Getting a feel for the Nginx Stack

Forgive me, I’m about to ramble here.

For the past several months now I’ve been dealing with trying to get my VPS configured in such a way that it was stable and used as few resources (mostly RAM) as possible. During this process I had considered switching the web server from Apache2 to one of the lighter replacements. More and more I’ve been reading about the preference for Nginx (pronounced engine-x), along with PHP-FPM, as the defacto standard for high performance PHP sites.

Time to investigate.

The Nginx Stack

I swapped Apache 2 and mod_php out on my dev machine with Nginx and php-fpm a couple of days ago. Mostly to make sure everything would go smoothly if I decided to move my VPS over and figure out what rule changes I’d have to make to get Nginx running.

To start with Nginx doesn’t use the 1-process per connection model Apache does, instead it uses async IO. This addresses one of the biggest Apache problems I’ve had to contend with, a sudden spike in traffic spawning off a 10s of new processes, is no longer an issue at all.

Nginx’s memory foot print is comparatively tiny too. I’m seeing about 10MB total for the 2 workers + the master process, instead of 4-8MB or more per process.

Couple that with a fixed number of cgi processes on the back end (either with fastcgi or php-fpm) and you can account for most if not all of resources that will be used under any load conditions.

PHP

With Apache gone, so to goes mod_php and mod_fcgid. Neither are ideal solutions to running PHP sites, but those are the breaks (devsrv was running mod_php because it’s was what Ubuntu setup back when I installed it, and mod_fcgid is what Dreamhost uses).

Nginx does things a bit differently. PHP is run as a stand alone CGI “server” that Nginx proxies requests to. I find there are a couple of really nice advantages to this, especially if you can run php-fpm.

For example, you can pool cgi processes based on actual more broadly defined considerations rather than Apache’s process class. Say you have 3 vhosts, each running Wordpress, they can be served by a single pool of php processes that can share resources like a php-apc cache.

Ultimately, this means you can still control the number of backend processes that used for PHP, but can do so while still sharing resources where it makes the most sense.

Figuring and Managing Resources

With a VPS like Dreamhost’s where there are hard limits on memory usage, and since there’s no swap space you can’t really deal with overages when they occur other than to have the watchdog kill your VPS. In short, you really want to be able to account for resource usage and deal with transient spikes in a way that doesn’t result in spiraling resource usage.

Apache has always made this a fun exercise. Yes it can be done, but like I said, it’s fun trying to tune a fuzzy system optimally when there are hard limits.

With Nginx, I can count on the server’s staying exactly, or very nearly exactly, where they were at initialization. 2 worker processes + a controller process nets me between 8 and 12MB of RAM used, and it’ll be that much regardless of whether I have 1 connection or 100 connections. The CGI upstream servers are likewise manageable. With php-fpm you can let it spawn more processes, but you can just as easily limit it to 1 or 2 if resources are scarce. Simply put, it’s much easier to account for how much resources you actually need.

Performance

The whole point of Nginx is performance, but I’m not use to seeing 2 processes handle a lot of requests and that takes some getting use to.

I ran Apache Bench (ab) against my dev server to see just how it performed against Apache setup similarly to how it is in production and the results were certainly impressive to me.

Serving static content (cached HTML files, images, css, js, etc.) 2 Nginx processes could handle more than 2000 connections per second (over 1000mbit network). This is about 2x more than Apache2 was able to handle in about 1/30th the memory foot print.

Serving dynamic content is of course considerably slower, ~3 connections/second can be handled going though the full Wordpress PHP + MySQL stack with 4 php-fpm workers and xcache running.

It’s real hard to completely quantify all the variables, especially when I’m deliberately trying not to.

Conclusions

Okay I admit this post was sparse on details, I’ll try and rectify that in the near future. For now, let me just say, if you can switch to Nginx with your php application you should see better performance and lower resource usage than the same thing running under Apache. I certainly have.