ProFTPD mini-HOWTO - ProFTPD and Memcache

## ProFTPD and Memcache

What is Memcache?
Memcache (or "memcached") is an open-source, high performance memory object caching system. A simple (and effective) key/value store accessible, efficiently, over the network.

How Can Memcache Be Useful for ProFTPD?
Like any high-performance object store, `memcached` offers several possibilities to a server like `proftpd`. Many sites use `memcached` for caching; it can also be used as an efficient shared storage mechanism, for sharing data among many different servers. And for ProFTPD specifically, the shared storage aspect is what is most useful. Things like SSL/TLS sessions can be cached and shared across a pool of `proftpd` servers, as can ban lists for badly-behaved clients.

Enabling Memcache Support for ProFTPD
OK, so you are interested enough in the possibilities that `memcached` offers that you want to try it out. Excellent! To do this, you will first need to make sure to build your `proftpd` executable using the `--enable-memcache` configure option. The `--enable-memcache` configure option automatically adds the `mod_memcache` module to your `proftpd` build.

The `mod_memcache` module uses the `libmemcached` library for talking to `memcached` servers. If your `libmemcached` library is installed in a non-standard location, you may need to tell the ProFTPD build system where to find the `libmemcached` header files and libraries using the `--with-includes` and `--with-libraries` configure options.

There are other modules which make use of memcached support when available, such as `mod_tls_memcache`. Thus to take advantage of modules like this, putting everything together, your configure command might look like this:

      $ ./configure --enable-memcache \
        --with-modules=...:mod_tls_memcache:... \
        --with-includes=/path/to/libmemcached/include \
        --with-libraries=/path/to/libmemcached/lib

Configuring `mod_memcache`
Now that you have compiled `proftpd` with the `mod_memcache` module, you need to add the necessary `mod_memcache` directives to your `proftpd.conf`. The following example demonstrates this:

      <IfModule mod_memcache.c>
        # Enable mod_memcache
        MemcacheEngine on

        # Tell mod_memcache where to log its messages
        MemcacheLog /path/to/proftpd/memcache.log

        # Tell mod_memcache where to find the memcached servers
        MemcacheServers 192.168.0.10:11211 192.168.0.11:11211
      </IfModule>

If you wish to see more detailed logging, at least while you are setting up your memcached servers for ProFTPD, you can enable trace logging for the `memcache` trace channel using e.g.:

      TraceLog /path/to/proftpd/trace.log
      Trace DEFAULT:10 memcache:20

Using Memcache for Shared Storage
You have now compiled support for memcached into proftpd, and you have told the `mod_memcache` module where to find your `memcached` servers. Is that all you need to do? No. Now you need to tell `proftpd` modules which bits of data to store in your `memcached` servers.

Currently, only two modules can take advantage of `memcached` support: `mod_ban` and `mod_tls_memcache`.

First, let us examine `mod_ban` and how it would use `memcached`. The `mod_ban` module manages ban lists, lists of clients/users which have been banned for various reasons. These lists are stored in shared memory by default; this works for a single `proftpd` server, but if a badly behaved client is banned by one `proftpd` server in pool of servers, that client can then connect to a different server which might not have a ban for that client -- and the client then gets another chance to be naughty. To configure `mod_ban` so that it stores its ban lists in `memcached`, simply use the following in your `proftpd.conf`:

      <IfModule mod_ban.c>
        BanEngine on

        # ...other mod_ban directives...

        # Tell mod_ban to store its ban lists using memcache
        BanCache memcache
      </IfModule>

With this, `mod_ban` will use `memcached` (as well as shared memory) for reading/writing its ban lists. And this, in turn, means that other `proftpd` servers' `mod_ban` modules can see those bans, and reject the badly behaved clients across the pool/cluster.

The `mod_tls_memcache` module uses `memcached` servers for storing SSL/TLS sessions; SSL/TLS session caching can greatly improve SSL/TLS session handshake times, particularly for data transfers using SSL/TLS. If you have a pool of `proftpd` servers, and you have FTPS clients which may connect to a different node every time, caching the SSL/TLS session data in a shared storage mechanism like `memcached` can be quite beneficial.

To use `memcached` for SSL/TLS session caching, then, you use the `TLSSessionCache` directive of the `mod_tls` module, using something like this in your `proftpd.conf`:

      <IfModule mod_tls.c>
        TLSEngine on

        # ...other mod_tls directives...

        <IfModule mod_tls_memcache.c>
          # Tell mod_tls to cache sessions using memcached
          TLSSessionCache memcache:
        </IfModule>
      </IfModule>

That's it. The `mod_tls` module now knows to give the SSL/TLS session data to `mod_tls_memcache`, and `mod_tls_memcache` knows how to talk to the `memcached` servers using `mod_memcache`.

Frequently Asked Questions
Question: If I don't use memcache, are there other ways for sharing data (such as ban lists) among different `proftpd` instances?
Answer: Not really. It might be possible using `mod_sql` and some `SQLLogInfo` directives, but that would only work for very specific information. For sharing things like ban lists and SSL/TLS sessions across a cluster of `proftpd` servers, memcache support is required.

Question: Can I use `mod_memcache` to cache rerequently accessed files, similar to `nginx+memcache`?
Answer: No. And in reality, caching of files like that will probably not give you the same performance gain for FTP transfers as it can for HTTP transfers.

Why not? Many HTTP transfers are for dynamically generated pages; the cost of generating each page is expensive, and the generated content may not change that frequently (relative to the rate of requests). FTP transfers, by contrast, are for static files; FTP servers do not (usually) dynamically generate the bytes of the files being downloaded. The cost of reading files from disk is probably less than reading files from `memcached` over the network, even a LAN.

Now the above may not be true in all cases -- there may be FTP servers serving files from network-mounted filesystems (e.g. NFS, CIFS et al). And for these very specific cases, having a cache of frequently access files on closer storage such as local disk (or `memcached`) could make a big difference; please contact the ProFTPD Project if you find yourself in this situation, and we will see what can be done to help.

Question: Why do I see the following error when `proftpd` starts up?

    mod_tls_memcache/0.1: notice: unable to register 'memcache' SSL session cache: Memcache support not enabled

Answer: This message means that your `proftpd` server has `mod_tls_memcache` built and loaded, but your `proftpd` server was not built with memcache support (i.e. the `--enable-memcache` configure option was not used when compiling `proftpd`).

The above is not a fatal or worrisome error; it is merely pointing out that some of your modules want to use a feature that was not enabled.

$Date: 2013-05-09 05:28:28 $