blog.sendapatch.se

pylibmc 1.2.3 on PyPI

An incremental release has been born!

• greater test coverage
• bug fixes and clean-up
• performance enhancements
• portability improvements

Why upgrade? Because you want to.

pylibmc 1.3.0 is around the corner, with new features! Come hang out in #sendapatch and say what you would want in the new version.

pylibmc 1.2.1 has been released!

Bug fix release, mitigates issues with Python 2.5 support, tests and NUL byte keys.

Some hilights from 1.2.0:

• Fixed GIL issues
• CAS support (ketralnis)
• SASL authentication (Remoun)
• Added auto_eject, num_replicas, remove_failed behaviors
• Improved tests
• Published documentation source

pylibmc 1.2.0 is out!

This release is for the people behind reddit.com, for helping push development forward. Keep doing your thing.

• sevmer.org versioning scheme
• Fixed GIL issues
• Added CAS support (ketralnis)
• Added SASL authentication (Remoun)
• Added more detail to errors (spladug)
• Added mapping-like behavior for clients
• Fixed build errors on Mac OS X
• Moved to nose for testing
• Added auto_eject behavior
• Added num_replicas behavior
• Added remove_failed behavior
• Removed cache_lookups behavior
• Improved repr of clients (noah256)
• Improved IPv6 support (JshWright)
• Improved pooling behavior so it doesn't cause lock-ups
• Improved tests and testing foundation
• Improved documentation and structure
• Internalized Sphinx documentation
• Bunch of other stuff

Fellow gentlemen and women—no wait, who am I kidding—I just released version 1.1.1 of my library pylibmc.

pylibmc is a wrapper around libmemcached, the most awesome memcached client library in C there is, as a Python C extension written by hand.

What's new? GIL handling has been fixed, which should solve issues for most people in multi-thread deployments (though who uses threads these days I wonder), and support for libmemcached 0.40 has been added.

For downloading, and a more detailed changelog, head on over to PyPI

pylibmc 1.1.1 on PyPI

And for a more exhaustive documentation on installing and using, see the pylibmc documentation.

Note: pylibmc 1.1.1 was released to mitigate an issue with running the test suite on Python 2.5.

My project pylibmc, the memcached client C extension for Python, has now reached the milestones I set out for it to go 1.0. So I'm making a 1.0 alpha release, i.e. what others might call a release candidate.

What does this mean? Normally, it would for a library mean that the API is considered stable. But for a project that has as a primary objective to mimic an already-existing API, that stability guarantee has already been made.

However, what I want you to do is to test pylibmc out on as many platforms as you can, file as many bug reports as you can over at the GitHub issue tracker thing.

pylibmc 1.0-alpha over at PyPI

Oh, and happy 2010, folks.

It's actually pretty easy to get some coverage statistics of C code, even with Python. You've just gotta grip it.

First, you shouldn't write half a script duplicating Tools/freeze/freeze.py in the Python distribution. Freezing for C libraries equals statically compiling! It's true.

Here's what I did, rougly:

1. Copy my C extension's C file and header to Modules/;
2. edit Modules/Setup.local, include a line saying foo foomodule.c -lblah;
3. make clean
4. ./config.status --recheck
5. make coverage
6. ./python.exe ~/path/to/tests.py
7. gcov -o foomodule

So by the way, I have about 65% test coverage in pylibmc.

A big hand goes out to me. And not for releasing a new version of my own software, but for forgetting to tell anyone about it!

pylibmc 0.9.1 adds a get_stats method, some more compatibility niceness, and the most oft requested feature: per-error exceptions.

Get it while it's hot! Wait, right, I missed my release announcement.

And this is what I mean by "per-error exceptions":

>>> import pylibmc
>>> mc = pylibmc.Client(["127.0.0.1"])
>>> try:
...   mc.incr("boegen", 123)
... except pylibmc.NotFound, e:
...   print "Oi!", e
... 
Oi! error 16 from memcached_increment: NOT FOUND
>>> 

It's that time again — pylibmc release time! The memcached client library all true Pythonistas must have.

This fine release brings two new amazingly useful features, one is pooling and one is the binary protocol.

The pooling is implemented in pure Python, and is described in detail in previous posts, and is immensely useful.

The other feature, the binary protocol, is all from the TangentOrg guys really, I just added an interface to it (that's what I do, so hey). Quite easily:

import pylibmc
mc = pylibmc.Client(["127.0.0.1"], binary=True)
vals = mc.get_multi(("abc", "def", "ghi"))

pylibmc 0.8.2 at PyPI

Yeah, I did it again - missed files in the manifest. But I've worked up a solution, so it won't happen again. :-)

Also, works fine with libmemcached 0.33.

In my previous post I mentioned having a pooling API in pylibmc, and then I got an idea. I've implemented it, and it's now up on Github.

Here's an example using the polymorphic API, and I think it's pretty self-explanatory:

import pylibmc
import threading

class Incrementer(threading.Thread):
    key = "my_key"
    limit = 10 ** 6  # A million

    def __init__(self, pool):
        super(Incrementer, self).__init__()
        self.pool = pool

    def increment(self):
        with self.pool.reserve() as mc:
            self.running = mc.incr(self.key) < self.limit

    def run(self):
        self.running = True
        while self.running:
            self.increment()

n_threads = 12
mc = pylibmc.Client(["127.0.0.1"])
mc.set("my_key", 1)

# This is the classical method, libmemcachedutil-style.
#pool = pylibmc.ClientPool()
#pool.fill(mc, n_threads)

# This is the more flexible style, adapted to threading.
pool = pylibmc.ThreadMappedPool(mc)

ts = [Incrementer(pool) for i in xrange(n_threads)]
map(Incrementer.start, ts)
map(Incrementer.join, ts)

print mc.get(Incrementer.key)
print pool

Duly note that the ThreadMappedPool won't be accessible unless your Python has threading enabled. But why would you be doing pooling without threads, I ask you, sillybear.

I was discussing how to implement pooling for pylibmc when I realized what libmemcachedutil's pooling is - or rather, what it isn't.

It's not a magical solution for concurrently doing anything at all, it's not anything like that -- it just helps you with thread-safety.

In Python, however, we've got the global interpreter lock, the GIL. This lock must always be held by the thread that is dealing with anything Python. The Python interpreter itself isn't thread-safe, or rather, it is with the GIL.

This means that whenever Python code is running, you'll be sure to have exclusive access to all of Python's memory (unless something is misbehaving.) In turn, this means that the usecase for using libmemcachedutil in a Python library is rather slim.

Let's have a look at some code for doing the equivalent in pure Python. This is a Werkzeug-based WSGI application which would be run in multiple threads, concurrently:

# Configuration
n_threads = 12
mc_addrs = "10.0.1.1", "10.0.1.2", "10.0.1.3"
mc_pool_size = n_threads

# Application
import pylibmc
from contextlib import contextmanager
from pprint import pformat
from werkzeug import Request, Response
from werkzeug.exceptions import NotFound

class ClientPool(list):
    @contextmanager
    def reserve(self):
        mc = self.pop()
        try:
            yield mc
        finally:
            self.append(mc)

mc = pylibmc.Client(mc_addrs)
mc_pool = ClientPool(mc.clone() for i in xrange(mc_pool_size))

@Request.application
def my_app(request):
    with mc_pool.reserve() as mc:
        key = request.path[1:].encode("ascii")
        val = mc.get(key)
        if not val:
            return NotFound(key)
        return Response(pformat(val))

It's fully-functional example of how one could implement pooling with pylibmc, and very much so in the same way that people do with libmemcachedutil. To start it, you could use Spawning like so: spawn -s 1 -t 12 my_wsgi_app.my_app

I don't know if I think the above methodology is the best one though, another possibility is to have a dict with thread names as keys and client objects for values, then, each thread would look up its own client object in the dict on each request, and if none exists, it clones a master just like the pooling thing above.

It'd be neat if there was a generic Python API for doing any variant of pooling, per-thread or the list-based version, and then you'd be able to switch between them seamlessly. Hm.

I found a critical bug in the behaviors manipulation of pylibmc, so I fixed that and here's the new release.

pylibmc 0.7.4 at PyPI

If you'd like to strike up a conversation with a handsome fellow like me, join #sendapatch on freenode.

Notably, this release does compile with libmemcached 0.32. However, due to a function signature change, you'll get a warning regarding types -- this is completely harmless as it's unsigned int to size_t, which is the exact same size, so there should be no issues there. I'll change the call signature in the next version, kept the old because I think most people will be using the older declaration.

So, the seventh minor version of my Python wrapper for libmemcached client has been released. The joy!

amix.dk was nice enough to take the time to both debug a long-withstanding bug and make speed comparisons to other libraries! Isn't that nice?

Update: I released a point one, because the manifest was old and the release contained code not suited for everyone (TCP_NODELAY set to 1).

Update²: I released a dot two, because the test runner wouldn't really test the correct module.

Update³: I released a spot three, because the C code broke on x86_64.

pylibmc 0.7.3 at PyPI

Also don't forget to stop by in #sendapatch on freenode!

My Python bindings for the memcached client library libmemcached is now on GitHub, for your forking needs.

They're fast as hell too, just like the C-level library is.

And guess what? It doesn't even use ctypes!

pylibmc on GitHub