The gevent project has allowed us to transparently bolt on asynchronous functionality to otherwise synchronous python WSGI based web frameworks and servers. However, piecing together this puzzle for yourself may prove less than convenient as libraries and tutorials remain incomplete, skipping over many pertinent details. To remedy this confusion we'll put together a starter project combining gevent, django and socket.io to build a simple asynchronous application, pausing along the journey to discuss the relevant details. But first we'll detour to discuss event loops, in particular how gevent's implementation runs.
There was a time when Python's various virtualization tools were enough to fit my needs. However, as I continued to build increasingly complex applications which relied on a variety of other system services, I started to notice moving from project to project was difficult. Each project required unique service configurations, and moving between them was a headache. What I needed was a way to virtualize the entire system, which is when I turned to Vagrant. After using Vagrant for the past few years, I've developed a simple template that helps me spin up new Django environments quickly.
Data processing is a common task in development which can often be solved through processing pipelines. Like the name implies, a pipeline is made up of a series of pipes where the output type of one pipe is the input type for its adjacent pipe. A simple, light weight, and powerful language construct for building pipes are coroutines, which can suspend and resume operation while maintaining state. If you're developing in a language with native coroutine support—like Python—then you can put that scaffolding to work developing the processing pipeline of your dreams. However, if you're not fortunate enough to have access to a native coroutine construct, they still serve as a great source for inspiration when developing your own solution.
While working on a small command line Java application (which will appear in a later post) I found myself needing to concurrently save lots and lots of files to a ZIP archive. Prior to the NIO package that debuted in Java 7 this wasn't something easily done. In this post we'll use some of the fundamental APIs from the NIO package along with concurrency utilities to quickly traverse a directory tree and copy file contents into a ZIP archive. However, to truly appreciate the speed boost from NIO's asynchronous file channels, we'll first visit and benchmark a more "vintage" approach leveraging the java.util.zip and java.io packages.