jcalderone

Greetings, and welcome back to "Twisted Web in 60 Seconds". In the previous entry, back at the beginning of December, I promised to cover Twisted Web's proxying capabilities. For various reasons I've decided to dump that topic and cover something else instead. So, prepare to learn about Twisted Web's CGI capabilities!

twisted.web.twcgi.CGIScript and twisted.web.twcgi.FilteredScript are the two most interesting classes in this area. They are both Resource subclasses, so many of the features of resources that I've covered so far apply to them. For example, you can use them as the resource in an .rpy script:

from twisted.web.twcgi import CGIScript
resource = CGIScript("/path/to/date-example.sh")

date-example.sh might look like this:

#!/bin/sh

echo "Content-Type: text/plain"
echo
/bin/date

That is, just a regular CGI - nothing special about Twisted Web going on there.

If you need to specify an interpreter for some reason (for example, the CGI itself isn't set executable, or doesn't specify its own interpreter with #! on the first line), you can use FilteredScript instead of CGIScript:

from twisted.web.twcgi import FilteredScript

resource = FilteredScript("/path/to/date-example.sh")
resource.filter = "/bin/sh"

Set up this way, /bin/sh will always be run and passed an argument of /path/to/date-example.sh.

Welcome back to "Twisted Web in 60 seconds". Over the previous two entries, I introduced Twisted Web's session APIs. This included accessing the session object, storing state on it, and retrieving it later. I described how the Session object has a lifetime which is tied to the notional session it represents. In this installment, I'll describe how you can exert some control over that lifetime and react when it expires.

The lifetime of a session is controlled by the sessionTimeout attribute of the Session class. This attribute gives the number of seconds a session may go without being accessed before it expires. The default is 15 minutes. In this example, I'll show you change that to a different value.

One way to override the value is with a subclass:

  from twisted.web.server import Session

  class ShortSession(Session):
      sessionTimeout = 60

To have Twisted Web actually make use of this session class, rather than the default, it is also necessary to override the sessionFactory attribute of Site. I could do this with another subclass, but I can also do it to just one instance of Site:

  from twisted.web.server import Site

  factory = Site(rootResource)
  factory.sessionFactory = ShortSession

Sessions given out for requests served by this Site will use ShortSession and only last one minute without activity.

You can have arbitrary functions run when sessions expire, too. This can be useful for cleaning up external resources associated with the session, tracking usage statistics, and more. This functionality is provided via Session.notifyOnExpire. It accepts a single argument: a function to call when the session expires. Here's a trivial example which prints a message whenever a session expires:

  from twisted.web.resource import Resource

  class ExpirationLogger(Resource):
      sessions = set()

      def render_GET(self, request):
          session = request.getSession()
          if session.uid not in self.sessions:
              self.sessions.add(session.uid)
              session.notifyOnExpire(lambda: self._expired(session.uid))
          return ""

      def _expired(self, uid):
          print "Session", uid, "has expired."
          self.sessions.remove(uid)

Keep in mind that using a method as the callback will keep the instance (in this case, the ExpirationLogger resource) in memory until the session expires.

With those pieces in hand, here's an example that prints a message whenever a session expires, and uses sessions which last for 5 seconds:

from twisted.web.server import Site, Session
from twisted.web.resource import Resource
from twisted.internet import reactor

class ShortSession(Session):
    sessionTimeout = 5

class ExpirationLogger(Resource):
    sessions = set()

    def render_GET(self, request):
        session = request.getSession()
        if session.uid not in self.sessions:
            self.sessions.add(session.uid)
            session.notifyOnExpire(lambda: self._expired(session.uid))
        return ""

    def _expired(self, uid):
        print "Session", uid, "has expired."
        self.sessions.remove(uid)

rootResource = Resource()
rootResource.putChild("logme", ExpirationLogger())
factory = Site(rootResource)
factory.sessionFactory = ShortSession

reactor.listenTCP(8080, factory)
reactor.run()

Since Site customization is required, this example can't be rpy-based, so it brings back the manual reactor.listenTCP and reactor.run calls. Run it and visit /logme to see it in action. Keep visiting it to keep your session active. Stop visiting it for five seconds to see your session expiration message.

That pretty much wraps things up for Twisted Web's built in session support. Next time I'll cover some of Twisted Web's proxying features.

Welcome to the 16th installment of "Twisted Web in 60 seconds". Last time I introduced the basic APIs for interacting with Twisted Web sessions. In this installment, I'll show you how you can persist objects across requests in the session object.

As I discussed last time, instances of Session last as long as the notional session itself does. Each time Request.getSession is called, if the session for the request is still active, then the same Session instance is returned as was returned previously. Because of this, Session instances can be used to keep other objects around for as long as the session exists.

It's easier to demonstrate how this works than explain it, so here's an example:

  >>> from zope.interface import Interface, Attribute, implements
  >>> from twisted.python.components import registerAdapter
  >>> from twisted.web.server import Session
  >>> class ICounter(Interface):
  ...     value = Attribute("An int value which counts up once per page view.")
  ...
  >>> class Counter(object):
  ...     implements(ICounter)
  ...     def __init__(self, session):
  ...         self.value = 0
  ...
  >>> registerAdapter(Counter, Session, ICounter)
  >>> ses = Session(None, None)
  >>> data = ICounter(ses)
  >>> print data
  <__main__.Counter object at 0x8d535ec>
  >>> print data is ICounter(ses)
  True
  >>>

What?, I hear you say.

What's shown in this example is the interface and adaption based API which Session exposes for persisting state. There are several critical pieces interacting here:

• ICounter is an interface which serves several purposes. Like all interfaces, it documents the API of some class of objects (in this case, just the value attribute). It also serves as a key into what is basically a dictionary within the session object: the interface is used to store or retrieve a value on the session (the Counter instance, in this case).
• Counter is the class which actually holds the session data in this example. It implements ICounter (again, mostly for documentation purposes). It also has a value attribute, as the interface declared.
• The registerAdapter call sets up the relationship between its three arguments so that adaption will do what we want in this case.
• Adaption is performed by the expression ICounter(ses). This is read as adapt ses to ICounter. Because of the registerAdapter call, it is roughly equivalent to Counter(ses). However (because of certain things Session does), it also saves the Counter instance created so that it will be returned the next time this adaption is done. This is why the last statement produces True.

If you're still not clear on some of the details there, don't worry about it and just remember this: ICounter(ses) gives you an object you can persist state on. It can be as much or as little state as you want, and you can use as few or as many different Interface classes as you want on a single Session instance.

With those conceptual dependencies out of the way, it's a very short step to actually getting persistent state into a Twisted Web application. Here's an example which implements a simple counter, re-using the definitions from the example above:

  from twisted.web.resource import Resource

  class CounterResource(Resource):
      def render_GET(self, request):
          session = request.getSession()
          counter = ICounter(session)
          counter.value += 1
          return "Visit #%d for you!" % (counter.value,)

Pretty simple from this side, eh? All this does is use Request.getSession and the adaption from above, plus some integer math to give you a session-based visit counter.

Here's the complete source for an rpy script based on this example:

cache()

from zope.interface import Interface, Attribute, implements
from twisted.python.components import registerAdapter
from twisted.web.server import Session
from twisted.web.resource import Resource

class ICounter(Interface):
    value = Attribute("An int value which counts up once per page view.")

class Counter(object):
    implements(ICounter)
    def __init__(self, session):
        self.value = 0

registerAdapter(Counter, Session, ICounter)

class CounterResource(Resource):
    def render_GET(self, request):
        session = request.getSession()
        counter = ICounter(session)   
        counter.value += 1
        return "Visit #%d for you!" % (counter.value,)

resource = CounterResource()

One more thing to note is the cache() call at the top of this example. As with the previous example where this came up, this rpy script is stateful. This time, it's the ICounter definition and the registerAdapter call that need to be executed only once. If we didn't use cache, every request would define a new, different interface named ICounter. Each of these would be a different key in the session, so the counter would never get past one.

There's one more interesting thing you can do with sessions in Twisted Web right out of the box. Tune in next time to find out what.

Welcome to the 15th installment of "Twisted Web in 60 seconds". As promised, I'll be covering sessions in this installment. Or, more accurately, I'll be covering a tiny bit of sessions. As this is the most complicated topic I've covered so far, I'm going to take a few installments to cover all the different aspects.

In this installment, you can expect to learn the very basics of the Twisted Web session API: how to get the session object for the current request and how to prematurely expire a session.

Before I get into the APIs, though, I should explain the big picture of sessions in Twisted Web. Sessions are represented by instances of Session. The Site creates a new instance of Session the first time an application asks for it for a particular session. Session instances are kept on the Site instance until they expire (due to inactivity or because they are explicitly expired). Each time after the first that a particular session's Session object is requested, it is retrieved from the Site.

With the conceptual underpinnings of the upcoming API in place, here comes the example. This will be a very simple rpy script which tells a user what their unique session identifier is and lets them prematurely expire it.

First, I'll import Resource so I can define a couple subclasses of it:

  from twisted.web.resource import Resource

Next I'll define the resource which tells the client what its session identifier is. This is done easily by first getting the session object using Request.getSession and then getting the session object's uid attribute.

  class ShowSession(Resource):
      def render_GET(self, request):
          return 'Your session id is: ' + request.getSession().uid

To let the client expire their own session before it times out, I'll define another resource which expires whatever session it is requested with. This is done using the Session.expire method.

  class ExpireSession(Resource):
      def render_GET(self, request):
          request.getSession().expire()
          return 'Your session has been expired.'

Finally, to make the example an rpy script, I'll make an instance of ShowSession and give it an instance of ExpireSession as a child using Resource.putChild (covered earlier).

  resource = ShowSession()
  resource.putChild("expire", ExpireSession())

And that is the complete example. You can fire this up and load the top page. You'll see a (rather opaque) session identifier that remains the same across reloads (at least until you flush the TWISTED_SESSION cookie from your browser or enough time passes). You can then visit the expire child and go back to the top page and see that you have a new session.

Here's the complete source for the example.

from twisted.web.resource import Resource

class ShowSession(Resource):
    def render_GET(self, request):
        return 'Your session id is: ' + request.getSession().uid

class ExpireSession(Resource):
    def render_GET(self, request):
        request.getSession().expire()
        return 'Your session has been expired.'

resource = ShowSession()
resource.putChild("expire", ExpireSession())

Next time I'll talk about how you can persist information in the session object.

Welcome to the 14th installment of "Twisted Web in 60 seconds". In many of the previous installments, I've demonstrated how to serve content by using existing resource classes or implementing new ones. In this installment, I'll demonstrate how you can use Twisted Web's basic or digest HTTP authentication to control access to these resources.

Guard, the Twisted Web module which provides most of the APIs which will be used in this example, helps you to add authentication and authorization to a resource hierarchy. It does this by providing a resource which implements getChild to return a dynamically selected resource. The selection is based on the authentication headers in the request. If those headers indicate the request is made on behalf of Alice, then Alice's resource will be returned. If they indicate it was made on behalf of Bob, his will be returned. If the headers contain invalid credentials, an error resource is returned. Whatever happens, once this resource is returned, URL traversal continues as normal from that resource.

The resource which implements this is HTTPAuthSessionWrapper, though it is directly is directly responsible for very little of the process. It will extract headers from the request and hand them off to a credentials factory to parse them according to the appropriate standards (eg HTTP Authentication: Basic and Digest Access Authentication) and then it hands the resulting credentials object off to a portal, the core of Twisted Cred, a system for uniform handling of authentication and authorization. I am not going to discuss Twisted Cred in much depth here. To make use of it with Twisted Web, the only thing you really need to know is how to implement a realm.

You need to implement a realm because the realm is the object which actually decides which resources are used for which users. This can be as complex or as simple as it suitable for your application. For this example, I'll keep it very simple: each user will have a resource which is a static file listing of the public_html directory in their UNIX home directory. First, I need to import implements from zope.interface and IRealm from twisted.cred.portal. Together these will let me mark this class as a realm (this is mostly - but notentirely - a documentation thing). I'll also need File for the actual implementation later.

  from zope.interface import implements

  from twisted.cred.portal import IRealm
  from twisted.web.static import File

  class PublicHTMLRealm(object):
      implements(IRealm)

A realm only needs to implement one method, requestAvatar. This is called after any successful authentication attempt (ie, Alice supplied the right password). Its job is to return the avatar for the user who succeeded in authenticating. An avatar is just an object that represents a user. In this case, it will be a File. In general, with Guard, the avatar must be a resource of some sort.

      def requestAvatar(self, avatarId, mind, *interfaces):
          if IResource in interfaces:
              return (IResource, File("/home/%s/public_html" % (avatarId,)), lambda: None)
          raise NotImplementedError()

A few notes on this method:

• The avatarId parameter is essentially the username. It's the job of some other code to extract the username from the request headers and make sure it gets passed here.
• The mind is always None when writing a realm to be used with Guard. You can ignore it until you want to write a realm for something else.
• Guard always passed IResource for the interfaces parameter. If interfaces only contains interfaces your code doesn't understand, raising NotImplementedError is the thing to do, as above. You'll only need to worry about getting a different interface when you write a realm for something other than Guard.
• If you want to track when a user logs out, that's what the last element of the returned tuple is for. It will be called when this avatar logs out. lambda: None is the idiomatic no-op logout function.
• Notice that I have written the path handling code in this example very poorly. This example may be vulnerable to certain unintentional information disclosure attacks. This sort of problem is exactly the reason FilePath exists. However, that's an example for another day...

We're almost ready to set up the resource for this example. To create an HTTPAuthSessionWrapper, though, we need two things. First, a portal, which requires the realm above, plus at least one credentials checker:

  from twisted.cred.portal import Portal
  from twisted.cred.checkers import FilePasswordDB

  portal = Portal(PublicHTMLRealm(), [FilePasswordDB('httpd.password')])

FilePasswordDB is that credentials checker I mentioned. It knows how to read passwd(5)-style (loosely) files to check credentials against. It is responsible for the authentication work after HTTPAuthSessionWrapper extracts the credentials from the request.

Next we need either BasicCredentialFactory or DigestCredentialFactory. The former knows how to challenge HTTP clients to do basic authentication; the latter, digest authentication. I'll use digest here:

  from twisted.web.guard import DigestCredentialFactory

  credentialFactory = DigestCredentialFactory("md5", "example.org")

The two parameters to this constructor are the hash algorithm and the http authentication realm which will be used. The only other valid hash algorithm is "sha" (but be careful, MD5 is more widely supported than SHA). The http authentication realm is mostly just a string that is presented to the user to let them know why they're authenticating (you can read more about this in the RFC).

With those things created, we can finally instantiate HTTPAuthSessionWrapper:

  from twisted.web.guard import HTTPAuthSessionWrapper

  resource = HTTPAuthSessionWrapper(portal, [credentialFactory])

There's just one last thing that needs to be done here. When I introduced rpy scripts, I mentioned that they're evaluated in an unusual context. This is the first example which actually needs to take this into account. It so happens that DigestCredentialFactory instances are actually stateful. Authentication will only succeed if the same instance is used to generate challenges and examine the responses to those challenges. However, the normal mode of operation for an rpy script is for it to be re-executed for every request. This leads to a new DigestCredentialFactory being created for every request, preventing any authentication attempt from ever succeeding.

There are two ways to deal with this. First, the better of the two ways, I could move almost all of the code into a real Python module, including the code which instantiates the DigestCredentialFactory. This would make ensure the same instance was used for every request. Second, the easier of the two ways, I could add a call to cache to the beginning of the rpy script:

  cache()

cache is part of the globals of any rpy script, so you don't need to import it (it's okay to be cringing at this point). Calling cache makes Twisted re-use the result of the first evaluation of the rpy script for subsequent requests too. Just what I want in this case.

Here's the complete example (with imports re-arranged to the more conventional style):

cache()

from zope.interface import implements

from twisted.cred.portal import IRealm, Portal
from twisted.cred.checkers import FilePasswordDB
from twisted.web.static import File
from twisted.web.resource import IResource
from twisted.web.guard import HTTPAuthSessionWrapper, DigestCredentialFactory

class PublicHTMLRealm(object):
    implements(IRealm)

    def requestAvatar(self, avatarId, mind, *interfaces):
        if IResource in interfaces:
            return (IResource, File("/home/%s/public_html" % (avatarId,)), lambda: None)
        raise NotImplementedError()

portal = Portal(PublicHTMLRealm(), [FilePasswordDB('httpd.password')])

credentialFactory = DigestCredentialFactory("md5", "localhost:8080")
resource = HTTPAuthSessionWrapper(portal, [credentialFactory])

And voila, a password-protected per-user Twisted Web server.

I've gotten several requests to write something about sessions, so there's a good chance that's what you'll find in the next installment.

Welcome to the 13th installment of Twisted Web in 60 seconds. For a while, I've been writing about how you can implement pages by working with the Twisted Web resource model. The very first example I showed you used an existing Resource subclass to serve static content from the filesystem. In this installment, I'll show you how to use WSGIResource, another existing Resource subclass which lets you serve WSGI applications in a Twisted Web server.

First, a few things about WSGIResource. It is a multithreaded WSGI container. Like any other WSGI container, you can't do anything asynchronous in your WSGI applications, even though this is a Twisted WSGI container. In the latest release of Twisted as of this post, 8.2, WSGIResource also has a few significant bugs. These are fixed in trunk (and the fixes will be included in 9.0), so if you want to play around with WSGI in any significant way, you probably want trunk for now.

The first new thing in this example is the import of WSGIResource:

  from twisted.web.wsgi import WSGIResource

Nothing too surprising there. We still need one of the other usual suspects, too:

  from twisted.internet import reactor

You'll see why in a minute. Next, we need a WSGI application. Here's a really simple one just to get things going:

  def application(environ, start_response):
      start_response('200 OK', [('Content-type', 'text/plain')])
      return ['Hello, world!']

If this doesn't make sense to you, take a look at one of these fine tutorials. Otherwise, or once you're done with that, the next step is to create a WSGIResource instance - as this is going to be another rpy script example.

  resource = WSGIResource(reactor, reactor.getThreadPool(), application)

I need to dwell on this line for a minute. The first parameter passed to WSGIResource is the reactor. Despite the fact that the reactor is global and any code that wants it can always just import it (as, in fact, this rpy script simply does itself), passing it around as a parameter leaves the door open for certain future possibilities. For example, having more than one reactor. There are also testing implications. Consider how much easier it is to unit test a function that accepts a reactor - perhaps a mock reactor specially constructed to make your tests easy to write ;) - rather than importing the real global reactor. Anyhow, that's why WSGIResource requires you to pass the reactor to it.

The second parameter passed to WSGIResource is a thread pool. WSGIResource uses this to actually call the application object passed in to it. To keep this example short, I'm passing in the reactor's internal threadpool here, letting me skip its creation and shutdown-time destruction. For finer control over how many WSGI requests are served in parallel, you may want to create your own thread pool to use with your WSGIResource. But for simple testing, using the reactor's is fine (although I'm cheating here a little - I apologize - getThreadPool is a new API, not present in 8.2: you need trunk for this example to work; please ask Chris Armstrong to release 9.0 already).

The final argument is the application object. This is pretty typical of how WSGI containers work.

The example, sans interruption:

  from twisted.web.wsgi import WSGIResource
  from twisted.internet import reactor

  def application(environ, start_response):
      start_response('200 OK', [('Content-type', 'text/plain')])
      return ['Hello, world!']

  resource = WSGIResource(reactor, reactor.getThreadPool(), application)

Up to the point where the WSGIResource instance defined here exists in the resource hierarchy, the normal resource traversal rules apply - getChild will be called to handle each segment. Once the WSGIResource is encountered, though, that process stops and all further URL handling is the responsibility of the WSGI application. Of course this application does nothing with the URL, so you won't be able to tell that.

Oh, and as was the case with the first static file example, there's also a command line option you can use to avoid a lot of this. If you just put the above application function, without all of the WSGIResource stuff, into a file, say, foo.py, then you can launch a roughly equivalent server like this:

  $ twistd -n web --wsgi foo.application

Tune in next time, when I'll discuss HTTP authentication.

Welcome to the twelfth installment of "Twisted Web in 60 seconds". The previous installment created a server which dealt with response errors by aborting response generation, potentially avoiding pointless work. However, it did this silently for any error. In this installment, I'll modify the previous example so that it logs each failed response.

This example will use the Twisted API for logging errors. As I mentioned in the first post covering Deferreds, errbacks are passed an error. In the previous example, the _responseFailed errback accepted this error as a parameter but ignored it. The only way this example will differ is that this _responseFailed will use that error parameter to log a message.

This example will require all of the imports required by the previous example, which I will not repeat here, plus one new import:

  from twisted.python.log import err

The only other part of the previous example which changes is the _responseFailed callback, which will now log the error passed to it:

      def _responseFailed(self, failure, call):
          call.cancel()
          err(failure, "Async response demo interrupted response")

I'm passing two arguments to err here. The first is the error which is being passed in to the callback. This is always an object of type Failure, a class which represents an exception and (sometimes, but not always) a traceback. err will format this nicely for the log. The second argument is a descriptive string that tells someone reading the log what the source of the error was.

Here's the full example with the two above modifications:

from twisted.web.resource import Resource
from twisted.web.server import NOT_DONE_YET
from twisted.internet import reactor
from twisted.python.log import err

class DelayedResource(Resource):
    def _delayedRender(self, request):
        request.write("Sorry to keep you waiting.")
        request.finish()

    def _responseFailed(self, failure, call):
        call.cancel()
        err(failure, "Async response demo interrupted response")

    def render_GET(self, request):
        call = reactor.callLater(5, self._delayedRender, request)
        request.notifyFinish().addErrback(self._responseFailed, call)
        return NOT_DONE_YET

resource = DelayedResource()

Run this server (see the end of the previous installment if you need a reminder about how to do that) and interrupt a request. Unlike the previous example, where the server gave no indication that this had happened, you'll see a message in the log output with this version.

Next time I'll show you about a resource that lets you host WSGI applications in a Twisted Web server.

Welcome to the eleventh installment of "Twisted Web in 60 seconds". Previously I gave an example of a Resource which generates its response asynchronously rather than immediately upon the call to its render method. When generating responses asynchronously, the possibility is introduced that the connection to the client may be lost before the response is generated. In such a case, it is often desirable to abandon the response generation entirely, since there is nothing to do with the data once it is produced. In this installment, I'll show you how to be notified that the connection has been lost.

This example will build upon the example from installment nine which simply (if not very realistically) generated its response after a fixed delay. I will expand that resource so that as soon as the client connection is lost, the delayed event is canceled and the response is never generated.

The feature this example relies on is provided by another Request method: notifyFinish. This method returns a new Deferred which will fire with None if the request is successfully responded to or with an error otherwise - for example if the connection is lost before the response is sent.

The example starts in a familiar way, with the requisite Twisted imports and a resource class with the same _delayedRender used previously:

  from twisted.web.resource import Resource
  from twisted.web.server import NOT_DONE_YET
  from twisted.internet import reactor

  class DelayedResource(Resource):
      def _delayedRender(self, request):
          request.write("<html><body>Sorry to keep you waiting.</body></html>")
          request.finish()

Before defining the render method, I'm going to define an errback (an errback being a callback that gets called when there's an error), though. This will be the errback attached to the Deferred returned by Request.notifyFinish. It will cancel the delayed call to _delayedRender.

      def _responseFailed(self, err, call):
          call.cancel()

Finally, the render method will set up the delayed call just as it did before, and return NOT_DONE_YET likewise. However, it will also use Request.notifyFinish to make sure _responseFailed is called if appropriate.

      def render_GET(self, request):
          call = reactor.callLater(5, self._delayedRender, request)
          request.notifyFinish().addErrback(self._responseFailed, call)
          return NOT_DONE_YET

Notice that since _responseFailed needs a reference to the delayed call object in order to cancel it, I passed that object to addErrback. Any additional arguments passed to addErrback (or addCallback) will be passed along to the errback after the Failure instance which is always passed as the first argument. Passing call here means it will be passed to _responseFailed, where it is expected and required.

That covers almost all the code for this example. Here's the entire example without interruptions, as an rpy script:

from twisted.web.resource import Resource
from twisted.web.server import NOT_DONE_YET
from twisted.internet import reactor

class DelayedResource(Resource):
    def _delayedRender(self, request):
        request.write("Sorry to keep you waiting.")
        request.finish()

    def _responseFailed(self, err, call):
        call.cancel()

    def render_GET(self, request):
        call = reactor.callLater(5, self._delayedRender, request)
        request.notifyFinish().addErrback(self._responseFailed, call)
        return NOT_DONE_YET

resource = DelayedResource()

Toss this into example.rpy, fire it up with twistd -n web --path ., and hit http://localhost:8080/example.rpy. If you wait five seconds, you'll get the page content. If you interrupt the request before then, say by hitting escape (in Firefox, at least), then you'll see perhaps the most boring demonstration ever - no page content, and nothing in the server logs. Success!

Next time I'll digress slightly to cover the basics of Twisted logging and expand this example to use it to show when clients fail to receive the response they requested.

Welcome to the tenth installment of "Twisted Web in 60 90 seconds". Previously I gave an example of a Resource which generates its response asynchronously rather than immediately upon the call to its render method. Though it was a useful demonstration of the NOT_DONE_YET feature of Twisted Web, the example itself didn't reflect what a realistic application might want to do. In this installment, I'll introduce Deferred, the Twisted class which is used to provide a uniform interface to many asynchronous events, and show you an example of using a Deferred-returning API to generate an asynchronous response to a request in Twisted Web¹.

Deferred is the result of two consequences of the asynchronous programming approach. First, asynchronous code is frequently (if not always) concerned with some data (in Python, an object) which is not yet available but which probably will be soon. Asynchronous code needs a way to define what will be done to the object once it does exist. It also needs a way to define how to handle errors in the creation or acquisition of that object. These two needs are satisfied by the callbacks and errbacks of a Deferred. Callbacks are added to a Deferred with Deferred.addCallback; errbacks are added with Deferred.addErrback. When the object finally does exist, it is passed to Deferred.callback which passes it on to the callback added with addCallback. Similarly, if an error occurs, Deferred.errback is called and the error is passed along to the errback added with addErrback. Second, the events that make asynchronous code actually work often take many different, incompatible forms. Deferred acts as the uniform interface which lets different parts of an asynchronous application interact and isolates them from implementation details they shouldn't be concerned with.

That's almost all there is to Deferred. To solidify your new understanding, now consider this rewritten version of DelayedResource which uses a Deferred-based delay API. It does exactly the same thing as the previous example. Only the implementation is different.

First, the example must import that new API I just mentioned, deferLater:

  from twisted.internet.task import deferLater

Next, all the other imports (these are the same as last time):

  from twisted.web.resource import Resource
  from twisted.web.server import NOT_DONE_YET
  from twisted.internet import reactor

With the imports done, here's the first part of the DelayedResource implementation. Again, this part of the code is identical to the previous version:

  class DelayedResource(Resource):
      def _delayedRender(self, request):
          request.write("<html><body>Sorry to keep you waiting.</body></html>")
          request.finish()

Next I also need to define the render method. Here's where things change a bit. Instead of using callLater, I'm going to use deferLater this time. deferLater accepts a reactor, delay (in seconds, as with callLater), and a function to call after the delay to produce that elusive object I was talking about above in my description of Deferreds. I'm also doing to use _delayedRender as the callback to add to the Deferred returned by deferLater. Since it expects the request object as an argument, I'm going to set up the deferLater call to return a Deferred which has the request object as its result.

      def render_GET(self, request):
          d = deferLater(reactor, 5, lambda: request)

The Deferred referenced by d now needs to have the _delayedRender callback added to it. Once this is done, _delayedRender will be called with the result of d (which will be request, of course — the result of (lambda: request)()).

          d.addCallback(self._delayedRender)

Finally, the render method still needs to return NOT_DONE_YET, for exactly the same reasons as it did in the previous version of the example.

          return NOT_DONE_YET

And with that, DelayedResource is now implemented based on a Deferred. The example still isn't very realistic, but remember that since Deferreds offer a uniform interface to many different asynchronous event sources, this code now resembles a real application even more closely; you could easily replace deferLater with another Deferred-returning API and suddenly you might have a resource that does something useful.

Finally, here's the complete, uninterrupted example source, as an rpy script:

from twisted.internet.task import deferLater
from twisted.web.resource import Resource
from twisted.web.server import NOT_DONE_YET
from twisted.internet import reactor

class DelayedResource(Resource):
    def _delayedRender(self, request):
        request.write("Sorry to keep you waiting.")
        request.finish()

    def render_GET(self, request):
        d = deferLater(reactor, 5, lambda: request)
        d.addCallback(self._delayedRender)
        return NOT_DONE_YET

resource = DelayedResource()

¹I know I promised an example of handling lost client connections, but I realized that example would also involve Deferreds, so I wanted to introduce Deferreds by themselves first. Tune in next time for the example I told you I'd show you this time.

• serve static content from a directory
• generate a page dynamically
• static URL dispatch
• dynamic URL dispatch
• error handling
• custom response codes
• handling POSTs
• rpy scripts (or, how to save yourself some typing)
• asynchronous responses
• asynchronous responses (via Deferred)
• interrupted responses
• logging errors
• WSGI
• HTTP authentication
• session basics
• storing objects in the session
• session endings
• CGI