Jp Calderone

April - May Reading List

Thu, 04 Jun 2009 13:52:42 GMT

Green Mars. Kim Stanley Robinson.
Blue Mars. Kim Stanley Robinson.

Note To Self: co_filename vs file

Tue, 26 May 2009 22:38:02 GMT

Python's traceback formatting code uses the co_filename attribute of code objects. However tempting the __file__ attribute may seem, don't use it if you want to get results that agree with the built-in traceback formatting code.

January through March Reading List

Thu, 02 Apr 2009 14:16:25 GMT

House of Leaves. Mark Danielewski.
Jhegaala. Steven Brust.
Red Mars. Kim Stanley Robinson.

Three Software Failures

Wed, 11 Feb 2009 22:29:08 GMT

I ran out of space on / on my desktop today. As a result of this three obviously wrong things happened. First, joe truncated a file to zero bytes and then segfaulted. Second, emacs truncated a file I was working on to zero bytes. Third, sticky notes lost all of my notes. If you write and release software that behaves like this, you need to be embarrassed.

September through December reading list

Thu, 08 Jan 2009 03:03:33 GMT

Dogs of God: Columbus, the Inquisition, and the Defeat of the Moors. James Reston Jr.
Zero: The Biography of a Dangerous Idea. Charles Seife.
The Best Of H.P. Lovecraft. H.P. Lovecraft.
The Given Day. Dennis Lehane.
The Sun Kings. Stuart Clark.
1453: The Holy War for Constantinople and the Clash of Islam and the West. Roger Crowley.

What is the SigQ field of /proc/PID/status in Linux?

Wed, 10 Dec 2008 17:15:18 GMT

I had to read kernel source to figure this out, and when I did I learned the value isn't at all what I thought, so I thought I'd let everyone else know.

From fs/proc/array.c:

    qsize = atomic_read(&p->user->sigpending);
    ...
    buffer += sprintf(buffer, "SigQ:\t%lu/%lu\n", qsize, qlim);

where p is a struct task_struct p. From include/linux/sched.h:

struct task_struct {
    ...
    /* process credentials */
    ...
    struct user_struct *user;

and

/*
 * Some day this will be a full-fledged user tracking system..
 */
struct user_struct {
    ...
    atomic_t sigpending;    /* How many pending signals does this user have? */

So the SigQ value in each process's status file is the total number of signals pending delivery across all processes owned by the UID of the process (or maybe the UID which started the process, I don't know how the user field of task_structs interacts with setuid).

Pretty far from what I've been assuming it meant, the number of signals pending delivery for each individual process. :/

ioctl TIOCSTTY EIO

Tue, 09 Dec 2008 17:45:06 GMT

What does it mean if a TIOCSTTY ioctl fails with EIO (errno 5) on Linux?

August reading list

Mon, 15 Sep 2008 00:17:47 GMT

Singularity Sky. Charles Stross.
Anvil of Stars. Greg Bear.
Dune. Frank Herbert.

Python and XPath

Wed, 06 Aug 2008 23:42:08 GMT

Quite some while ago, I decided to experiment with writing unit tests for some Nevow-based application code using XPath instead of DOM. DOM-based assertions in unit tests tend to be quite verbose compared to the XPath equivalent. Beyond that, though, the XPath seems to be able to express the intent of the test more accurately. It doesn't rely on as many irrelevant details (like how many TextElements some CDATA is broken up into) - or at least that's the idea I wanted to try out.

Unfortunately I only wrote a few tests like this and shortly afterwards I started focusing on non-XML tasks. As far as the experiment went, I think it went well, but it's basically incomplete at this time.

That's not what this post is about.

A while ago, Ubuntu's Hardy Heron was released. In Hardy, the Python package xml.xpath has vanished. Apparently the package was unmaintained for some time and either Ubuntu or Debian or some combination of the two decided that it would no longer be supported. Sad for me, because now all my XPath-based tests are broken on Ubuntu, which happens to be my primary platform. It seems there are some other XPath libraries for Python available in Hardy, so today I tried to update my tests so that they will work on systems which still have xml.xpath but can also try to use a newer XPath library if xml.xpath is unavailable.

Unfortunately (again) this turned out to be more difficult than I expected. The fall-back library I selected is lxml.etree. The first problem I encountered is that the Python API for processing XPath in lxml.etree is incompatible with the API in xml.xpath. So instead of just changing a couple imports as I had hoped I would be able to do, I wrote a couple thin wrappers to expose both APIs in the same way. The second problem I encountered is that the result objects returned by lxml.etree have a different API than the result objects returned by xml.xpath. So I changed my thin wrappers to be more specific, extracting the exact data my unit tests required, rather than being somewhat more general and letting each test grab what it needed. This meant widening the API to two wrappers from one.

try:
    from xml.xpath import Evaluate
except ImportError:
    from lxml.etree import XPath, fromstring
    def evaluateXPath(path, document):
        return XPath(path).evaluate(fromstring(document))
    def evaluateTextXPath(path, document):
        return evaluateXPath(path, document)[0]
else:
    def evaluateXPath(path, document):
        return Evaluate(path, minidom.parseString(document))
    def evaluateTextXPath(path, document):
        return evaluateXPath(path, document)[0].wholeText

After solving these two problems, I had two unit tests passing on my Hardy system. Not a complete failure. I have more than two unit tests that use XPath, though. As it turns out, the next set of tests I looked at need to examine the values of attributes of nodes. This is another area where xml.xpath and lxml.etree present different APIs. If I want to make these tests pass, I need to write another wrapper function to expose attributes in a uniform manner. Here's where I give up on fixing this for the moment. It seems it's not reasonable to expect to transparently switch between xml.xpath and lxml.etree.

Perhaps there's another XPath library out there I can use instead of lxml.etree. Anyone have any tips they'd like to share?

May through July Reading List

Sun, 27 Jul 2008 21:19:28 GMT

Against The Day. Thomas Pynchon.
Nature Girl. Carl Hiaasen.
HaltinG StatE. Charles Stross.

TCP in the browser?

Tue, 01 Jul 2008 14:12:49 GMT

Allen brought to my attention a blog post by Michael Carter about how Orbited now offers a JavaScript API for making TCP connections. I'm not sure how useful this is right now (and I actually mean that I'm not sure, not that I think it's useless and am trying to avoid saying so), though I can see how if, at some future date it's possible to make TCP connections without going through a proxy, apps written this way will suddenly make a lot of sense and be easy to port from apps already based on an API like this.

Anyhow, I thought this would be simple to implement with Athena, but I wanted to see how simple. Five minutes later I had this untested code:

# athenatcp.py
from twisted.internet.protocol import ClientCreator, Protocol
from twisted.internet import reactor

from nevow.athena import LiveElement, expose

class TCPElement(LiveElement):
    jsClass = 'Network.TCP'

    @expose
    def connect(self, host, port):
        def connected(proto):
            self.proto = proto
            self.proto.dataReceived = self.dataReceived
            self.proto.connectionLost = self.connectionLost
        cc = ClientCreator(reactor, Protocol)
        d = cc.connectTCP(host, port)
        d.addCallback(connected)
        return d

    @expose
    def write(self, bytes):
        self.proto.transport.write(bytes)

    def dataReceived(self, data):
        self.callRemote('dataReceived', data)

    def connectionLost(self, reason):
        self.callRemote('connectionLost', reason.getErrorMessage())

// athenatcp.js

// import Nevow.Athena

Network.TCP = Nevow.Athena.Widget.subclass('Network.TCP');
Network.TCP.methods(
    function connect(self, host, port) {
        return self.callRemote('connect', host, port);
    },

    function dataReceived(self, bytes) {
        // override this
    },

    function connectionLost(self, reason) {
        // override this
    });

It'd probably be better if connect were actually a factory function returning TCP instances, but the general idea would be the same. Anyone think this is a really great idea? Anyone want to turn this spike into a real library for Athena?

re: inventory meme

Sat, 28 Jun 2008 22:42:58 GMT

Glyph tagged me in his Inventory meme several weeks back. At first, I didn't respond because I was too busy. Then, because I forgot. Then, because I decided it would be cheating to wait until I had the maximum number of items possible and then post. Just now I happen to have been reminded of the meme, and I think I can spare the three minutes total this post is going to take, and I'm totally unprepared. So, here it is.

Your hands are empty.  You are completely naked.

February through April Reading List

Sat, 26 Apr 2008 17:32:26 GMT

Something Wicked This Way Comes. Ray Bradbury.
The Eyre Affair. Jasper Fforde.
The Star Rover. Jack London.
Travels With Herodotus. Ryszard Kapuściński.
Omnivore's Dilemma. Michael Pollan.
Lankhmar. Fritz Leiber.
Suite Française. Irène Némirovsky.
Persepolis: The Story of a Childhood. Marjane Satrapi.
Persepolis 2: The Story of a Return. Marjane Satrapi.
Maus 1: My Father Bleeds History. Art Spiegelman.
Maus 2: And Here My Troubles Began. Art Spiegelman.

pyOpenSSL 0.7 final released

Fri, 11 Apr 2008 16:32:09 GMT

Alright, it's done. Here are the highlights since 0.6:

Bug fixes:

Memory leak in X509.get_pubkey eliminated
Memory leaks in X509Name.__getattr__ and X509Name.__setattr__ eliminated
RuntimeWarning from X509Name comparison eliminated
X509Name reference counting issues resulting in memory corruption eliminated
Uninitialized PKeys are now rejected by X509Req signature APIs
Memory leaks in X509Name.__getattr__ and X509Name.__setattr__ eliminated

New features:

SSL_get_shutdown and SSL_set_shutdown exposed as Connection.get_shutdown and Connection.set_shutdown
SSL_SENT_SHUTDOWN and SSL_RECEIVED_SHUTDOWN exposed as SSL.SENT_SHUTDOWN and SSL.RECEIVED_SHUTDOWN
X509_verify_cert_error_string exposed as OpenSSL.crypto.X509_verify_cert_error_string
X509.get_serial_number and X509.set_serial_number now accept long integers
Expose notBefore and notAfter on X509 certificates for inspection and mutation
Expose low-level X509Name state with X509Name.get_components
Expose hashing and DER access on X509Names

pyOpenSSL 0.7a1

Sat, 22 Mar 2008 18:25:31 GMT

Hot off the presses, pyOpenSSL 0.7a1 released. Give it a spin and let me know how it goes.

Ever relevant

Wed, 06 Feb 2008 19:19:05 GMT

It is good to re-read Alan Perlis' epigrams in programming from time to time.

Some which stand out to me today:

32. Programmers are not to be measured by their ingenuity and their logic but by the completeness of their case analysis.
41. Some programming languages manage to absorb change, but withstand progress.
64. Often it is the means that justify the ends: Goals advance technique and technique survives even when goal structures crumble.
94. Interfaces keep things tidy, but don't accelerate growth: Functions do.

One which I don't think has ever stood out to me before:

20. Wherever there is modularity there is the potential for misunderstanding: Hiding information implies a need to check communication.

And of course, there are the old favorites:

11. If you have a procedure with ten parameters, you probably missed some.
95. Don't have good ideas if you aren't willing to be responsible for them.
101. Dealing with failure is easy: Work hard to improve. Success is also easy to handle: You've solved the wrong problem. Work hard to improve.
117. It goes against the grain of modern education to teach children to program. What fun is there in making plans, acquiring discipline in organizing thoughts, devoting attention to detail and learning to be self-critical?

notes on netgear support website

Fri, 04 Jan 2008 13:53:14 GMT

comments are mangled incorrectly, doubling single quotes, presumably in a confused attempt to prevent SQL injection attacks.
page caching behavior is wrong, so after a ticket is updated by netgear support staff, the update will not appear unless the browser is made to disregard its cache.
timestamps on comments are wrong, perhaps by roughly (UTC - PST) hours.
netgear support staff can't actually fix any problem you have.

One in twenty eight and falling...

Fri, 04 Jan 2008 02:25:10 GMT

Asteroid on near-miss course with Mars

No one taught these guys to label their plots, unfortunately (nice of them to put "mars" and "sun" on there though). To make it clearer, the cyan line is the probable trajectory of 2007 WD5. White dots indicate a possible position at the time it would intercept mars' orbital, if it does.

Will there be a collision? Probably not. One in twenty eight odds, though? Those are most certainly not astronomical odds.

Filesystem structure of a Python project

Fri, 21 Dec 2007 22:41:11 GMT

Do:

name the directory something related to your project. For example, if your project is named "Twisted", name the top-level directory for its source files Twisted. When you do releases, you should include a version number suffix: Twisted-2.5.
create a directory Twisted/bin and put your executables there, if you have any. Don't give them a .py extension, even if they are Python source files. Don't put any code in them except an import of and call to a main function defined somewhere else in your projects.
If your project is expressable as a single Python source file, then put it into the directory and name it something related to your project. For example, Twisted/twisted.py. If you need multiple source files, create a package instead (Twisted/twisted/, with an empty Twisted/twisted/__init__.py) and place your source files in it. For example, Twisted/twisted/internet.py.
put your unit tests in a sub-package of your package (note - this means that the single Python source file option above was a trick - you always need at least one other file for your unit tests). For example, Twisted/twisted/test/. Of course, make it a package with Twisted/twisted/test/__init__.py. Place tests in files like Twisted/twisted/test/test_internet.py.
add Twisted/README and Twisted/setup.py to explain and install your software, respectively, if you're feeling nice.

Don't:

put your source in a directory called src or lib. This makes it hard to run without installing.
put your tests outside of your Python package. This makes it hard to run the tests against an installed version.
create a package that only has a __init__.py and then put all your code into __init__.py. Just make a module instead of a package, it's simpler.
try to come up with magical hacks to make Python able to import your module or package without having the user add the directory containing it to their import path (either via PYTHONPATH or some other mechanism). You will not correctly handle all cases and users will get angry at you when your software doesn't work in their environment.

Incompatabilities between classic and new-style Python classes

Mon, 03 Dec 2007 20:08:07 GMT

radix asked me if I had a blog post about why changing a class from classic to new-style is a bad idea. After I told him that I didn't he insisted that I should write one. Since doing so will take less work than finding something else to distract his attention, here it is:

attribute lookup

Attributes on instances of classic classes override attributes of the same name on their class. For example:

>>> class SimpleDescriptor(object):
...     def __get__(self, instance, type):
...             print 'getting'
...             return instance.__dict__['simple']
...     def __set__(self, instance, value):
...             print 'setting'
...             instance.__dict__['simple'] = value
...
>>> class classic:
...     simple = SimpleDescriptor()
...
>>> x = classic()
>>> x.simple = 10
>>> x.simple
10
>>> class newstyle(object):
...     simple = SimpleDescriptor()
...
>>> x = newstyle()
>>> x.simple
getting
Traceback (most recent call last):
  File "", line 1, in ?
  File "", line 4, in __get__
KeyError: 'simple'
>>> x.simple = 10
setting
>>> x.simple
getting
10

As you can see, the descriptor on the new-style class can both handle the setattr and continue to operate after shadowing itself with an instance variable.

Special methods

A particularly interesting consequence of the previous point is that the behavior of special methods changes in some cases:

>>> class x:
...     def __init__(self):
...             self.__eq__ = lambda other: True
...
>>> x() == 10
True
>>> class x(object):
...     def __init__(self):
...             self.__eq__ = lambda other: True
...
>>> x() == 10
False
>>>

MRO

Rules for determining the method resolution forbid new-style classes in places where classic classes are acceptable. Consider:

>>> class x: pass
...
>>> class y(object, x): pass
...
>>> class x(object): pass
...
>>> class y(object, x): pass
...
Traceback (most recent call last):
  File "", line 1, in ?
TypeError: Error when calling the metaclass bases
    Cannot create a consistent method resolution
order (MRO) for bases object, x
>>>

Hopefully that's enough to satisfy radix. Know of other incompatibilities? Please comment.

Sat, 10 Nov 2007 18:19:31 GMT

3.7V li-poly:

$7, 1300mAh, 25 grams, 2.5A max
$6, 850mAh, 20 grams, 1.75A max
$8, 430mAh, 12 grams, 6.5A max
$10, 220mAh, 5.4 grams, .44A max
$5, 160mAh, 5 grams, 2A max
$7, 130mAh, 3.4 grams, 4.6A max

October through January Reading List

Sat, 20 Oct 2007 00:15:09 GMT

Smoke and Mirrors. Neil Gaiman.
The Long Tomorrow. Leigh Brackett.
Norstrillia. Cordwainer Smith.
Nova. Samuel R. Delany.
Iron Sunrise. Charles Stross.
The Lies of Locke Lamora. Scott Lynch.
The Road. Cormac McCarthy.
Paradise Lost. John Milton.
Lord of Light. Roger Zelazny.

Hair today...

Tue, 02 Oct 2007 23:57:28 GMT

Some pictures of Quechee Gorge up. Including a 360° panorama I assembled. Which took forever with stupid gimp. And it didn't come out very well anyway. Stupid parallax, I'll get you next time.

Off to San Antonio tomorrow, back next week. If there's anything worth taking pictures of (other than my brother) maybe those'll show up in a week or so.

September Reading List

Mon, 01 Oct 2007 00:21:28 GMT

By way of The Book of Jhereg:

Jhereg Steven Brust.
Yendi Steven Brust.
Teckla Steven Brust.

Also some Milton, for class: On the Morning of Christ's Nativity, At a Solemn Music, L'Allegro, Il Penseroso, Lycidas, On the New Forcers of Conscience, To Mr. Cyriack Skinner Upon His Blindness, and sonnets 11, 12, 15, 16, 19.

Wed, 12 Sep 2007 02:39:12 GMT

chevre, pine nut, raisin, onion, garlic, bacon, duck pizza