Jp Calderone ([info]jcalderone) wrote,
@ 2008-12-09 12:44:00
Previous Entry  Add to memories!  Tell a Friend  Next Entry
Current mood: enraged

ioctl TIOCSTTY EIO
What does it mean if a TIOCSTTY ioctl fails with EIO (errno 5) on Linux?


(9 comments) - (Post a new comment)


[info]toidinamai
2008-12-09 08:57 pm UTC (link)
Surely you mean TIOCSCTTY? There does not seem any return path leading to EIO in drivers/char/tty_io.c. Maybe your file descriptor is not a terminal?

(Reply to this) (Thread)

[info]jcalderone
2008-12-09 10:19 pm UTC (link)
Oops, yes, TIOCSCTTY. Superficial inspection of the code which encounters this error suggests the file descriptor is for a terminal. The file descriptor comes back from Python's pty.openpty (a wrapper around openpty(3)). Then a fork() happens and the child tries to do the ioctl. Once in a while, the ioctl fails. Interestingly, os.isatty (a wrapper around isatty(3)) returns True for the FD being passed to the ioctl call before the ioctl call and usually returns True after the ioctl, but in the cases where the ioctl fails with EIO, it returns False afterwards. So the ioctl is breaking the file descriptor? Or the file description? But again, only sometimes (perhaps 10% or so, but it seems non-deterministic).

(Reply to this) (Parent)(Thread)

[info]toidinamai
2008-12-09 11:15 pm UTC (link)
Just a guess:

>>> import os
>>> master, slave = os.openpty()
>>> os.isatty(slave)
True
>>> os.close(master)
>>> os.isatty(slave)
False
>>>

(Reply to this) (Parent)(Thread)

[info]jcalderone
2008-12-09 11:17 pm UTC (link)
I think it's a bit more involved than that. The master does get closed in the relevant codepath, but only in the child process after fork(). So it's still open in the parent, so the slave is still valid. I actually instrumented the code in question (it's not really a minimal example yet, unfortunately) and saw that isatty returned True immediately before the ioctl call and then False immediately afterwards when the ioctl itself failed with EIO.

(Reply to this) (Parent)(Thread)

[info]toidinamai
2008-12-09 11:34 pm UTC (link)
What happens between the first call to isatty and the ioctl? I bet this is a timing issue with the parent somehow closing the master. Is this code publicly visible? Twisted.Conch maybe?

(Reply to this) (Parent)(Thread)

[info]jcalderone
2008-12-09 11:53 pm UTC (link)
I guess lots of stuff might happen in the parent. The code is indeed public - it's the PTY suppose for reactor.spawnProcess on POSIX for all reactors. The way I've been able to reproduce it so far is to run some tests repeatedly, eg:

trial -u twisted.internet.test.test_process.PTYProcessTestsBuilder_SelectReactor

It can take a lot of iterations to trigger (I just did a run where it took almost 400), but sometimes it triggers after only 5 or 10. Unfortunately the misbehavior doesn't cause these tests to fail, it just causes some garbage to be written to stderr. I'm using this patch to make the test run stop after the problem occurs:

Index: twisted/internet/process.py
===================================================================
--- twisted/internet/process.py (revision 25605)
+++ twisted/internet/process.py (working copy)
@@ -368,6 +368,7 @@
 
         collectorEnabled = gc.isenabled()
         gc.disable()
+        parentPID = os.getpid()
         try:
             self.pid = os.fork()
         except:
@@ -395,6 +396,7 @@
                     self._execChild(path, settingUID, uid, gid,
                                     executable, args, environment)
                 except:
+                    os.kill(parentPID, 9)
                     # If there are errors, bail and try to write something
                     # descriptive to stderr.
                     # XXX: The parent's stderr isn't necessarily fd 2 anymore, or


That causes the child to kill its parent if it gets into the error handling codepath for any reason. I've also tried sticking a pdb.set_trace() in there instead to be able to explore (with all the ttys and forking going on, the result is a bit hard to use, but not impossible).

The output when it finally fails with the patch applied looks something like this:

Test Pass 386
twisted.internet.test.test_process
  PTYProcessTestsBuilder_SelectReactor
    test_callWhenRunningSpawnProcessWarningFree ...                        [OK]
    test_processExitedWithSignal ...                                       [OK]
    test_spawnProcessTooEarlyWarns ... Upon execvpe /usr/bin/python ['/usr/bin/python', '-c', ''] in environment id 142718604
:Traceback (most recent call last):
  File "/home/exarkun/Projects/Twisted/trunk/twisted/internet/process.py", line 395, in _fork
    self._setupChild(**kwargs)
  File "/home/exarkun/Projects/Twisted/trunk/twisted/internet/process.py", line 859, in _setupChild
    fcntl.ioctl(slavefd, termios.TIOCSCTTY, '')
IOError: [Errno 5] Input/output error
Killed

(Reply to this) (Parent)(Thread)

[info]toidinamai
2008-12-10 12:16 am UTC (link)
Hm, under strace it seems to fail a lot quicker. I still think it's a timing issue and not a problem with the ioctl in particular. Changing the ioctl call to os.write seems to confirm that. I'm not familiar enough with the reactor stuff to see what's really going on there and it's too late for me to dig in further, sorry.

(Reply to this) (Parent)(Thread)

[info]jcalderone
2008-12-10 12:35 am UTC (link)
strace seems to make it take longer to fail on my system. :) I did finally manage to get it to fail under strace a couple times though. First time strace itself had a bit of a problem though, spitting out "trace: ptrace(PTRACE_SYSCALL, ...): No such process" right before the trial process failed.

Anyhow, thanks for the discussion so far. At least I'm a bit closer to understanding the problem than I was before.

(Reply to this) (Parent)(Thread)

[info]jcalderone
2008-12-10 04:43 pm UTC (link)
Figured out what was going on, at least. It ended up being a bug in the unit test. The test didn't wait for the process to get into any particular state, it just immediately started cleaning up. Sometimes it would close the PTY before the child got past the setup. :/ So the fix is just to make the test wait for the child to finish initialization before trying to clean it up.

So it is actually the case you gave an example of earlier, albeit it split across two processes.

(Reply to this) (Parent)

(9 comments) - (Post a new comment)

Create an Account
Forgot your login or password?
Login w/ OpenID
English • Español • Deutsch • Русский…