[applause] Welcome, everybody The URL there is where on my side I usually keep the PDFs of my slides I will also upload them to the PyCon side What’s right there right now is not exactly what I’m going to present I promise this afternoon I will update everything As a kind of a bet, I picked the least fascinating, the least exciting and most indispensable chapter of the third edition of Python in a Nutshell, which is already available in an early release That’s Chapter 5, which is about errors and exception handling Incidentally, if it still works, because we already may have mixed something with the I’m sorry The Python 45 code should allow a discount at checkout if you get the e-book And what you get is, as usual with O’Reilly, a DRM-free e-book in all forms and permission to copy them on all of your devices And the reason I have this slide is not so much because I like to sell more copies, because at these discounts, it doesn’t really matter It’s because I want to get feedback while we can still do major changes, “we” being me and my two coauthors, my wife Anna and Steve Holden, the initiator of PyCon many, many years ago All right. Exceptions in Python are not always errors This is crucial for people coming from other languages where avoiding exceptions is important because they always mean errors They can be errors, like this “1 divided by 0” exception They can be perfectly normal things such as an iteration is finished Remember, I will use “exc” all the time and pronounce it “exception” for shortness to fit on the slides The core thing that every Python programmer needs to know to deal with exception is a ‘try’ statement It’s, fortunately in 2.7 and 3.anything, become completely regular You can have an ‘except’, more than one ‘except’ You can have an ‘else’, only if you have ‘except’ You can have ‘finally’ optionally at the end You can also have just ‘try/finally,’ in which case you’re not handling an exception, just doing cleanup work at the end A few things that are absolutely important When you have ‘except’ clauses, put the narrowest one first, because the matching is strictly beginning to end So if you match a more general exception, the further ‘except’ clauses that might match more closely will not even be looked at, so be careful with that ‘Else’ is a somewhat unpopular clause, but I find it precious It means: execute, without any exception guard, what follows if no exceptions have been called so far You can often get the same effect by moving whatever is after the ‘else’ after the whole ‘try/except’ statement, but it’s much more explicit, much more readable to have things in the ‘else’ clause Another not-well-known thing is that in the ‘finally’ clause, you cannot have a ‘continue’ If you have a ‘continue’, Python will scold you and give you a syntax error, but if you put there a breaker return, Python will not tell you anything, but your exceptions will be hidden, swallowed, disappear And that’s so unexpected that it is sufficient to make your code completely unreadable Also, if you think about the semantics of ‘try/finally’, there’s never any reason to say ‘finally: return’ or ‘finally: break’ So just use a linter or your own discipline to never use ‘break’ or ‘return’ Sometimes, you want to be the one initiating the exception by raising the exception object In Python version 2, which I always call v2, while Python version 3 I always call v3 for shortness — the whole book is about both Python 2.7 and Python 3.5, so the differences and distinctions and so on are mentioned over and over —

it could be a subclass of exception, in which case it would be instantiated for you That was never a great idea, and it doesn’t work in v3 anyway, so just avoid it, keep your code clean, always raise an object Or raise without anything That must be an except clause, directly or indirectly Could be in a function you’re calling from an except clause It just re-raises the exception you just got This is a very common thing You want to do some cleanup, some arrangement and then just let the exception propagate When do you want to raise? Well, if you’re playing old-fashioned poker and you have that hand, you probably want to raise, but that’s a different meaning of the word “raise.” When you want to give some diagnostics where Python wouldn’t For example, suppose you’re writing a cross product function, which, for reasons that have to do with the semantics of the library you’re writing, cannot accept empty sequences If either sequence is empty, that’s a ValueError Type is correct, but value is not Otherwise, it does the normal way Of course, you could use itertools to do it, or whatever But the point is, I am not checking all the other things that could be wrong with seq1 and seq2 For example, if they’re not iteratable, Python itself will check for you when you try to iterate on them in this comprehension and raise a type error Never duplicate error checks that Python intrinsically does for you anyway That just clutters your code to no good extent If you want to raise a different error in those cases, use a ‘try/except’ around the relevant code for that purpose In v3 only — it’s one of the big improvements that I see in the language — there’s an automatic wrapping of exceptions Instead of having the traceback and the exception as two separate things that just happen to be there at the same time, the traceback is held by the exception object You can change the — you can get a copy of an exception with a different traceback object with a with_traceback method That’s not all that frequent a need, but if you ever have it, that’s how you do it But the most important thing is that automatically, the last exception that was called and is being handled becomes the __context__ attribute of the new one So that happens automatically for you Explicitly, you can also set some exception object as the cause of the new one with the new form of the statement ‘raise’ Raise the new exception object from — None means no cause, or from existing exception instance, that sets the __cause__ What’s the effect? It’s diagnostics So, this is a function that wants to compute the inverse, and we decide that the inverse of zero is not really a ZeroDivisionError, it’s a ValueError; because we could be computing the inverse in other ways than by dividing whatever, it needs to be a ValueError However, to avoid hiding what really happened, the cause of that ValueError, we use the ‘raise from’ form And in this case, if it just propagates up to the top level, the diagnostics says that the ZeroDivisionError was the direct cause of the exception that actually happened Note the exception that actually propagates is ValueError So if you want to catch it, it’s ‘try/except ValueError’ But if the diagnostics happen, the __cause__ being set by the ‘raise from’ statement, you get a much more precise — much more precise diagnostics Similarly, for context, context is typically not something you set explicitly, it just happens Suppose that in the handler, in an except clause, you just happen to have a bug So, this is pretty explicit, right? The ‘try 1/0’ obviously raises a ZeroDivisionError You catch it in the except where accidentally you’re trying to sum a number to a string, which is an error of its own So in this case, instead of just seeing the TypeError, as you would have in v2,

in v3, if this propagates to the top level, what you see is a ZeroDivisionError, and “During handling of the above exception, “the following exception occurred.” So again, what’s propagating is the inner one, but it carries along the context in which it was raised in __context__, which is sometimes very useful in pointing out exactly what’s happening where your bugs are Another alternative to ‘try/finally’, but sometimes also to ‘try/except’, is a ‘with’ statement where you need a context manager X — means it has a __enter__ method in which it may return something that you catch with the ‘as something’ — ‘as y’ in this case — and then imagine that the block that’s literally inside the ‘with’ is in fact executed inside the ‘try’ clause of a ‘try/except/finally’ The point being that if there is any exception, that exception — here I’m using the exc_info function of module sys to get the triple for the exception — you span through the __exit__ method of the context manager, which may decide whether it propagates further or not Not the way I’ve spelled it: ‘if not x.__exit__’ and so on, ‘raise’ If x.__exit__ chooses to return true, that means the exception has been handled, it must not propagate further That’s why it’s kind of like a ‘try/except’ block And finally, if no exception has occurred, the __exit__ is called with Nones, meaning, “OK, we’re exiting.” Why is the distinction important? Because this is essentially the same as the “resource acquisition is initialization” idiom of C++, but in C++, you do get to run the destructor, but it’s not immediately obvious whether it’s a normal exit or an exception With a Python ‘with’, you do know immediately So for example, in a context manager like SQLite3’s connection, the ‘exit in case of exception’ has a rollback of whatever you’ve been doing in the block, so you don’t have half of the operations destroy your database In case of success, it has a commit so that the database reflects the operations you’ve done The best way to make a context manager, the simplest in most cases, instead of writing a class with __init__, __enter__ and __exit__ special methods, is the contextlib.contextmanager decorator of the contextlib standard library module Essentially, you use typically a ‘try/except/finally’ for — to map exactly what the context manager will do, except you yield the result that you’d normally — you can use in the ‘as something’ part of the ‘with’ statement Moving on, we finally have, in 2.7 and better, a sensible way to handle exceptions in generators, and therefore, to have the caller, which in a sense in many cases in modern versions of Python is more a controller of the generator object that’s executing, causes throw an exception inside the generator That’s generator.throw Possibly, the most common case is the close method, which is just exactly equivalent to throwing an instance of GeneratorExit Note again, this is another case, though a more modern, newer one, where an exception doesn’t mean an error It’s pretty normal for the controller to decide to terminate the generator objects it’s using Most generators should just do the — when they’re getting the — yielding a previous result and getting the new stuff, so ‘result = yield previous’ should do that in the ‘try’ clause of a ‘try/except’, so at GeneratorExit they can do cleanup Of course, they can also refuse to exit,

but that’s not normally the intent It’s just to afford an opportunity to do cleanup exactly where it’s clearest and simplest I’m pretty sure you’re all familiar with the exception propagation concept around the stack of calls Just one thing I want to point out is that all the exception handling, all the ‘finally’ clause and possibly at the very end sys.excepthook, whatever function you may have set or the default one, all take place before any termination function you register with the atexit module It’s better not to rely on this, not to have things depend too heavily on the order in which they’re destroyed, but if you have to, this is the order Python gives you The atexit register functions are last Now, exceptions are one of the few places where Python really, truly desperately cares about inheritance hierarchies Why? Because the way the except clauses work is to catch any instance of the exception types you list in it, which means also any instance of their subtypes Therefore, the hierarchy of exceptions, standard exceptions, is very important In version 2, we tried to make sure that all errors came from StandardError, with many types and subtypes, and exceptions that aren’t really errors were not — were still inheriting from BaseException indirectly — directly or indirectly, but not from StandardError That didn’t really work out too well On the other hand, the EnvironmentError blocking together all the IO and OS errors, to be perfectly honest, whether some anomalous situation during IO is done for us by the operating system, so whether something is an IO error or operating system error isn’t usually immediately clear, like, what? However, in version 3, things have been simplified We don’t have the StandardError level anymore So, essentially, both non-errors like StopIteration and actual errors inherit from Exception Only the two exit kinds, Generator and SystemExit, and the KeyboardInterrupt inherit from BaseException and not Exception Perhaps more interestingly, the EnvironmentError and IOError are now just synonyms for the OSError, because in the end, that’s the real semantics This is an error the operating system has reported to me during some system call On the other hand, there’s a bunch of new classes, which subclass directly OSError, which are pretty obvious and specific One example is FileNotFoundError That’s an error we’ve all met, and we all know what it means “Whoops, there should be a file here, and there isn’t.” So for example, in version 2, the first version of this, almost inevitably, when you caught an IOError, you had to start looking at its errno to be like, “OK, what happened? “Is it a missing file? A permission problem?” Also, the fact that missing file is encoded as ENOENT is not immediately obvious to anybody but C programmers with a Unix background, I suspect So this was a typical tree We’re just trying to read and return the contents of the file at ‘path’, otherwise returning the default In version 3, I believe things become much more readable, essentially, because all we do is catch the very specific FileNotFoundError and return the default Anything else, any other IO error or OS error or environment error — call them as you wish — will just propagate like we had to do explicitly with that raise over there So, with all these standard exceptions, do you ever need to define your own? The answer is, if you’re writing a library or framework, you probably do

So, why? Because the client — code that’s client of your module or package can easily catch all the exceptions that are specific and peculiar to your package and handle them some way Best practice is to define a class, commonly called just ‘Error’, within your module, so your module.error or your package.error, which directly sublcasses Exception and only provides a docstring and then if you need to get more specific, you further subclass that For this latter purpose, I have this idiosyncratic taste I believe this is the one place in Python where you really want to use multiple inheritance It’s — I know it’s a minority position, but I’m quite convinced. Why? Well, because it correlates to essentially the only part of Python where you really have to use inheritance because the way the except clauses work totally relies on inheritance You can’t do it otherwise If you do have to use inheritance, you can’t adopt the normal principle, prefer composition over inheritance, then the multiple inheritance may come in handy Specifically, suppose there’s a kind of type error that is — for example, any kind of standard exception will do — that is very specific to your module or package What I like to do then is define — I would normally call it ‘MyTypeError’ or something, but I don’t have space on the slide, so I called it ‘MyTE’ I inherit from the class error I defined before, which means this is an error coming from my own package, and then from the standard exception that I actually closely resemble, in this case a type error This means that a client that would need at this point to catch all type errors It doesn’t care where they’re coming from exactly; it can catch type error. That will work Multiple inheritance and ‘except’ clauses work just fine And a different client which wants to catch all exceptions typical only of your module can do that just as easily, too, so you get the best of both worlds And this concludes the first half — the tactical part of “OK, this is how you handle exceptions in Python.” Now we move to the much more interesting part, and not all that different between versions 2 and 3, which are error handling strategies So, I was responsible many, many years ago for introducing this terminology, two acronyms, to define the two big classes, big categories of error handling strategies: LBYL, EAFP You can spell it out: “Look Before You Leap” versus “it’s Easier to Ask Forgiveness than Permission.” If you’re discussing this a lot, the phrases are a bit long; you’ll want to use the acronyms Why was that important and possibly still is? Because people who come from many other languages where exceptions are really a big problem to have, strive, go overboard to avoid any exception by using LBYL strategies The LBYL means: first, check all the preconditions are met If they’re not, raise the exception or give some other indication of error Only if all the preconditions are met, perform the operation expecting no exception The “Easier to Ask Forgiveness than Permission,” which by the way is a phrase I borrowed from one of the most amazing women ever to bless computer science, Grace Murray Hopper, is simply: just try performing the operation Any prerequisite violations Python will catch for you The exception will raise You may want to catch to raise something different or do cleanups or whatever, but you’re done No huge bunches of code checking prereqs Why? Well, because LBYL has problems First of all, it breaks one of the principles

I enumerated earlier: Never check something that Python will check for you anyway Makes no sense LBYL duplicates work that Python performs anyway to check the preconditions Next, and even worse, it obscures code clarity You can have five or six lines of code which do the actual work buried after fifty lines that just check if everything is hunky dory That’s not the way to help readers and maintainers of your code If you have to check a lot of preconditions, you’re quite likely to forget some, in which case the exceptions you didn’t expect will happen anyway, so you have to be ready to handle anyway, so why all the wasted pre-checking? One of the worst cases: things change all the time Panta Rei Things change Suppose, instead of just trying to read a file and catching the FileNotFoundError, you did, “Well, I will first check if it exists, “and only if it exists will read it.” Well, news for you: it existed when you checked, and then some other process running on your machine may have deleted it You can’t control that It’s another process outside of your control So this is exactly the example shown in both ways I wish we had time to go over the LBYL version and spot all the many possible problems, but essentially, I’ve boiled it down to: If some file exists at the path, well, open it and return its content Otherwise return the default value While the other version is the one I already showed, that example of the new exceptions like FileNotFoundError The second one is rock solid If anything is a problem, it will either raise an exception — for example, suppose — the first one would raise the exception too — suppose path is a dictionary; then you can’t open it as a file That’s an exception, but according to the function specs, that exception needs to be raised, and so it does So, once having been converted to the Grace Murray Hopper school of programming, easier to ask forgiveness than permission, how to do it right? Narrow down as much as possible the area in which you’re guarding against exception That’s absolutely crucial The first version is a very natural way of doing things So the trycall specifications are: try calling a method called attr on the object called obj with arguments a and k (*a, **k) If that attribute is not present, then return a default value instead And so, a very natural way of doing it is just do the getattr If that’s an attribute error, then return the default, otherwise go on with the call What’s the problem here? Well, this catches not only an attribute error raised by the getattr you can see on screen; that is, there is no attribute or method attr in obj, but if there’s a bug in that method, which causes a completely unrelated attribute error, this also returns the default instead of propogating the attribute error as it should So this is a second — the second version is EAFP done right In the ‘try’, only do the getattr and use a little local variable to get the result as the method It’s also more readable That’s a side benefit Catch the attribute error to return the default only for the ‘try’ clause, and this is a good example of using the ‘else’ clause to be very explicit about what is not guarded by this ‘try’ statement: ‘except’ clauses So this is a trivial example because I needed to fit it on a single slide, but it applies to more complicated cases even more so

Don’t guard too many operations in a ‘try’ clause In general, large programs — a program that’s twice as big gives you more than twice as many heartaches as the one that’s half as big One of them is in error handling You have to think, even in the architecture of your program, about all possible causes of errors There could be errors in your code, unfortunately, errors in the libraries you use — no library is 100% debugged Those are the place for unit tests If you do find bugs in the libraries you use, send the unit test as a patch, or pull request as they say these days, and ideally the fix too, but this is what unit tests are for If any bug survives your unit test, then you don’t have enough unit tests Just add the unit test to your suite as soon as you notice the bug It should be caught by your unit test Subtler are mismatches between the prerequisites of your libraries and your understanding These you can’t really fix with unit tests You may mock the library, but your mock will behave as you think the library should behave, not the way the library actually behaves, so that’s why we have integration tests, where you’re actually running your code calling the library, or if it’s a framework, calling back to you Third, and actually pretty frequent, invalid inputs to your code This is where ‘try/except’ is really your friend, not so much for errors caused by bugs or misunderstandings of how a library works But it’s when you take external input, which is outside of your control You’re never going to be able to unit test or integration test a human being that’s interacting with your program And unfortunately, as we all know, human beings are still in release 1.0 beta and they’ve been for a while, and I do not expect any improvement anytime soon So this is where you want to ‘try/except’ Python will check things for you; you just don’t want to crash when it does Sometimes you’ll add a few more tests for very specific stuff, very specific to your program and arrays The same goes — when I say “input,” that includes human interaction; it also includes a configuration file or some environment variable, stuff that some human being directly or indirectly would have set — inputs in this sense Then, consider: “Whoops, I caught a bug, now what?” Well, I can just crash and show a beautiful traceback Well, there’s one thing — the area of programming is very vast and varied, but one thing I am absolutely sure of: any program that shows a user a traceback is broken I don’t care what It is a broken program Why? Because the info that the coder needs to understand and fix a problem is never the same the user needs to at least understand, ideally remedy, like, make the input correct Never show the user a traceback It may be important information for you; archive it somewhere, ask the user permission to please send it to you — there’s many ways to deal with it — but what you want to show to the user is an error message that’s a core part of your user interface Realize it or not, users who are just meeting an error are in an emotionally fragile state They particularly need — they particularly need friendly messages, not 500 lines of Java traceback, or even just 50 of Python traceback I mean, it’s a bit better, but not by much Focus on: What can they do? What should they do? Like, “Please enter a number above 17,” or — if there’s a simple problem in one of their inputs, or “We’re very sorry, we’ll be back to you” “as soon as possible with a fixed version,” whatever Once you’ve caught, diagnosed, and blah, blah, blah your program, if at all feasible in your architecture, restart it You may think, “Oh, well, I can fix all of the ways “the state had been invalidly altered.”

And go on Maybe you can More likely than not, you’ll miss something Just if you can possibly restart, of course that works better if at various periodic intervals, when you’re sure things have been doing well so far, you snapshot the state of your program to files or databases or somewhere so you can restore it from there Ideally, since your program might end up in a completely unusable state, having a separate watchdog program that, like, looks for a heartbeat or health check or something and kills your program and restarts it as needed, maybe, but here we’re getting into advanced architecture Just simply restarting your program, ‘exec’ in any Unix-like system, for example, would do it Logging is your friend, your bestest friend ever You may be scared of it because it’s so rich and complex That’s because it’s used in a huge variety of situations It’s worth your effort to understand at least that subset of logging that you will be using all the time That’s how you debug; that’s how you optimize Even if there’s no bug, but things are a bit slow, how do you follow what’s happening on the server? Make sure your logs are machine parsible You don’t want to spend hours staring manually at mostly repetitive logs and hoping you can spot the point where things aren’t as they should be Then design your log parsing scripts just as carefully as you design any other part of your code Once you’ve done that, when in doubt, you think, “Should I log something here?” The answer is yes I don’t need to know anything else You think that maybe at some point that information could be useful for debugging or whatever? Log it That makes very large logs, right? Which is why first you make sure you can have the computer deal with them. Don’t log just errors I’ve seen people with otherwise excellent programs do that They only think, “Oh, wow, this is broken “I should log the state. OK.” What would the state be if things weren’t broken? “Oh, whoops, I don’t log that.” That’s what logging is for: gives you a base case comparison of how the state should be Last but not least, far from least, avoid ‘assert’ Unless you’re using some weird testing system which changes the meaning of ‘assert’ and so on — let’s not get into that It seems an attractive way to check sanity, right? For example, I have my function Well, I’m not using type checking yet, so first thing I said ‘assert’ is instance exempt, apart from the type checking polemic Don’t use ‘assert’ It’s an attractive nuisance It’s there to trap you and hide bugs from you because it will become a null operation whenever you run with -o But have you debugged your users? Or have your users — you can have debugged your software, maybe I’ve heard somewhere of a big program without any bug I suspect it’s a legend, but maybe it exists But is there a user who never makes a mistake, never gives you wrong input? I very, very much doubt so Moreover, what people assert usually checks stuff that Python is checking anyway, so it’s kind of a subcase of Look Before You Leap Just don’t do it The only reason ‘assert’ is there is while you’re developing doing a sanity check, not on inputs, on internal state, and provide so‑called “executable docs,” although I prefer docstrings and the doctest module for that OK, this is the non-Q&A part of the talk I remind you of those two URLs The first one is to get — please wait, give me a couple hours to have lunch and then update the slides And the second one is to get copies of the book And please send me feedback [applause]

(host) So, just before lunch, we have five minutes for questions If you have a question that you want to ask, come here to the mic (audience member) Hey. So, my background is with common Lisp and it has some similar issues And with the Look Before You Leap thing, the concern would be that it invokes a lot of overhead with manipulating the stack and stuff Do you want to speak to — in the Look Before You Leap case, it can be very efficient to just check a particular simple thing, as opposed to, it seems like there would be a lot of overhead to conjuring stack traces and stuff Are those done lazily? Can you speak to the performance implications? (Alex Martelli) In terms of implementation, exception handling in Python compared to, say, a function call doesn’t really add overhead to the extent that, say, it would in C++ It’s the same order of magnitude So considering that anything you’re planning to check that Python will again check for you internally is going to be a function call, I do not really see any advantage there So there’s another situation, which I wish — so, let’s say, if the — if there’s a key ‘foo’ in the dictionary, then I want its value, otherwise I want the value ‘bar’ as a default There used to be a lot of angst about, “Well, I can save a nanosecond by checking if ‘key’ is in ‘dict’.” Well, as long as it’s almost never there, yes If it’s there 99% of the time, then definitely not On the other hand, why not a ‘try/except’ key error? The right solution is of course to use the ‘get’ method of the dictionary, which is designed to do only one hash lookup and the checking internally for you That’s why so many of my examples are ‘get’-like; that is, “Do this or give me this default value.” It’s really a good pattern or idiom, if you will, but it appears in so many different situations Apart from those, there’s really no good case for Look Before You Leap, seriously (audience member) Hi, thank you for the talk About the context, the __context__ object on the exception Will it do this more than once if there’s several, and is there any control flow significance or is that output trace its only purpose? (Alex Martelli) So, the exception that propagates is the one that was raised last in execution order Its context, if any, is the exception that was being handled in the lexically closest — is that right? — no, execution-wise closest ‘except’ clause There is no infinite stack of context, but the exception that is the context can in turn have a context (audience member) Oh, yes. That was — (Alex Martelli) That’s not forbidden. Or a cause or both You could have a raise from something that sets the cause That’s very explicit It might also implicitly set the context On the other hand, I believe the — I should check — the excepthook in the top level interpreter only gives you — only shows you one So you need a custom excepthook if you are in the habit of nesting ‘try/except’ forever (audience member) Thank you (audience member) Hi, I have a question about whether checking first or be sorry So I have seen a lot of checking first when people want to provide a really good error messaging So you check for things, and the message that you can tell the user is very specific, while if you let Python just check around and throw everything if it’s not found or something, the error can be less specific, and it’s harder to add that on the exception handling So I was wishing to know, how do you deal with that? (Alex Martelli) How to do deal with what? I’m sorry (audience member) Yeah, in this comparison of either check first or be sorry and do things later, I have seen that kind of approach when people — (Alex Martelli) I’m not sure I understand what you’re asking about, but in general, the only reason to check is that if you didn’t check, Python, before repeating the check, would do some immutable —

some irretrievable mutation of state And there is essentially no cases in most sane systems One might be, you know you want to write two gigabytes to disk, you might want to first check that the disk has the space That’s because nobody is going to check until the disk is full, and then you’ve only written a part If writing a part of the data is worse than writing nothing, that might be the case. You are going to be in trouble anyway because other processes could be writing at the same time, so it only ameliorates if you’re already short of disk space If you’re running short of disk space regularly, my recommendation would be to buy bigger disks, but — [laughter] You could check and resize the disk dynamically, at least if you’re running on Google Cloud platform, you can resize the disk while your program is running, resize the virtual disk under you Depends on the operating system; if you’re running Windows, that doesn’t work, but with most Linux versions, it can react to a disk increasing dynamically (host) OK, thank you very much, Alex, and we can continue the question session in the hallway. Thank you [applause]