Python footgun: __getattr__

Python classes can implement a special method __getattr__ that is called during attribute access if an attribute cannot be found “normally”. This can be useful to provide fallback values if an attribute is not defined:

class DefaultAttr:
    def __getattr__(self, key):
        try:
            return self.DEFAULT_VALS[key]
        except KeyError:
            raise AttributeError(f"{self.__class__.__name__!r} object "
                f"has no attribute {key!r}") from None

class Sample(DefaultAttr):
    DEFAULT_VALS = {'x': 42}

s = Sample()
print(s.x)  # 42
s.x = 25
print(s.x)  # 25
print(s.y)  # AttributeError: 'Sample' object has no attribute 'y'

Sample objects behave like normal objects, except that if a missing attribute is accessed that is present in the DEFAULT_VALS class dictionary, the default value is returned instead. If the attribute is present on the object, __getattr__is not called and the code proceeds normally.

Note that if DEFAULT_VALS does not provide a default values, __getattr__ raises an AttributeError which mimics the normal AttributeError raised by Python if the `getattr is not defined.

So all of this is fine, right? Well…

class Sample2(DefaultAttr):
    DEFAULT_VALS = {'x': 42}

    @staticmethod
    def cur_time():
        return datetime.daettime.now()

    @property
    def prop1(self):
        return 123

    @property
    def prop2(self):
        return self.cur_time()

print(Sample2().prop1)  # 123
print(Sample2().prop2)  # AttributeError: 'Sample2' object 
                        # has no attribute 'prop2' ??

What is going on here? prop2 is a property, same as prop1, they are defined on the class so why do we get an AttributeError when calling it? Well, the documentation doesn’t exactly says that __getattr__ is called when the attribute is not found ¹, it says that it’s called when accessing the attribute fails with an AttributeError. And AttributeError can come from many places…

The issue is that when .prop2 is accessed on a Sample2 object, the prop2 method is called. This method calls cur_time, and cur_time raises an AttributeError because of a typo (“daettime”). Since accessing that property raised an AttributeError, Python calls __getattr__, which doesn’t find prop2 in the default values dictionary and raises its own version of AttributeError.

So this results in a confusing error message (which the version of Python I’m using “helpfully” improves by adding “Did you mean: 'prop1'?” at the end when displaying it), and the original AttributeError is lost with its traceback.

After stumbling onto this, I found this issue about this subject, but this is mostly a non-useful discussion about wether this is a bug or a feature request.

I personally think confusing error messages are bad, especially when they cause the loss of useful debug information.

I’ve found two ways to improve the situation. The first one is to not raise an AttributeError when wanting “default behavior”, but call object.__getattribute__ instead:

class DefaultAttr:
    def __getattr__(self, key):
        try:
            return self.DEFAULT_VALS[key]
        except KeyError:
            return object.__getattribute__(self, key)

With this code, the following happens:

1. The code accesses .prop2, which raises an AttributeError
2. Python turns around and calls __getattr__
3. __getattr__ wants to perform default behavior so it calls object.__getattribute__
4. This results in .prop2 being accessed again, which again raises an AttributeError
5. Since the access is done by the lower-level __getattribute__ function, the object __getattr__ is ignored
6. The exception from the property is correctly thrown to the caller, with full traceback

The obvious disadvantage of this method is that the property is called twice if it fails with an AttributeError. I don’t think it’s an issue because 1) property getters should not have side effects, 2) if your property getter has side effects, it should hopefully not have side effects when raising an exception, and 3) you should rarely catch AttributeError and let the program quickly die if it’s raised anyway.

Another method is to use gasp __getattribute__:

class DefaultAttr:
    def __getattribute__(self, key):
        try:
            return super().__getattribute__(key)
        except AttributeError as err:
            try:
                return self.DEFAULT_VALS[key]
            except KeyError:
                raise err

With this, we basically emulate what Python does when accessing a property (with the super().__getattribute__(key) call), but instead of immediately throwing away any AttributeError that occurs, we store it and re-raise it if no default value is available. This has the advantage of calling the property only once even if it fails, but it requires the use of __getattribute__ which is called for every attribute access on the object. That means that if you mess up, you will end up with it being called recursively until your program dies on a stack overflow. It probably makes every property access on the object a little bit slower, too.

My conclusion is that while Python is a great programming language, some parts of its API are sub-optimal and cause issues², and this is one of it. I think the best way to solve this in a backwards compatible matter would be to add a constant to the language, maybe called AttributeNotImplemented (to be similar with the NotImplemented return value). __getattr__ could return this value when it wants “default processing” to occur (i.e. let the original AttributeError exception be propagated).

It is overkill to add a constant just for this? Maybe, but I think it would be justified in that case.

---

1. it used to say that in Python 2, but this was apparently changed for Python 3

2. My personal list of Python footguns that I struggle with regularly: strings are iterable, bytes objects can be silently converted to strings, and f-strings require that f"" prefix.