从 int 继承的 Cython 扩展类型导致 MemoryError

Question

我正在尝试创建一个继承自 int 或 cython.int 的扩展类型。这对我来说是必要的，因为我需要能够将这种类型用作某些 lists/arrays.

的索引

这是在 Python 2.7.9 Win32（我是运行 Windows 7）上使用 Cython v0.22 在 Anaconda 上重现错误的代码：

import cython
cimport cython


import sys


cdef class ExtendedInt(int):


    #cdef object __weakref__ # explicitly enable weakref for this extension type, will self-destruct when it is no longer strongly referenced.


    def __add__(a, b):
        return a+b


# Simple test case
def main():
    total_it = 1000
    for i in xrange(total_it):
        for j in xrange(10000000):
            ExtendedInt(j)
        sys.stdout.write("\rGenerating lists of ExtendedInt : %i/%i" % (i, total_it))

发生的情况是，如果您尝试创建大量 ExtendedInt，Python 解释器有时会因 MemoryError 而崩溃。使用上面的代码片段，在我有 4 GB 内存的机器上，它在第 11 次迭代时崩溃，但这会因机器规格而异。我尝试启用 weakref 但它没有解决问题。

但是，如果您将 "cdef class ExtendedInt(int):" 替换为 "class ExtendedInt(int)"（因此它不再是扩展类型，而是简单的 Python class），那么问题确实存在没有发生，没有 MemoryError 崩溃。

因此似乎没有正确释放从整数继承的扩展类型，即使我启用了 weakref。这是我应该在跟踪器上填写的错误还是我遗漏了什么？

这是当前Cython版本的bug吗？我在错误追踪器上找不到任何参考...或者我遗漏了可以解决问题的东西？

Answer 1

Stefan Behnel discovered that the bug is in Python 2.7 inside intobject.c:

Your extension type automatically inherits the "int_free" slot function from its base type, so the "else" case in "int_dealloc()" will in fact call "int_free()" and append the object to the free list despite not being exactly of type PyInt. Then, when creating a new ExtendedInt instance, Python's int-subtype instantiation code ignores the free list and instead creates a completely new object.

Thus, the free list keeps growing until it fills all memory and the process dies. I created a CPython ticket.

这是link to the CPython ticket。

这是 Guido Van Rossum 的回答：

The intended solution is to require that int subclasses override tp_free.

Note that much of this code is written pretty much assuming that subclasses are created using class statements (in a regular Python module, not Cython) which take care of all these details. That doesn't mean Cython is wrong to try this, but it does mean there isn't a lot of documentation, and it also means I don't think the thing you reported qualifies as a bug in CPython.

所以我不确定这是否会得到修复，或者 Cython 是否会实施解决方法。不管怎样，对于那些和我遇到同样问题的人，你可以通过继承对象来解决这个问题，然后重新实现整数的魔术方法，比如 int、index 、add 等，正如我用 a concrete example here 解释的那样。

从 int 继承的 Cython 扩展类型导致 MemoryError

Cython Extension Type inheriting from int cause a MemoryError

python

crash

inheritance

cython