如何加速 python 数百万对象的实例初始化？

Question

我定义了一个名为Edge的pythonclass如下：

class Edge:
    def __init__(self):
        self.node1 = 0
        self.node2 = 0
        self.weight = 0

现在我必须使用以下方法创建大约 10^6 到 10^7 个 Edge 实例：

edges= []
for (i,j,w) in ijw:
    edge = Edge()
    edge.node1 = i
    edge.node2 = j
    edge.weight = w
    edges.append(edge)

我在桌面上花了大约 2 秒钟。有没有更快的方法呢？

Answer 1

您不能使它 快很多 ，但我当然会使用 __slots__ 来节省内存分配。还可以在创建实例时传入属性值：

class Edge:
    __slots__ = ('node1', 'node2', 'weight')
    def __init__(self, node1=0, node2=0, weight=0):
        self.node1 = node1
        self.node2 = node2
        self.weight = weight

有了更新后的 __init__，您可以使用列表理解：

edges = [Edge(*args) for args in ijw]

这些一起可以节省大量创建对象的时间，大约需要一半的时间。

比较创建100万个对象；设置：

>>> from random import randrange
>>> ijw = [(randrange(100), randrange(100), randrange(1000)) for _ in range(10 ** 6)]
>>> class OrigEdge:
...     def __init__(self):
...         self.node1 = 0
...         self.node2 = 0
...         self.weight = 0
...
>>> origloop = '''\
... edges= []
... for (i,j,w) in ijw:
...     edge = Edge()
...     edge.node1 = i
...     edge.node2 = j
...     edge.weight = w
...     edges.append(edge)
... '''
>>> class SlotsEdge:
...     __slots__ = ('node1', 'node2', 'weight')
...     def __init__(self, node1=0, node2=0, weight=0):
...         self.node1 = node1
...         self.node2 = node2
...         self.weight = weight
...
>>> listcomploop = '''[Edge(*args) for args in ijw]'''

和时间：

>>> from timeit import Timer
>>> count, total = Timer(origloop, 'from __main__ import OrigEdge as Edge, ijw').autorange()
>>> (total / count) * 1000 # milliseconds
722.1121070033405
>>> count, total = Timer(listcomploop, 'from __main__ import SlotsEdge as Edge, ijw').autorange()
>>> (total / count) * 1000 # milliseconds
386.6706900007557

快了将近 2 倍。

将随机输入列表增加到10^7项，时序差异保持：

>>> ijw = [(randrange(100), randrange(100), randrange(1000)) for _ in range(10 ** 7)]
>>> count, total = Timer(origloop, 'from __main__ import OrigEdge as Edge, ijw').autorange()
>>> (total / count)
7.183759553998243
>>> count, total = Timer(listcomploop, 'from __main__ import SlotsEdge as Edge, ijw').autorange()
>>> (total / count)
3.8709938440006226

Answer 2

另一种选择是跳过 Edge class 并通过 table 或邻接矩阵实现边缘。

例如

A = create_adjacency_graph(ijw)  # Implement to return a IxJ (sparse?) matrix of weights
edge_a_weight = A[3, 56]
edge_b_weight = A[670, 1023]
# etc...

尽管这确实降低了一些灵活性，但创建和使用起来应该都非常快。

Answer 3

还有另一种快速且节省内存的方法使用recordclass library:

from recordclass import dataobject

from random import randrange
import sys
ijw = [(randrange(100), randrange(100), randrange(1000)) for _ in range(10 ** 7)]

class EdgeDO(dataobject):
    __fields__ = 'node1', 'node2', 'weight'

class EdgeSlots:
    __slots__ = 'node1', 'node2', 'weight'

    def __init__(self, node1, node2, weight):
         self.node1 = node1
         self.node2 = node2
         self.weight = weight
            
def list_size(lst):
    return sum(sys.getsizeof(o) for o in lst)

%time list_do = [EdgeDO(n1, n2, w) for n1, n2, w in ijw]
%time list_slots = [EdgeSlots(n1, n2, w) for n1, n2, w in ijw]

print('size (dataobject):', list_size(list_do))
print('size (__slots__): ', list_size(list_slots))

有输出：

CPU times: user 2.23 s, sys: 20 ms, total: 2.25 s
Wall time: 2.25 s
CPU times: user 6.79 s, sys: 84.1 ms, total: 6.87 s
Wall time: 6.87 s
size (dataobject): 400000000
size (__slots__):  640000000

P.S。从 0.15 开始，有选项 fast_new 可以更快地创建实例。低于 python 3.9、64 位的新性能计数器。

从随机导入随机数导入系统 ijw = [(randrange(100), randrange(100), randrange(1000)) for _ in range(10 ** 7)]

class EdgeDO(dataobject, fast_new=True):
    __fields__ = 'node1', 'node2', 'weight'

class EdgeSlots:
    __slots__ = 'node1', 'node2', 'weight'

    def __init__(self, node1, node2, weight):
         self.node1 = node1
         self.node2 = node2
         self.weight = weight
        
def list_size(lst):
    return sum(sys.getsizeof(o) for o in lst)

print('dataobject timinig:')
%time list_do = [EdgeDO(*args) for args in ijw]
print('__slots__ timinig:')
%time list_slots = [EdgeSlots(*args) for args in ijw]

print('size (dataobject):', list_size(list_do))
print('size (__slots__): ', list_size(list_slots))
print(list_size(list_do)/list_size(list_slots)*100, "%")

结果：

dataobject timinig:
CPU times: user 804 ms, sys: 16 ms, total: 820 ms
Wall time: 819 ms
__slots__ timinig:
CPU times: user 5.54 s, sys: 23.9 ms, total: 5.56 s
Wall time: 5.56 s
size (dataobject): 400000000
size (__slots__):  560000000
71.42857142857143 %

如何加速 python 数百万对象的实例初始化？

How to speed up python instance initialization for millions of objects?

python

performance

instance