numpy.std 在 memmapped ndarray 上失败并出现 MemoryError

Question

我有一个巨大的 (30GB) ndarray 内存映射：

arr = numpy.memmap(afile, dtype=numpy.float32, mode="w+", shape=(n, n,))

用一些值填充后（非常好 - 最大内存使用量低于 1GB）我想计算标准偏差：

print('stdev: {0:4.4f}\n'.format(numpy.std(arr)))

此行因 MemoryError 而惨败。

我不确定为什么会失败。我将不胜感激如何以节省内存的方式计算这些的提示？

环境：venv + Python3.6.2 + NumPy 1.13.1

Answer 1

确实，numpy 对 std 和 mean 的实现制作了数组的完整副本，并且内存效率极低。这是一个更好的实现：

# Memory overhead is BLOCKSIZE * itemsize. Should be at least ~1MB 
# for efficient HDD access.
BLOCKSIZE = 1024**2
# For numerical stability. The closer this is to mean(arr), the better.
PIVOT = arr[0]

n = len(arr)
sum_ = 0.
sum_sq = 0.
for block_start in xrange(0, n, BLOCKSIZE):
     block_data = arr[block_start:block_start + BLOCKSIZE]
     block_data -= PIVOT
     sum_ += np.sum(block_data)
     sum_sq += np.sum(block_data**2)
stdev = np.sqrt(sum_sq / n - (sum_ / n)**2)

Answer 2

import math
BLOCKSIZE = 1024**2
# For numerical stability. The closer this is to mean(arr), the better.
PIVOT = arr[0]


n = len(arr)
sum_ = 0.
sum_sq = 0.
for block_start in xrange(0, n, BLOCKSIZE):
    block_data = arr[block_start:block_start + BLOCKSIZE]
    block_data -= PIVOT
    sum_ += math.fsum(block_data)
    sum_sq += math.fsum(block_data**2)

stdev = np.sqrt(sum_sq / n - (sum_ / n)**2)

numpy.std 在 memmapped ndarray 上失败并出现 MemoryError

numpy.std on memmapped ndarray fails with MemoryError

python

numpy

out-of-memory

python-3.x