pickle 在 Ubuntu 14.04 上引发 EOFError 但在 Mac OS 和 Linux Mint 上有效

Question

我正在使用 Python 的 pickle 模块保存一些大数据。更准确地说，我将两个对象转储到同一个泡菜文件中，如下所示：

def save(sim):
    if sim.tstamp > 0:
        with open(filepath(sim.identifier), 'wb') as f:
            pickle.dump(sim.serialize(), f)
            pickle.dump(sim, f)
    else:
        raise ValueError("Simulation not yet simulated")

sim是自写的class，有不同的属性和一个叫serialize()的方法。此方法使用类似以下内容将所有属性及其值转换为字典：

def serialize(self):
    keys = ['height', 'diameter', 'rpm', 'packaging']
    serialized = dict((key, getattr(self, key)) for key in keys)
    serialized['sectors'] = [sector.serialize() for sector in self.sectors]
    return serialized

所以基本上 sim.serialize() returns 一个字典，作为第一个对象被转储到 pickle 文件中。第二个对象是模拟对象本身。这是由于每个模拟对象的尺寸都很大。提供字典可以实现一种摘要，以防止在并非真正必要时加载完整的模拟。

从我使用的 pickle 文件加载数据：

def load(fname, params_only=False):
    simulation = find(fname) # find returns file name
    try:
        with open(simulation, 'r') as json_file:
            return json.load(json_file)
    except (UnicodeDecodeError, ValueError) as e:
        with open(simulation, 'rb') as pickle_file:
            params = pickle.load(pickle_file)
            if params_only:
                return params
            return pickle.load(pickle_file)

由于较旧的 API 也可以处理 JSON 文件，因此我需要同时处理 JSON 和 pickle 文件。

转储对象按预期工作（没有出现错误消息）。但是，在请求某些数据图时，API 需要解开引发以下错误的数据：

[2016-08-07 14:39:34,637] ERROR in app: Exception on /api/simulations/20160807_123707581/heatflux_distribution.png [GET]
Traceback (most recent call last):
 File "/home/abkos/abkosproject/abkos/_tools.py", line 36, in load
   return json.load(json_file)
 File "/usr/lib/python3.4/json/__init__.py", line 265, in load
   return loads(fp.read(),
 File "/usr/lib/python3.4/codecs.py", line 319, in decode
   (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1988, in wsgi_app
   response = self.full_dispatch_request()
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1641, in full_dispatch_request
   rv = self.handle_user_exception(e)
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1544, in handle_user_exception
   reraise(exc_type, exc_value, tb)
 File "/usr/local/lib/python3.4/dist-packages/flask/_compat.py", line 33, in reraise
   raise value
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1639, in full_dispatch_request
   rv = self.dispatch_request()
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1625, in dispatch_request
   return self.view_functions[rule.endpoint](**req.view_args)
 File "./app.py", line 118, in plot_figure
   img = plotter(identifier, figname, request.args.items())
 File "./app.py", line 122, in plotter
   sim = Simulation.from_file(identifier)
 File "/home/abkos/abkosproject/abkos/calculation.py", line 56, in from_file
   params_or_simulation = load(fname, params_only=new)
 File "/home/abkos/abkosproject/abkos/_tools.py", line 42, in load
   return pickle.load(pickle_file)
_pickle.UnpicklingError
10.0.2.2 - - [07/Aug/2016 14:39:34] "GET /api/simulations/20160807_123707581/heatflux_distribution.png HTTP/1.1" 500 -
[2016-08-07 14:39:38,691] ERROR in app: Exception on /api/simulations/20160807_123707581/heatmap_axial_temperatures.png [GET]
Traceback (most recent call last):
 File "/home/abkos/abkosproject/abkos/_tools.py", line 36, in load
   return json.load(json_file)
 File "/usr/lib/python3.4/json/__init__.py", line 265, in load
   return loads(fp.read(),
 File "/usr/lib/python3.4/codecs.py", line 319, in decode
   (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1988, in wsgi_app
   response = self.full_dispatch_request()
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1641, in full_dispatch_request
   rv = self.handle_user_exception(e)
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1544, in handle_user_exception
   reraise(exc_type, exc_value, tb)
 File "/usr/local/lib/python3.4/dist-packages/flask/_compat.py", line 33, in reraise
   raise value
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1639, in full_dispatch_request
   rv = self.dispatch_request()
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1625, in dispatch_request
   return self.view_functions[rule.endpoint](**req.view_args)
 File "./app.py", line 118, in plot_figure
   img = plotter(identifier, figname, request.args.items())
 File "./app.py", line 122, in plotter
   sim = Simulation.from_file(identifier)
 File "/home/abkos/abkosproject/abkos/calculation.py", line 56, in from_file
   params_or_simulation = load(fname, params_only=new)
 File "/home/abkos/abkosproject/abkos/_tools.py", line 42, in load
   return pickle.load(pickle_file)
EOFError
10.0.2.2 - - [07/Aug/2016 14:39:38] "GET /api/simulations/20160807_123707581/heatmap_axial_temperatures.png HTTP/1.1" 500 -

使用 Python 3.4.3，代码在 MacOS 和 Linux Mint 上运行良好。但是，它不适用于 Ubuntu 14.04 (Python 3.4.3)。

关于从哪里开始调查错误有什么想法吗？

Answer 1

由于我找不到任何编码错误，所以我将这三个系统相互比较，发现从软件的角度来看它们几乎完全相同。他们都在使用 Python 3.4.3 64 位，pickle 是 Python 默认模块的一部分。

所以我得出结论，这可能是与硬件相关的问题。由于 MacOS 机器和 Mint 机器都是 'real' 机器而 Ubuntu 机器只是一台虚拟机，所以我比较了一般的硬件规格，比如 CPU 和 RAM 的数量。

真机分别配备8 GB RAM。 Mint 12 GB。所以我将 VM 的 RAM 从大约 1 GB 增加到 3 GB，一切正常。

所以即使没有MemoryError或类似的提升，EOFError也可能与内存有关。

pickle 在 Ubuntu 14.04 上引发 EOFError 但在 Mac OS 和 Linux Mint 上有效

pickle raises EOFError on Ubuntu 14.04 but works on Mac OS and Linux Mint

python

macos

ubuntu

pickle

python-3.x