pickle.dumps() 是否有详细的日志记录模式？

Question

我正在尝试使用以下代码将大约 25000 张图像编码到 pickle 文件中

f = open("dataset_encoding.pickle", "wb")
f.write(pickle.dumps(sampleDataset))
f.close()

Python 在我的 PC 上执行此功能会花费大量时间，是否有 pickle.dumps() 的详细日志记录选项，以便我知道它仍然是运行？

我找不到任何东西here

Answer 1

如评论中所述，我试图对长度为 4 的列表进行 pickle。为了减少每个 pickle 文件的内存使用量，我决定将我的列表保存为 4 个不同的 pickle。以下是我在酸洗完成时用于打印 ProgressBar 的代码：

import pickle

def printProgressBar (iteration, total, prefix = '', suffix = '', decimals = 1, length = 100, fill = '█', printEnd = "\r"):
"""
Call in a loop to create terminal progress bar
@params:
    iteration   - Required  : current iteration (Int)
    total      - Required  : total iterations (Int)
    prefix    - Optional  : prefix string (Str)
    suffix    - Optional  : suffix string (Str)
    decimals    - Optional  : positive number of decimals in percent complete (Int)
    length    - Optional  : character length of bar (Int)
    fill        - Optional  : bar fill character (Str)
    printEnd    - Optional  : end character (e.g. "\r", "\r\n") (Str)
"""
    percent = ("{0:." + str(decimals) + "f}").format(100 * (iteration / float(total)))
    filledLength = int(length * iteration // total)
    bar = fill * filledLength + '-' * (length - filledLength)
    print('\r%s |%s| %s%% %s' % (prefix, bar, percent, suffix), end = printEnd)
    if iteration == total: 
        print() 

def verbosePickle(data_list, filename, prefix_text):
    print("sampleDataset["+str(index)+"] Length :"+str(len(sample)))
    with open(filename, "wb") as f:
        pickle.dump(len(data_list), f)
        printProgressBar(0, len(data_list)-1, prefix = prefix_text, suffix = "Complete", length = 50)
        for data_index, data in enumerate(data_list):
            printProgressBar(data_index, len(data_list)-1, prefix = prefix_text, suffix = "Complete", length = 50)
            pickle.dump(data, f)

for index, sample in enumerate(sampleDataset):
    print("Saving Pickle Files")
    verbosePickle(sample, "sampleDataset"+index+".pickle", "Saving pickle "+index+" :")

标准输出：

Saving pickle 0 : |█████████████████████-----------------------------| 44.0% Complete

pickle.dumps() 是否有详细的日志记录模式？

Is there a verbose logging mode for pickle.dumps()?

python

pickle