dask 在 jupyterlab 上运行时存储文件在哪里

Question

我运行正在使用 jupyterlab。我正在尝试在存储我的 python 文件的主目录中保存一些文件，并且它是运行正确但我无法找到我的文件保存位置。所以我在主目录中创建了一个名为 output 的文件夹以在其中保存文件，但是当我在其中保存文件时出现以下错误：

PermissionError: [Errno 13] Permission denied: b'/home/jovyan/Output/20190101_0200'

这是我的尝试：

from dask_gateway import Gateway
gateway = Gateway(
    address="http://traefik-pangeo-dask-gateway/services/dask-gateway",
    public_address="https://pangeo.aer-gitlab.com/services/dask-gateway",
    auth="jupyterhub",
)
options = gateway.cluster_options()
options

cluster = gateway.new_cluster(
    cluster_options=options,
)
cluster.adapt(minimum=10, maximum=50)
client = cluster.get_client()
cluster
client

def get_turb(file, name):
    
    d=[name[0:4],name[4:6],name[6:8],name[9:11],name[11:13]] 
    f_zip = gzip.decompress(file)

    yr=d[0]
    mo=d[1]
    da=d[2]
    hr=d[3]
    mn=d[4]
    
    fs = s3fs.S3FileSystem(anon=True)

    period = pd.Period(str(yr)+str('-')+str(mo)+str('-')+str(da), freq='D')
    # period.dayofyear
    dy=period.dayofyear

    cc=[7,8,9,10,11,12,13,14,15,16]  #look at the IR channels only for now
    dat = xr.open_dataset(f_zip)
    dd=dat[['recNum','trackLat','trackLon','altitude','maxEDR']]
    dd=dd.to_dataframe()
    dd = dd.sort_values(by=['maxEDR'])
    dd = dd.dropna()
    dd['num'] = np.arange(len(dd))
    dd.to_csv('Output/edr.csv') <----- Saving the file

edr_files = []

for i in range(2):
    print(names[i])
    s3_ds = dask.delayed(get_turb)(filedata[i], names[i])
    edr_files.append(s3_ds)

edr_files = dask.compute(*edr_files)

请让我知道我做错了什么或可能的解决方案。

而且当我尝试使用以下代码直接将文件保存在 S3 存储桶上时：

    s3.Bucket('temp').upload_file(file_name+'.zip', file_name+'.zip')

它抛出了这个错误：

distributed.protocol.pickle - INFO - Failed to serialize <function get_temp at 0x7f20a9cb8550>. Exception: cannot pickle '_thread.lock' object
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/worker.py in dumps_function(func)
   3319         with _cache_lock:
-> 3320             result = cache_dumps[func]
   3321     except KeyError:

/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/utils.py in __getitem__(self, key)
   1572     def __getitem__(self, key):
-> 1573         value = super().__getitem__(key)
   1574         self.data.move_to_end(key)

/srv/conda/envs/notebook/lib/python3.8/collections/__init__.py in __getitem__(self, key)
   1009             return self.__class__.__missing__(self, key)
-> 1010         raise KeyError(key)
   1011     def __setitem__(self, key, item): self.data[key] = item

KeyError: <function get_temp at 0x7f20a9cb8550>

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/protocol/pickle.py in dumps(x, buffer_callback, protocol)
     52                 buffers.clear()
---> 53                 result = cloudpickle.dumps(x, **dump_kwargs)
     54         elif not _always_use_pickle_for(x) and b"__main__" in result:

/srv/conda/envs/notebook/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py in dumps(obj, protocol, buffer_callback)
     72             )
---> 73             cp.dump(obj)
     74             return file.getvalue()

/srv/conda/envs/notebook/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py in dump(self, obj)
    562         try:
--> 563             return Pickler.dump(self, obj)
    564         except RuntimeError as e:

TypeError: cannot pickle '_thread.lock' object

Answer 1

您似乎运行 dask 和 jupyterlab 在 docker 中？

也许你应该添加一些标志，比如 fellowing:

--user root \
-v new-longhorn-volume:/home/jovyan \
-e CHOWN_HOME=yes \
-e CHOWN_HOME_OPTS='-R' \

Answer 2

尝试使用 s3fs 而不是 boto3 在 S3 上上传文件。这可能有用。

dask 在 jupyterlab 上运行时存储文件在哪里

Where does dask store files while running on juputerlab

python

python-3.x

dask

dask-delayed

dask-distributed

dask 在 jupyterlab 上 运行 时存储文件在哪里

Where does dask store files while running on juputerlab

python

python-3.x

dask

dask-delayed

dask-distributed

dask 在 jupyterlab 上运行时存储文件在哪里