Azure Databricks：将文件保存在 Azure Datalake 目录文件夹中

Question

我想将文件保存在我的 azure datalake gen2 目录中，因为它不起作用。

当我在没有目录的情况下保存文件时它可以工作，但是对于目录“数据”它不起作用。

这是代码：

# Create a file in local data directory to upload and download
local_path = "./Data"
local_file_name = "quickstart" + str(uuid.uuid4()) + ".txt"
upload_file_path = os.path.join(local_path, local_file_name)

# Write text to the file
file = open(upload_file_path, 'w')
file.write("Hello, World!")
file.close()

# Create a blob client using the local file name as the name for the blob
blob_client = blob_service_client.get_blob_client(container=container_name, blob=local_file_name)

print("\nUploading to Azure Storage as blob:\n\t" + local_file_name)

# Upload the created file
with open(upload_file_path, "rb") as data:
    blob_client.upload_blob(data)

这里是错误：

FileNotFoundError: [Errno 2] No such file or directory: './Data/quickstart564851de-67a3-4b28-9af1-049298f26408.txt'
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<command-2012583145450168> in <module>
      5 
      6 # Write text to the file
----> 7 file = open(upload_file_path, 'w')
      8 file.write("Hello, World!")
      9 file.close()

FileNotFoundError: [Errno 2] No such file or directory: './Data/quickstart564851de-67a3-4b28-9af1-049298f26408.txt'

我的数据湖中存在“数据”文件夹：

from azure.storage.filedatalake import FileSystemClient

file_system = FileSystemClient.from_connection_string(connect_str, file_system_name="flatdata")

paths = file_system.get_paths()

for path in paths:
    print(path.name + '\t')

输出：

Data    
doto    
new.csv 
quickstart.csv  
quickstartc0afe722-3851-4f5a-a6fa-83fd97389c43.txt  
saved.csv

这是微软的文档：click here to see the Documentation

感谢您的帮助

Answer 1

参考documentation。我创建了一个测试，用于将本地文件上传到我的 Azure 数据湖的 folder。
这是我的本地项目文件结构：

文件已上传到我的 Azure 数据湖的 folder。

这是我的python代码

    import os, uuid
    from azure.storage.filedatalake import DataLakeServiceClient

    # Create a file in local data directory to upload and download
    local_path = "./data"
    local_file_name = "quickstart" + str(uuid.uuid4()) + ".txt"
    upload_file_path = os.path.join(local_path, local_file_name)

    # Write text to the file
    file = open(upload_file_path, 'w')
    file.write("Hello, World!")
    file.close()
    
    #upload the file to the folder
    file_system_client = service_client.get_file_system_client(file_system="test10")

    directory_client = file_system_client.get_directory_client("data")

    file_client = directory_client.create_file(local_file_name)

    local_file = open(upload_file_path, 'rb')

    file_contents = local_file.read()

    file_client.append_data(data=file_contents, offset=0, length=len(file_contents))

    file_client.flush_data(len(file_contents))

Azure Databricks：将文件保存在 Azure Datalake 目录文件夹中

Azure Databricks : Save file in Azure Datalake directory folder

python-3.x

databricks

azure-databricks

azure-data-lake-gen2