使用 Azure 存储数据移动库防止多次传输操作

Question

我刚开始使用Azure Storage Data Movement Library (DML) having code similar to public DML sample code。用例是（增量地）将本地目录同步到 blob 存储多次，同时目录中的某些文件会不时更新。

UploadDirectoryOptions options = new UploadDirectoryOptions
{
    SearchPattern = "*.*",
    Recursive = false,
    BlobType = BlobType.BlockBlob
};

// Register for transfer event.
DirectoryTransferContext context = new DirectoryTransferContext();

//only copy newer files - similar to AzCopy /XO /XN switches
//https://github.com/Azure/azure-storage-net-data-movement/issues/12
context.ShouldOverwriteCallback = (source, destination) =>
{
    var sourceFile = new FileInfo((string)source);
    var destBlob = destination as CloudBlob;
    return sourceFile.LastWriteTimeUtc > destBlob.Properties.LastModified;
};

// Start the upload
var transferStatus = await TransferManager.UploadDirectoryAsync(sourceDirPath, destDir, options, context);

目前传输操作是由应用程序事件触发的。在短时间内试验了几个事件后，这引发了一个异常，如下所示。由于 TransferManager 是静态的 class，如何在我的应用程序中解决此要求并防止此类异常？

Microsoft.WindowsAzure.Storage.DataMovement.TransferException was unhandled
Message: An unhandled exception of type 'Microsoft.WindowsAzure.Storage.DataMovement.TransferException' occurred in mscorlib.dll
Additional information: A transfer operation with the same source and destination already exists.

Answer 1

您是否向 TransferManager 添加了超过 1 个具有相同 source/destination 的传输任务（例如，运行以上代码在您的应用程序中多次添加）。如果是这样，您将遇到问题，因为 DMlib 不允许运行超过 1 个传输任务同时具有相同的 source/destination。

Answer 2

以下是运行上传10次的代码。请根据您的要求进行修改。请注意，您需要等待之前的任务完成，然后添加一个具有相同源和目标以及新传输上下文的新传输任务。

        for (int i = 0; i < 10; i++)
        {
            // Create Transfer Context
            DirectoryTransferContext context = new DirectoryTransferContext();

            //if dest blob exist just overwrite it 
            context.ShouldOverwriteCallback = TransferContext.ForceOverwrite;

            // Start the upload and wait for it to finish
            Task<TransferStatus> task = TransferManager.UploadDirectoryAsync(sourceDirPath, destDir, options, context);
            task.Wait();

            //Add some code to check the task.Result, following code only print the result
            Console.WriteLine(string.Format("{0}\t{1}\t{2}\t{3}", task.Result.NumberOfFilesTransferred, task.Result.NumberOfFilesFailed, task.Result.NumberOfFilesSkipped, task.Result.BytesTransferred));
        }

Answer 3

我觉得你的问题没有优化好。我觉得你遇到了和我一样的问题。让我简单解释一下。你是运行异步 DirectoryUpload，当然在这种情况下 "multiple transfer operations" 会。

在大多数情况下，如果跳过文件，则会发生此异常。这就是为什么您收到 "A transfer operation with the same source and destination already exists." 错误的原因。在这种情况下，进度将跳过图像，您必须如下捕获这些图像。查看完整示例 here

 DirectoryTransferContext context = new DirectoryTransferContext();
            context.FileTransferred += FileTransferredCallback;
            context.FileFailed +=FileFailedCallback;
            context.FileSkipped += FileSkippedCallback;

            context.SetAttributesCallback = (destination) =>
            {
                CloudBlob destBlob = destination as CloudBlob;
                destBlob.Properties.ContentType = "image/png";
            };

            context.ShouldTransferCallback = (source, destination) =>
            {
                // Can add more logic here to evaluate whether really need to transfer the target.
                return true;
            };

            // Start the upload
            var trasferStatus = await TransferManager.UploadDirectoryAsync(sourceDirPath, destDir, options, context);

private static void FileTransferredCallback(object sender, TransferEventArgs e)
    {
        Console.WriteLine("Transfer Succeeds. {0} -> {1}.", e.Source, e.Destination);
    }

    private static void FileFailedCallback(object sender, TransferEventArgs e)
    {
        Console.WriteLine("Transfer fails. {0} -> {1}. Error message:{2}", e.Source, e.Destination, e.Exception.Message);
    }

    private static void FileSkippedCallback(object sender, TransferEventArgs e)
    {
        Console.WriteLine("Transfer skips. {0} -> {1}.", e.Source, e.Destination);
    }

示例中缺少的是如何处理覆盖现有图像（如果它们不相同）。

我不确定这是否在 0.3 版中可用，但我也是在安装 0.4.1 版 Microsoft.WindowsAzure.Storage.DataMovement 软件包后才发现的。下面的代码将比较源和目标并决定是否应该覆盖。

   context.ShouldOverwriteCallback = (source, destination) =>
                    {
                      if(TransferContext.Equals(source, destination)==true)
                            return false;
                      else
                            return true;                        

                    };

使用 Azure 存储数据移动库防止多次传输操作

Prevent multiple transfer operations with Azure Storage Data Movement Library

c#

azure

azure-storage