使用 .net SDK 提交 U-SQL 作业

Question

我引用了网站 https://github.com/toddkitta/azure-content/blob/master/articles/data-lake-analytics/data-lake-analytics-get-started-net-sdk.md for submitting a U-SQL job but the above GitHub sample showing error in the console application in the Authentication AcquireToken method.

请给我一个提交 U-SQL 作业的示例 using.Net SDK

已编辑： PlatformParameters 在我的 Visual Studio 中显示错误。我想，忘记包括 class。

Answer 1

The sample from msdn not having the AcquireTokenAsync method - Arron

有时文档滞后。他们可能更新了包，但还没有更新文档。使用异步方法，你会没事的。

您可以轻松地使用 AcquireTokenAsync 方法。像下面这样称呼它。这将等待异步方法和 return 结果，即使在同步方法中也是如此。

var token = authContext.AcquireTokenAsync(parameters).Result;

要使用的参数，看文档：AuthenticationContext.AcquireTokenAsync Method

您可以在线找到多个工作代码示例。 Here's one example.

Answer 2

Rick van den Bosch提到我们需要使用AcquireTokenAsync方法，我也跟着文档做了一个demo。并在我这边更改一些代码，然后它在我这边正常工作。以下是我的详细步骤。

准备：

1.Registry一个原生AD应用并添加Windows Azure Service Management API权限，更多详情请参考Azure official tutorials.之后我们可以从Azure Portal获取tenantId, appId, Redirect URI

2.Create Data Lake Analytics 和 Data Lake Store 帐户。我给data lake store中的文件夹或文件分配了权限，更多细节请参考另一个.

3.Prepare 本地文件夹中的脚本文件，我从你提到的 document.

中获取脚本

C:\Tom\SampleUSQLScript.txt

4.Upload SearchLog.tsv 到 Azure 存储帐户并将其设置为 public

步骤：

1.Create一个console项目和对应SDK的参考，详情请参考packages.config部分。

2.Add如何获取TokenCredentials函数

   public static TokenCredentials AuthenticateUser(string tenantId, string resource, string appClientId, Uri appRedirectUri, string userId = "")
        {
            var authContext = new AuthenticationContext("https://login.microsoftonline.com/" + tenantId);

            var tokenAuthResult = authContext.AcquireTokenAsync(resource, appClientId, appRedirectUri, new PlatformParameters(PromptBehavior.Auto),
                 UserIdentifier.AnyUser).Result;

            return new TokenCredentials(tokenAuthResult.AccessToken);
        }

测试 TokenCredentials 函数

3.Add SetupClients 函数

 public static void SetupClients(TokenCredentials tokenCreds, string subscriptionId)
    {
        _adlaClient = new DataLakeAnalyticsAccountManagementClient(tokenCreds) {SubscriptionId = subscriptionId};

        _adlaJobClient = new DataLakeAnalyticsJobManagementClient(tokenCreds);

        _adlaCatalogClient = new DataLakeAnalyticsCatalogManagementClient(tokenCreds);

        _adlsClient = new DataLakeStoreAccountManagementClient(tokenCreds) {SubscriptionId = subscriptionId};

        _adlsFileSystemClient = new DataLakeStoreFileSystemManagementClient(tokenCreds);
    }

4.Add SubmitJobByPath 函数

 public static string SubmitJobByPath(string scriptPath, string jobName)
        {
            var script = File.ReadAllText(scriptPath);
            var jobId = Guid.NewGuid();
            var properties = new USqlJobProperties(script);
            var parameters = new JobInformation(jobName, JobType.USql, properties, priority: 1000, degreeOfParallelism: 1);
            var jobInfo = _adlaJobClient.Job.Create(_adlaAccountName,jobId, parameters);
            return jobId.ToString();
        }

5.Add其他相关函数

 public static void UploadFile(string srcFilePath, string destFilePath, bool force = true)
        {
            var parameters = new UploadParameters(srcFilePath, destFilePath, _adlsAccountName, isOverwrite: force);
            var frontend = new DataLakeStoreFrontEndAdapter(_adlsAccountName, _adlsFileSystemClient);
            var uploader = new DataLakeStoreUploader(parameters, frontend);
            uploader.Execute();
        }

        // Download file
  public static void DownloadFile(string srcPath, string destPath)
        {
            var stream = _adlsFileSystemClient.FileSystem.Open(srcPath, _adlsAccountName);
            var fileStream = new FileStream(destPath, FileMode.Create);

            stream.CopyTo(fileStream);
            fileStream.Close();
            stream.Close();
        }


 public static JobResult WaitForJob(string jobId)
        {
            var jobInfo = _adlaJobClient.Job.Get(_adlaAccountName,Guid.Parse(jobId));
            while (jobInfo.State != JobState.Ended)
            {
                jobInfo = _adlaJobClient.Job.Get(_adlaAccountName, Guid.Parse(jobId)); 
            }
            return jobInfo.Result.Value;
        }

public static void WaitForNewline(string reason, string nextAction = "")
        {
            if (!String.IsNullOrWhiteSpace(nextAction))
            {
                Console.WriteLine(reason + "\r\nPress ENTER to continue...");
                Console.ReadLine();
                Console.WriteLine(nextAction);
            }
            else
            {
                Console.WriteLine(reason + "\r\nPress ENTER to continue...");
                Console.ReadLine();
            }

6.Add测试提交作业代码。

 private static void Main(string[] args)
    {
        _adlsAccountName = "data lake store account"; // TODO: Replace this value with the name for a created Store account.
        _adlaAccountName = "data lake analytics"; // TODO: Replace this value with the name for a created Analytics account.
        string localFolderPath = @"C:\tom\"; // TODO: Make sure this exists and contains the U-SQL script.

        // Authenticate the user
        // For more information about applications and instructions on how to get a client ID, see: 
        // https://azure.microsoft.com/en-us/documentation/articles/resource-group-create-service-principal-portal/
        var tokenCreds = AuthenticateUser("common", "https://management.core.windows.net/",
            "application id", new Uri("http://localhost")); // TODO: Replace applicaion id and redirect url values.

        SetupClients(tokenCreds, "subscription id"); // TODO: Replace subscription value.

        // Run sample scenarios

        // Transfer the source file from a public Azure Blob container to Data Lake Store.
        CloudBlockBlob blob = new CloudBlockBlob(new Uri("https://tomnew.blob.core.windows.net/adls-sample-data/SearchLog.tsv"));
        blob.DownloadToFile(localFolderPath + "SearchLog.tsv", FileMode.Create); // from WASB
        UploadFile(localFolderPath + "SearchLog.tsv", "/mytempdir/SearchLog.tsv"); // to ADLS
        WaitForNewline("Source data file prepared.", "Submitting a job.");

        // Submit the job
        string jobId = SubmitJobByPath(localFolderPath + "SampleUSQLScript.txt", "My First ADLA Job");
        WaitForNewline("Job submitted.", "Waiting for job completion.");

        // Wait for job completion
        WaitForJob(jobId);
        WaitForNewline("Job completed.", "Downloading job output.");


    }

7.Debug 来自本地并从 Azure 门户检查结果。

packages.config

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Microsoft.Azure.KeyVault.Core" version="1.0.0" targetFramework="net46" />
  <package id="Microsoft.Azure.Management.DataLake.Analytics" version="3.0.0" targetFramework="net46" />
  <package id="Microsoft.Azure.Management.DataLake.Store" version="1.0.4" targetFramework="net46" />
  <package id="Microsoft.Azure.Management.DataLake.StoreUploader" version="1.0.1-preview" targetFramework="net46" />
  <package id="Microsoft.Data.Edm" version="5.8.2" targetFramework="net46" />
  <package id="Microsoft.Data.OData" version="5.8.2" targetFramework="net46" />
  <package id="Microsoft.Data.Services.Client" version="5.8.2" targetFramework="net46" />
  <package id="Microsoft.IdentityModel.Clients.ActiveDirectory" version="3.14.1" targetFramework="net46" />
  <package id="Microsoft.Rest.ClientRuntime" version="2.3.8" targetFramework="net46" />
  <package id="Microsoft.Rest.ClientRuntime.Azure" version="3.3.7" targetFramework="net46" />
  <package id="Newtonsoft.Json" version="6.0.8" targetFramework="net46" />
  <package id="System.ComponentModel.EventBasedAsync" version="4.0.11" targetFramework="net46" />
  <package id="System.Dynamic.Runtime" version="4.0.0" targetFramework="net46" />
  <package id="System.Linq.Queryable" version="4.0.0" targetFramework="net46" />
  <package id="System.Net.Requests" version="4.0.11" targetFramework="net46" />
  <package id="System.Spatial" version="5.8.2" targetFramework="net46" />
  <package id="WindowsAzure.Storage" version="8.1.4" targetFramework="net46" />
</packages>

更新：

如果我们想使用静默登录，我们可以使用下面的代码来获取tokenCreds var tokenCreds = AuthenticateSlientUser("https://management.core.windows.net/", tenantId, applicationId, secretKey)

public static TokenCredentials AuthenticateSlientUser(string resource,string tenantId, string appClientId, string secretKey)
        {
            var authContext = new AuthenticationContext("https://login.microsoftonline.com/" + tenantId);

            var tokenAuthResult = authContext.AcquireTokenAsync(resource, new ClientCredential(appClientId, secretKey)).Result;

            return new TokenCredentials(tokenAuthResult.AccessToken);
        }

使用 .net SDK 提交 U-SQL 作业

Submit a U-SQL job using .net SDK

.net

sdk

azure

azure-data-lake

u-sql