顶点运行在 ADLA 中准备时间过长

Question

我有一个 U-SQL 脚本，当我运行脚本针对单个文件一切正常，但是针对所有文件将其打开到运行我每次都会收到以下错误并且作业永远不会超过 'Preparing' 阶段（花费大约 30 分钟):

System: E_SYSTEM_COMPILER_COMPILERINTERNALERROR
SEVERITY: ERROR
DESCRIPTION: Compilation failed due to internal system error.
MESSAGE: Compilation failed due to internal system error. (255)
DETAILS: Compilation failed for Deep.; JobId: 6ddc1049-c064-46d4-    b31f-1073ffd2cc99; Duration: 00:31:10.4668250; Exit Code: 255; Configuration: kobo_vnext_signed_5395848; Runtime: kobo_vnext_signed_5395848; Output: Deleted file D:\data\CCS\jobsddc1049-c064-46d4-b31f-1073ffd2cc99_v0\securityInfo.
Deleted file D:\data\CCS\jobsddc1049-c064-46d4-b31f-1073ffd2cc99_v0\jobEncryptionKey.
Server Version: 10.2.2.0
Client Version: 10.2.2.0
Target: Cluster
Stat: FetchReal
SkipPath: False
Syntax Parsing finished.
Resource Resolution finished.
Input/Output Resolution finished.
HandleCodeBehind finished.
Wait until structured stream metadata loading end finished.
Binder Pass finished.
Semantic Analyzer finished.
Error Reporter finished.
OptTree Generator finished.

--==== CQO Optimization ====--
Traceflag 608 is ON
Traceflag 551 is ON
Traceflag 402 is ON
Traceflag 550 is ON
Traceflag 21 is ON
Traceflag 552 is ON
Traceflag 554 is ON
Traceflag 515 is ON
Traceflag 534 is ON
Traceflag 541 is ON
Traceflag 435 is ON
Traceflag 106 is ON
Traceflag 62 is ON
Traceflag 571 is ON
Traceflag 514 is ON
Traceflag 568 is ON
Traceflag 569 is ON
Traceflag 203 is ON
Traceflag 546 is ON
Traceflag 562 is ON
Traceflag 563 is ON
Traceflag 505 is ON
Traceflag 506 is ON
Traceflag 542 is ON
Polaris scale up query execution is not enabled
Timer[CQO - Init] = 0.0172413 s
Timer[CQO - CopyIn] = 0.154232 s
Timer[CQO - Search - Init] = 0.000229223 s
Timer[CQO - Search - NNF] = 0.0124772 s
Timer[CQO - Search - PreOptimization] = 0.000576905 s
Pruning partitions and streamsets
Simplifying relational tree

Timer[CQO - Total] = 749.761 s
CQO: Optimization failed.


;

RESOLUTION

在 VS2017 的“诊断”选项卡上我还看到：

Vertex ran too long: 00:31:10

这可能是什么原因造成的？我正在努力解决问题

Answer 1

您可能想尝试 FileSet 选项 (https://github.com/Azure/AzureDataLake/blob/master/docs/Release_Notes/2017/2017_04_24/USQL_Release_Notes_2017_04_24.md#input-file-set-scales-orders-of-magnitudes-better-opt-in-statement-is-now-provided)

我运行几周前遇到过类似的问题。我首先通过增加 AU 的数量来解决问题。但是，使用

 SET @@FeaturePreviews = "FileSetV2Dot5:on";

使用较少量的 AU 有助于缩短编译时间

顶点运行在 ADLA 中准备时间过长

Vertex Ran Too Long on Preparing in ADLA

azure

azure-data-lake

u-sql

顶点 运行 在 ADLA 中准备时间过长

Vertex Ran Too Long on Preparing in ADLA

azure

azure-data-lake

u-sql

顶点运行在 ADLA 中准备时间过长