如何使用 PowerShell 或 Azure CLI 检索 Azure 存储表中的所有记录

How retrieve all of records in Azure storage tables using PowerShell or Azure CLI

我正在使用 Azure CLI,我可以连接到存储 table 并使用波纹管循环中的代码从 table 中为我的存储中的每个 table 获取项目:

"get metrics from $table"
$temp_result = (az storage entity query --table-name $table --connection-string $connection_string --accept "none" --filter "CounterName eq '\Memory\PercentUsedMemory' or CounterName eq '\NetworkInterface\BytesReceived' or CounterName eq '\NetworkInterface\BytesTransmitted' or CounterName eq '\Processor\PercentProcessorTime'" --select "TIMESTAMP" "Average" "CounterName" | ConvertFrom-Json)
($temp_result.items).Length

结果是这样的:

get metrics from WADMetricsPT1HP10DV2S20171007
88
get metrics from WADMetricsPT1HP10DV2S20171017
688
get metrics from WADMetricsPT1HP10DV2S20171027
464
get metrics from WADMetricsPT1HP10DV2S20180326
88
get metrics from WADMetricsPT1HP10DV2S20180405
72
get metrics from WADMetricsPT1MP10DV2S20171007
1000
get metrics from WADMetricsPT1MP10DV2S20171017
1000
get metrics from WADMetricsPT1MP10DV2S20171027
1000
get metrics from WADMetricsPT1MP10DV2S20180326
1000
get metrics from WADMetricsPT1MP10DV2S20180405
1000

但在我的一个 table 中,我有超过 37000 条记录,我想获取所有这些记录。我试着像下面这样设置 --num-results 9999

"get metrics from $table"
$temp_result = (az storage entity query --table-name $table --connection-string $connection_string --accept "none" --filter "CounterName eq '\Memory\PercentUsedMemory' or CounterName eq '\NetworkInterface\BytesReceived' or CounterName eq '\NetworkInterface\BytesTransmitted' or CounterName eq '\Processor\PercentProcessorTime'" --select "TIMESTAMP" "Average" "CounterName" --num-results 9999 | ConvertFrom-Json)
($temp_result.items).Length

但它会return这个错误:

{"odata.error":{"code":"InvalidInput","message":{"lang":"en-US","value":"One of the request inputs is not valid.\nRequestId:d8ccda32-f002-0034-2619-cd42aa000000\nTime:2018-04-05T20:07:56.7891454Z"}}}

你能指导我哪里出了问题以及如何使用 Powershell 或 Azure CLI 获取 Azure 存储 tables 中的所有记录吗?

更新:

正如路肇兴在回答中所说,我尝试使用--marker。但是有一些问题。

首先,这是我当前的代码:

$connection_string = (az storage account show-connection-string --resource-group $resourceGroup --name $storageAccount | ConvertFrom-Json).connectionString
$table_list = (az storage table list --connection-string $connection_string | ConvertFrom-Json).name

if ($table_list.Length -eq 0)
{
    "There is no table in '$storageAccount' storage"
}
else
{
    "There is "+$table_list.Length+" tables stored in $storageAccount."
    ###########################################
    ## Find those tables that stored metrics ##
    ###########################################
    $tables = @()
    Foreach($temp_table_name in $table_list)
    {
        if ($temp_table_name.StartsWith("WADMetrics"))
        {
            $tables += $temp_table_name
        }
    }

    if ($tables.Length -eq 0)
    {
        "There is no table starting with 'WADMetrics' as prefix in storage ($storageAccount) tables, then we can't detect any Metric."
    }
    else
    {
        ($tables.Length).tostring()+" tables stored metrics"
        Foreach($table in $tables)
        {
            "get metrics from $table"
            $is_more_results = 0
            $nextMarker = @{}
            $nextpartitionkey = ""
            $nextrowkey = ""

            Do
            {
                $filter_str = "CounterName eq '\Memory\PercentUsedMemory' or CounterName eq '\NetworkInterface\BytesReceived' or CounterName eq '\NetworkInterface\BytesTransmitted' or CounterName eq '\Processor\PercentProcessorTime'"
                $temp_result

                if ($nextpartitionkey -ne "")
                {
                    "Call with marker"
                    $temp_result = az storage entity query --table-name $table --connection-string $connection_string --accept "minimal" --filter "CounterName eq '\Memory\PercentUsedMemory' or CounterName eq '\NetworkInterface\BytesReceived' or CounterName eq '\NetworkInterface\BytesTransmitted' or CounterName eq '\Processor\PercentProcessorTime'" --select "TIMESTAMP" "Average" "CounterName" --num-results 50 --marker $nextMarker
                }
                else
                {
                    "Call without marker"
                    $temp_result = az storage entity query --table-name $table --connection-string $connection_string --accept "minimal" --filter "CounterName eq '\Memory\PercentUsedMemory' or CounterName eq '\NetworkInterface\BytesReceived' or CounterName eq '\NetworkInterface\BytesTransmitted' or CounterName eq '\Processor\PercentProcessorTime'" --select "TIMESTAMP" "Average" "CounterName" --num-results 50
                }

                $temp_result = [string]$temp_result
                $temp_result = $temp_result | ConvertFrom-Json
                ($temp_result.items).Length

                if (($temp_result.nextMarker.nextpartitionkey).Length -gt 0)
                {
                    #there is more items in the requested query
                    $nextpartitionkey = $temp_result.nextMarker.nextpartitionkey
                    $nextrowkey = $temp_result.nextMarker.nextrowkey
                    $nextMarker["nextpartitionkey"] = $nextpartitionkey
                    $nextMarker["nextrowkey"] = $nextrowkey
                    $is_more_results = 1
                }
                else
                {
                    $is_more_results = 0
                }
            } While ($is_more_results -ne 0)
        }
    }
}

以上代码将return输出为:

There is 16 tables stored in testresource********.
10 tables stored metrics
get metrics from WADMetricsPT1HP10DV2S20171007
Call without marker
50

items                                                                                                                                                        
-----                                                                                                                                                        
{@{Average=10.095833333333333; CounterName=\Memory\PercentUsedMemory; TIMESTAMP=2017-10-16T22:00:00+00:00; etag=W/"datetime'2017-10-16T23%3A00%3A05.500086...
Call with marker
az : ERROR: 'str' object has no attribute 'get'
At line:66 char:36
+ ... mp_result = az storage entity query --table-name $table --connection- ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (ERROR: 'str' ob...attribute 'get':String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

Traceback (most recent call last):
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\knack\cli.py", line 197, in invoke
    cmd_result = self.invocation.execute(args)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\commands\__init__.py", line 347, in execute
    six.reraise(*sys.exc_info())
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\commands\__init__.py", line 319, in execute
    result = cmd(params)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\commands\__init__.py", line 180, in __call__
    return super(AzCliCommand, self).__call__(*args, **kwargs)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\knack\commands.py", line 109, in __call__
    return self.handler(*args, **kwargs)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\__init__.py", line 420, in default_command_handler
    result = op(**command_args)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\multiapi\cosmosdb\v2017_04_17\table\tableservice.py", line 730, in 
query_entities
    resp = self._query_entities(*args, **kwargs)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\multiapi\cosmosdb\v2017_04_17\table\tableservice.py", line 776, in 
_query_entities
    next_partition_key = None if marker is None else marker.get('nextpartitionkey')
AttributeError: 'str' object has no attribute 'get'
ConvertFrom-Json : Cannot bind argument to parameter 'InputObject' because it is null.
At line:75 char:47
+                 $temp_result = $temp_result | ConvertFrom-Json
+                                               ~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (:) [ConvertFrom-Json], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.ConvertFromJsonCommand

0

我尝试使用下面的代码直接传递 --marker:

if (($temp_result.nextMarker.nextpartitionkey).Length -gt 0)
{
    #there is more items in the requested query
    $nextpartitionkey = $temp_result.nextMarker.nextpartitionkey

    $nextMarker = $temp_result.nextMarker
    $is_more_results = 1
}

但错误是一样的:

There is 16 tables stored in testresource********.
10 tables stored metrics
get metrics from WADMetricsPT1HP10DV2S20171007
Call without marker
50

items                                                                                                                                                        
-----                                                                                                                                                        
{@{Average=10.095833333333333; CounterName=\Memory\PercentUsedMemory; TIMESTAMP=2017-10-16T22:00:00+00:00; etag=W/"datetime'2017-10-16T23%3A00%3A05.500086...
Call with marker
az : ERROR: 'str' object has no attribute 'get'
At line:66 char:36
+ ... mp_result = az storage entity query --table-name $table --connection- ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (ERROR: 'str' ob...attribute 'get':String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

Traceback (most recent call last):
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\knack\cli.py", line 197, in invoke
    cmd_result = self.invocation.execute(args)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\commands\__init__.py", line 347, in execute
    six.reraise(*sys.exc_info())
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\commands\__init__.py", line 319, in execute
    result = cmd(params)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\commands\__init__.py", line 180, in __call__
    return super(AzCliCommand, self).__call__(*args, **kwargs)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\knack\commands.py", line 109, in __call__
    return self.handler(*args, **kwargs)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\__init__.py", line 420, in default_command_handler
    result = op(**command_args)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\multiapi\cosmosdb\v2017_04_17\table\tableservice.py", line 730, in 
query_entities
    resp = self._query_entities(*args, **kwargs)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\multiapi\cosmosdb\v2017_04_17\table\tableservice.py", line 776, in 
_query_entities
    next_partition_key = None if marker is None else marker.get('nextpartitionkey')
AttributeError: 'str' object has no attribute 'get'
ConvertFrom-Json : Cannot bind argument to parameter 'InputObject' because it is null.
At line:75 char:47
+                 $temp_result = $temp_result | ConvertFrom-Json
+                                               ~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (:) [ConvertFrom-Json], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.ConvertFromJsonCommand

0

我什至尝试删除 --marker 并通过查询传递 nextpartitionkeynextrowkey!,它没有 return 任何错误,但它会 return总是一样,循环将永远持续下去。

if ($nextpartitionkey -ne "")
{
    $filter_str = $filter_str + " and PartitionKey eq '$nextpartitionkey' and RowKey eq '$nextrowkey'"
}
$temp_result = az storage entity query --table-name $table --connection-string $connection_string --accept "minimal" --filter $filter_str --select "TIMESTAMP" "Average" "CounterName" --num-results 50

更新 2:

我试过将标记传递给 stirng。波纹管字符串是我用作字符串的标记示例:

"nextMarker": {
    "nextpartitionkey":  "1!260!OjAwMkZzdWJzY3JpcHRpb25zOjAwMkYzMTM1Nzc1YTowMDJEMTA2ZDowMDJENGVmODowMDJEODlhYTowMDJEN2VkYjM2YjRjMzU2OjAwMkZyZXNvdXJjZUdyb3Vwcz
owMDJGVGVzdFJlc291cmNlR3JvdXAyOjAwMkZwcm92aWRlcnM6MDAyRk1pY3Jvc29mdDowMDJFQ29tcHV0ZTowMDJGdmlydHVhbE1hY2hpbmVzOjAwMkZjcHV1c2FnZXRlc3Q-",
    "nextrowkey":  "1!72!OjAwNUNNZW1vcnk6MDA1Q1BlcmNlbnRVc2VkTWVtb3J5X18yNTE4OTQxMzExOTk5OTk5OTk5"
}

我也试过这个作为字符串:

{
    "nextpartitionkey":  "1!260!OjAwMkZzdWJzY3JpcHRpb25zOjAwMkYzMTM1Nzc1YTowMDJEMTA2ZDowMDJENGVmODowMDJEODlhYTowMDJEN2VkYjM2YjRjMzU2OjAwMkZyZXNvdXJjZUdyb3Vwcz
owMDJGVGVzdFJlc291cmNlR3JvdXAyOjAwMkZwcm92aWRlcnM6MDAyRk1pY3Jvc29mdDowMDJFQ29tcHV0ZTowMDJGdmlydHVhbE1hY2hpbmVzOjAwMkZjcHV1c2FnZXRlc3Q-",
    "nextrowkey":  "1!72!OjAwNUNNZW1vcnk6MDA1Q1BlcmNlbnRVc2VkTWVtb3J5X18yNTE4OTQxMzExOTk5OTk5OTk5"
}

我都收到了这个错误:

az : ERROR: 'str' object has no attribute 'get'
At C:\Users\Reza\Desktop\ndbench\Azure\Automation_get_metrics\add_target_to_tables - runbook.ps1:87 char:36
+ ... mp_result = az storage entity query --table-name $table --connection- ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (ERROR: 'str' ob...attribute 'get':String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

Traceback (most recent call last):
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\knack\cli.py", line 197, in invoke
    cmd_result = self.invocation.execute(args)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\commands\__init__.py", line 347, in execute
    six.reraise(*sys.exc_info())
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\commands\__init__.py", line 319, in execute
    result = cmd(params)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\commands\__init__.py", line 180, in __call__
    return super(AzCliCommand, self).__call__(*args, **kwargs)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\knack\commands.py", line 109, in __call__
    return self.handler(*args, **kwargs)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\__init__.py", line 420, in default_command_handler
    result = op(**command_args)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\multiapi\cosmosdb\v2017_04_17\table\tableservice.py", line 730, in 
query_entities
    resp = self._query_entities(*args, **kwargs)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\multiapi\cosmosdb\v2017_04_17\table\tableservice.py", line 776, in 
_query_entities
    next_partition_key = None if marker is None else marker.get('nextpartitionkey')
AttributeError: 'str' object has no attribute 'get'
ConvertFrom-Json : Cannot bind argument to parameter 'InputObject' because it is null.
At C:\Users\Reza\Desktop\ndbench\Azure\Automation_get_metrics\add_target_to_tables - runbook.ps1:112 char:47
+                 $temp_result = $temp_result | ConvertFrom-Json
+                                               ~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (:) [ConvertFrom-Json], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.ConvertFromJsonCommand

那么在夏天,问题是我应该如何度过--marker?

您收到错误消息是因为不可能一次 return 9999 个结果。正如 Query Entities 文档所示,-num-results 的最大值为 1000。

为了获得 table 中的所有实体,您需要通过延续对象进行 az storage entity query 循环。要获取下一页查询结果,请将 --marker 设置为上一次迭代的 $temp_result.next_marker 的值。

这是 CLI 中的错误。我们将在下一个版本 v2.0.32 中修复。 基本上没有解析出提供给 --marker 参数的属性。

阅读此处了解更多信息:https://github.com/Azure/azure-cli/issues/6194