GCP 数据丢失防护 API 身份验证：是否需要使用服务帐户？

Question

我正在尝试使用 API 自动执行 DLP 扫描。唯一阻碍我完成这个项目的是身份验证。尽管我自己的帐户拥有对所有项目的组织所有者访问权限，但在每个项目中创建和使用具有 BigQuery、存储和 DLP 管理员权限的服务帐户似乎是避免权限被拒绝错误的唯一方法。这是 IAM 问题还是 DLP API 的某些要求以及令牌的使用？为什么我的令牌不起作用？我们有很多项目，将来可能会有更多项目，我们希望避免必须在每个项目中创建服务帐户的繁琐和乏味的问题，尤其是必须一次又一次地创建以保持脚本正常运行。自动化应该少而不是多。

我在下面的项目中使用自己的帐户令牌遇到的错误是 DLP API 确实已启用。

"error": { "code": 403, "message": "Cloud Data Loss Prevention (DLP) API has not been used in project ###### before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/dlp.googleapis.com/overview?project=###### then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.", "status": "PERMISSION_DENIED", "details": [ { "@type": "type.googleapis.com/google.rpc.Help", "links": [ { "description": "Google developers console API activation", "url": "https://console.developers.google.com/apis/api/dlp.googleapis.com/overview?project=######"

Answer 1

查看文档 Authenticating to the Cloud DLP API:

You must authenticate to the Cloud DLP API in order to use it. The Cloud DLP API can handle both API keys and authentication. The main distinction between these two methods is:

API keys identify the calling project—the app or site—that is making the call to an API.

Auth tokens identify a user—the person—that is using the project.

和

To use a service account to authenticate to the Cloud DLP API:

Follow the instructions to create a service account. Select JSON as your key type, and grant the user the DLP User role (roles/dlp.user).

一般来说，您应该遵循以下步骤：

enable billing
enable DLP API

create a service account，将角色授予服务帐户 roles/dlp.user 并下载 key.json 文件：

$ gcloud iam service-accounts create test-dlp --description "test-cloud" --display-name "test-dlp"
$ gcloud projects add-iam-policy-binding class-run --member serviceAccount:test-dlp@class-run.iam.gserviceaccount.com --role roles/dlp.user
$ gcloud iam service-accounts keys create key.json --iam-account test-dlp@class-run.iam.gserviceaccount.com

设置环境变量：

export GOOGLE_APPLICATION_CREDENTIALS=[PATH_TO_key.json_FILE]

运行你的脚本

此外，看看 Quickstart: Using the command-line tool section Permissions:

Inspecting content requires the serviceusage.services.use permission for the project that's specified in parent. The roles/editor, roles/owner, and roles.dlp.user roles contain the required permission or you can define your own custom role.

如果问题仍然存在，请尝试按照以下步骤进行故障排除：

使用命令 gcloud auth list
检查 DLP API 是否启用 gcloud services list --enabled | grep DLP
使用命令 gcloud auth activate-service-account 激活您的 DLP 服务帐户并再次运行您的脚本

并使用命令和输出更新您的问题。另外，请说明您在脚本中以何种方式使用 DLP 服务帐户。

Answer 2

是的，您可以将自己的用户帐户（所有者）用于 api。

  1. gcloud auth login 

  2. gcloud auth list 
    #ACTIVE  ACCOUNT with owner role
    *       user@cloud.com .

  3. gcloud auth print-access-token
    #xxxxxxxxxxxxxx

  4. curl -s -H 'Content-Type: application/json'   -H 'Authorization: Bearer xxxxxxxxxxx'   'https://dlp.googleapis.com/v2/infoTypes'

  #OR

  5. ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
     curl -s -H 'Content-Type: application/json'   -H "Authorization: Bearer $ACCESS_TOKEN"   'https://dlp.googleapis.com/v2/infoTypes'
   #you need Double Quotes and not Single Quotes for -H flag

3.1.2.2 Single Quotes

Enclosing characters in single quotes (') preserves the literal value of each character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash.

3.1.2.3 Double Quotes

Enclosing characters in double quotes (") preserves the literal value of all characters within the quotes, with the exception of $, , \, and, when history expansion is enabled, !. The characters $ and retain their special meaning within double quotes (see Shell Expansions). The backslash retains its special meaning only when followed by one of the following characters: $, `, ", \, or newline. Within double quotes, backslashes that are followed by one of these characters are removed. Backslashes preceding characters without a special meaning are left unmodified. A double quote may be quoted within double quotes by preceding it with a backslash. If enabled, history expansion will be performed unless an ! appearing in double quotes is escaped using a backslash. The backslash preceding the ! is not removed.

The special parameters * and @ have special meaning when in double quotes (see Shell Parameter Expansion).

输出：


{
  "infoTypes": [
    {
      "name": "AMERICAN_BANKERS_CUSIP_ID",
      "displayName": "American Bankers CUSIP identifier",
      "supportedBy": [
        "INSPECT"
      ],
      "description": "An American Bankers' Committee on Uniform Security Identification Procedures (CUSIP) number is a 9-character alphanumeric code that identifies a North American financial security."
    },
    {
      "name": "AUSTRALIA_DRIVERS_LICENSE_NUMBER",
      "displayName": "Australia driver's license number",
      "supportedBy": [
        "INSPECT"
      ],
      "description": "An Australian driver's license number."
    },
    {
      "name": "AUSTRALIA_MEDICARE_NUMBER",
      "displayName": "Australia medicare number",
      "supportedBy": [
        "INSPECT"

..................

Answer 3

需要注意的一点是，DLP 有点不同。每个请求实际上涉及两个服务帐户。你用来调用API的那个，但是服务本身使用dlp服务账号转过来调用其他服务读取存储。

当您启用 dlp 时，您会看到该帐户已自动创建。看起来像这样... service-[PROJECT_NUMBER]@dlp-api.iam.gserviceaccount.com 并在此处讨论 https://cloud.google.com/dlp/docs/iam-permissions#service_account

我见过的一个常见设置是安全团队将在单个项目中启用 DLP 以扫描他们的整个组织。他们将通过授予项目中的服务-[PROJECT_NUMBER]@dlp-api.iam.gserviceaccount.com 他们运行访问组织中所有其他项目的权限来做到这一点。这将让它在整个公司读取 GCS/BQ。

Answer 4

如果您发现此 post 是因为您想通过 Java 客户端仅使用 API 密钥调用 DLP API，请参阅此要点我是如何让它工作的（在 Scala 中）。基本上我自己必须通过 header 并明确表示没有其他身份验证机制

https://gist.github.com/kylejmcintyre/b7ff56cf08addb87ebcf53defa18461f

在我的例子中，我将所有数据传递给 API 检查，因此除了调用检查 API.[=11= 之外，我不关心任何其他身份验证]

GCP 数据丢失防护 API 身份验证：是否需要使用服务帐户？

GCP Data Loss Prevention API Authentication: Does it require the use of a service account?

api

automation

service-accounts

google-cloud-platform

google-cloud-dlp