如何在资源 update/created 之后 运行 AWS Glue Crawler？

Question

我在我的 Terraform 中定义了一个资源来创建我不想安排的 Glue Crawler。但我希望它在创建和更新后成为 运行。我在文档中找不到任何关于如何触发它的内容。

resource "aws_glue_crawler" "my_crawler" {
  database_name = "my_db"
  name          = "my_crawler"
  role          = "arn:aws:iam::111111111111:role/service-role/someRole"

  s3_target {
    path = "s3://my_bucket/key/prefix"
  }

}

Answer 1

如果 Terraform 是 ci/cd 的一部分，您可以在 运行 爬虫部署后将 shell/python 脚本写入

Answer 2

您可以使用 local-exec provisioner 来使用 AWS CLI 在 Glue 爬虫创建后触发它：

resource "aws_glue_crawler" "my_crawler" {
  database_name = "my_db"
  name          = "my_crawler"
  role          = "arn:aws:iam::111111111111:role/service-role/someRole"

  s3_target {
    path = "s3://my_bucket/key/prefix"
  }

  provisioner "local-exec" {
    command = "aws glue start-crawler --name ${self.name}"
  }
}

这只会在创建爬虫时触发，而不会在任何其他时候触发，例如如果您更改了 s3_target.path 或其他任何内容。

如果您希望在更改 s3_target.path 时能够触发它，您需要使用 null_resource with a trigger:

resource "aws_glue_crawler" "my_crawler" {
  database_name = "my_db"
  name          = "my_crawler"
  role          = "arn:aws:iam::111111111111:role/service-role/someRole"

  s3_target {
    path = "s3://my_bucket/key/prefix"
  }
}

resource "null_resource" "run_crawler" {
  # Changes to the crawler's S3 path requires re-running
  triggers = {
    s3_path = "${aws_glue_crawler.my_crawler.s3_target.0.path}"
  }

  provisioner "local-exec" {
    command = "aws glue start-crawler --name ${aws_glue_crawler.my_crawler.name}"
  }
}