Geeting ModuleNotFoundError: No module named 'pytz' in composer DAG run
Geeting ModuleNotFoundError: No module named 'pytz' in composer DAG run
我在 composer DAG 中执行 Bigquery 操作并收到以下错误：
Event with job id 36cc0fe962103bf2bb7a9c Failed
[2021-08-19 15:13:07,807] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,806] {pod_launcher.py:125} INFO - b' from google.cloud.bigquery.routine import RoutineReference\n'
[2021-08-19 15:13:07,807] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,807] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery/routine/__init__.py", line 18, in <module>\n'
[2021-08-19 15:13:07,810] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,808] {pod_launcher.py:125} INFO - b' from google.cloud.bigquery.enums import DeterminismLevel\n'
[2021-08-19 15:13:07,810] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,808] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery/enums.py", line 21, in <module>\n'
[2021-08-19 15:13:07,811] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,808] {pod_launcher.py:125} INFO - b' from google.cloud.bigquery.query import ScalarQueryParameterType\n'
[2021-08-19 15:13:07,812] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,809] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery/query.py", line 23, in <module>\n'
[2021-08-19 15:13:07,812] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,809] {pod_launcher.py:125} INFO - b' from google.cloud.bigquery.table import _parse_schema_resource\n'
[2021-08-19 15:13:07,812] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,809] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery/table.py", line 23, in <module>\n'
[2021-08-19 15:13:07,813] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,809] {pod_launcher.py:125} INFO - b' import pytz\n'
[2021-08-19 15:13:07,813] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,809] {pod_launcher.py:125} INFO - b"ModuleNotFoundError: No module named 'pytz'\n"
我没有在气流或 BQ 方法中使用 pytz。我试图在我的作曲家环境中添加 pytz，但它没有 work.Please 建议。
此错误是由于损坏 https://github.com/googleapis/python-bigquery/issues/885 and a fix for this is released on Bigquery version 2.24.1。
您可以使用命令 gcloud composer environments update 更新您的 bigquery 依赖项。有关详细信息，请参阅 Installing a Python dependency from PyPI。
出于测试目的，环境有 Composer Version=1.16.14 和 Airflow Version=1.10.15。使用 Cloud Composer pre installed packages, I only copied the Google dependencies and updated the packages below. This is because they are dependencies of Bigquery as per Bigquery constraints.
中列出的预安装包
- google-cloud-bigquery==2.24.1
- google-api-核心==1.29.0
- grpcio==1.38.1
完整 requirements.txt:
google-ads==4.0.0
google-api-core==1.29.0
google-api-python-client==1.12.8
google-apitools==0.5.31
google-auth==1.28.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.4.3
google-cloud-automl==2.2.0
google-cloud-bigquery==2.24.1
google-cloud-bigquery-datatransfer==3.1.0
google-cloud-bigquery-storage==2.1.0
google-cloud-bigtable==1.7.0
google-cloud-build==2.0.0
google-cloud-container==1.0.1
google-cloud-core==1.6.0
google-cloud-datacatalog==3.1.0
google-cloud-dataproc==2.3.0
google-cloud-datastore==1.15.3
google-cloud-dlp==1.0.0
google-cloud-kms==2.2.0
google-cloud-language==1.3.0
google-cloud-logging==2.2.0
google-cloud-memcache==0.3.0
google-cloud-monitoring==2.0.0
google-cloud-os-login==2.1.0
google-cloud-pubsub==2.3.0
google-cloud-pubsublite==0.3.0
google-cloud-redis==2.1.0
google-cloud-secret-manager==1.0.0
google-cloud-spanner==1.19.1
google-cloud-speech==1.3.2
google-cloud-storage==1.36.2
google-cloud-tasks==2.2.0
google-cloud-texttospeech==1.0.1
google-cloud-translate==1.7.0
google-cloud-videointelligence==1.16.1
google-cloud-vision==1.0.0
google-cloud-workflows==0.2.0
google-crc32c==1.1.2
google-pasta==0.2.0
google-resumable-media==1.2.0
googleapis-common-protos==1.53.0
grpc-google-iam-v1==0.12.3
grpcio==1.38.1
grpcio-gcp==0.2.2
命令：
gcloud composer environments update your-composer-environment-name --update-pypi-packages-from-file requirements.txt --location your-composer-location
我在 composer DAG 中执行 Bigquery 操作并收到以下错误：
Event with job id 36cc0fe962103bf2bb7a9c Failed
[2021-08-19 15:13:07,807] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,806] {pod_launcher.py:125} INFO - b' from google.cloud.bigquery.routine import RoutineReference\n'
[2021-08-19 15:13:07,807] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,807] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery/routine/__init__.py", line 18, in <module>\n'
[2021-08-19 15:13:07,810] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,808] {pod_launcher.py:125} INFO - b' from google.cloud.bigquery.enums import DeterminismLevel\n'
[2021-08-19 15:13:07,810] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,808] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery/enums.py", line 21, in <module>\n'
[2021-08-19 15:13:07,811] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,808] {pod_launcher.py:125} INFO - b' from google.cloud.bigquery.query import ScalarQueryParameterType\n'
[2021-08-19 15:13:07,812] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,809] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery/query.py", line 23, in <module>\n'
[2021-08-19 15:13:07,812] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,809] {pod_launcher.py:125} INFO - b' from google.cloud.bigquery.table import _parse_schema_resource\n'
[2021-08-19 15:13:07,812] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,809] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery/table.py", line 23, in <module>\n'
[2021-08-19 15:13:07,813] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,809] {pod_launcher.py:125} INFO - b' import pytz\n'
[2021-08-19 15:13:07,813] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,809] {pod_launcher.py:125} INFO - b"ModuleNotFoundError: No module named 'pytz'\n"
我没有在气流或 BQ 方法中使用 pytz。我试图在我的作曲家环境中添加 pytz，但它没有 work.Please 建议。
此错误是由于损坏 https://github.com/googleapis/python-bigquery/issues/885 and a fix for this is released on Bigquery version 2.24.1。
您可以使用命令 gcloud composer environments update 更新您的 bigquery 依赖项。有关详细信息，请参阅 Installing a Python dependency from PyPI。
出于测试目的，环境有 Composer Version=1.16.14 和 Airflow Version=1.10.15。使用 Cloud Composer pre installed packages, I only copied the Google dependencies and updated the packages below. This is because they are dependencies of Bigquery as per Bigquery constraints.
中列出的预安装包- google-cloud-bigquery==2.24.1
- google-api-核心==1.29.0
- grpcio==1.38.1
完整 requirements.txt:
google-ads==4.0.0
google-api-core==1.29.0
google-api-python-client==1.12.8
google-apitools==0.5.31
google-auth==1.28.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.4.3
google-cloud-automl==2.2.0
google-cloud-bigquery==2.24.1
google-cloud-bigquery-datatransfer==3.1.0
google-cloud-bigquery-storage==2.1.0
google-cloud-bigtable==1.7.0
google-cloud-build==2.0.0
google-cloud-container==1.0.1
google-cloud-core==1.6.0
google-cloud-datacatalog==3.1.0
google-cloud-dataproc==2.3.0
google-cloud-datastore==1.15.3
google-cloud-dlp==1.0.0
google-cloud-kms==2.2.0
google-cloud-language==1.3.0
google-cloud-logging==2.2.0
google-cloud-memcache==0.3.0
google-cloud-monitoring==2.0.0
google-cloud-os-login==2.1.0
google-cloud-pubsub==2.3.0
google-cloud-pubsublite==0.3.0
google-cloud-redis==2.1.0
google-cloud-secret-manager==1.0.0
google-cloud-spanner==1.19.1
google-cloud-speech==1.3.2
google-cloud-storage==1.36.2
google-cloud-tasks==2.2.0
google-cloud-texttospeech==1.0.1
google-cloud-translate==1.7.0
google-cloud-videointelligence==1.16.1
google-cloud-vision==1.0.0
google-cloud-workflows==0.2.0
google-crc32c==1.1.2
google-pasta==0.2.0
google-resumable-media==1.2.0
googleapis-common-protos==1.53.0
grpc-google-iam-v1==0.12.3
grpcio==1.38.1
grpcio-gcp==0.2.2
命令：
gcloud composer environments update your-composer-environment-name --update-pypi-packages-from-file requirements.txt --location your-composer-location