从拥抱面权重构建张量流模型的问题

Question

我需要使用来自 Huggingface with Tensorflow 的预训练 BERT 模型 ('dbmdz/bert-base-italian-xxl-cased')（位于 this link）。

在网站上阅读这篇文章后，

Currently only PyTorch-Transformers compatible weights are available. If you need access to TensorFlow checkpoints, please raise an issue!

我提出了这个问题，并立即向我提供了 link 包含以下文件的存档下载。文件如下：

$ ls bert-base-italian-xxl-cased/
config.json                    model.ckpt.index               vocab.txt
model.ckpt.data-00000-of-00001 model.ckpt.meta

我现在正在尝试加载模型并使用它，但我尝试的一切都失败了。

我尝试遵循来自 Huggingface 讨论网站的 this 建议：

bert_folder = str(Config.MODELS_CONFIG.BERT_CHECKPOINT_DIR) # folder in which I have the files extracted from the archive
from transformers import BertConfig, TFBertModel
config = BertConfig.from_pretrained(bert_folder) # this gets loaded correctly

在这之后我尝试了几种组合来加载模型但总是失败。

例如：

model = TFBertModel.from_pretrained("../../models/pretrained/bert-base-italian-xxl-cased/model.ckpt.index", config=config)

model = TFBertModel.from_pretrained("../../models/pretrained/bert-base-italian-xxl-cased/model.ckpt.index", config=config, from_pt=True)

model = TFBertModel.from_pretrained("../../models/pretrained/bert-base-italian-xxl-cased/model.ckpt.index", config=config, from_pt=True)

model = TFBertModel.from_pretrained("../../models/pretrained/bert-base-italian-xxl-cased", config=config, local_files_only=True)

总是导致此错误：

404 Client Error: Not Found for url: https://huggingface.co/models/pretrained/bert-base-italian-xxl-cased/model.ckpt.index/resolve/main/tf_model.h5
...
...
OSError: Can't load weights for '../../models/pretrained/bert-base-italian-xxl-cased/model.ckpt.index'. Make sure that:

- '../../models/pretrained/bert-base-italian-xxl-cased/model.ckpt.index' is a correct model identifier listed on 'https://huggingface.co/models'

- or '../../models/pretrained/bert-base-italian-xxl-cased/model.ckpt.index' is the correct path to a directory containing a file named one of tf_model.h5, pytorch_model.bin.

所以我的问题是：如何从这些文件中加载这个预训练的 BERT 模型并在 tensorflow 中使用它？

Answer 1

您可以尝试使用以下代码片段在 tensorflow 中加载 dbmdz/bert-base-italian-xxl-cased。

from transformers import AutoTokenizer, TFBertModel
model_name = "dbmdz/bert-base-italian-cased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = TFBertModel.from_pretrained(model_name)

如果你想从给定的 tensorflow checkpoint 加载，你可以这样尝试：

model = TFBertModel.from_pretrained("../../models/pretrained/bert-base-italian-xxl-cased/model.ckpt.index", config=config, from_tf=True)

从拥抱面权重构建张量流模型的问题

Problem building tensorflow model from huggingface weights

python

keras

tensorflow

tensorflow2.0

huggingface-transformers