AttributeError: 'NoneType' object has no attribute 'tokenize'

Question

我正在尝试通过转换器使用 XLNET。但是我不断收到问题“AttributeError：'NoneType' 对象没有属性 'tokenize'”。我不确定如何进行。如果有人能指出我正确的方向，我将不胜感激。

tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased', do_lower_case=True)

print(' Original: ', X_train[1])

# Print the tweet split into tokens.
print('Tokenized: ', tokenizer.tokenize(X_train[1]))

# Print the tweet mapped to token ids.
print('Token IDs: ', tokenizer.convert_tokens_to_ids(tokenizer.tokenize(X_train[1])))




Original:  hey angel duh sexy really thanks haha
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-67-2b1b432b3e15> in <module>()
      2 
      3 # Print the tweet split into tokens.
----> 4 print('Tokenized: ', tokenizer.tokenize(X_train[2]))
      5 
      6 # Print the tweet mapped to token ids.

AttributeError: 'NoneType' object has no attribute 'tokenize'

Answer 1

我认为：

from transformers import XLNetTokenizerFast
tokenizer = XLNetTokenizerFast.from_pretrained('xlnet-base-cased', do_lower_case=True)

有效吗？在这种情况下，您只是缺少 sentencepiece 包：

pip install sentencepiece

Answer 2

如果安装了 SenencePience 但仍然有错误

确实如此，@cronoik 's answer is the correct one. No doubt regarding it. But if you have installed the SenencePience打包还是有错误，只是restart the runtime environment and it will work.

AttributeError: 'NoneType' object has no attribute 'tokenize'

AttributeError: 'NoneType' object has no attribute 'tokenize'

python

huggingface-transformers

如果安装了 SenencePience 但仍然有错误