我正在尝试使用 pytorch 的 XLNet 实现并在加载 XLNet 时得到 'Trying to create tensor with negative dimension -1: [-1, 768]'

Question

大约两个月前，我开始在 Google Colab 上为一个中期项目做这件事，一切都很完美。现在我正在为最终项目修改它并不断收到错误 'RuntimeError: Trying to create tensor with negative dimension -1: [-1, 768]'。 pytorch最近好像推了1.5的新版本，于是降级到1.4版本，还是报同样的错误。与 1.3 相同，而且我知道自去年发布以来我没有使用任何更低的版本。我用我的期中代码检查了它，仍然有同样的错误，所以我不知道发生了什么。这是与下载和使用模型相关的代码块。

    train_inputs, validation_inputs, train_labels, validation_labels = train_test_split(inputIds, 
                                                                                        labels, 
                                                                                        random_state=2020, 
                                                                                        test_size=0.2)
    train_masks, validation_masks, _, _ = train_test_split(attention_masks, inputIds, random_state=2020, 
                                                           test_size=0.2)
    # Turn data into torch tensors
    train_inputs = torch.tensor(train_inputs)
    validation_inputs = torch.tensor(validation_inputs)
    train_labels = torch.tensor(train_labels)
    validation_labels = torch.tensor(validation_labels)
    train_masks = torch.tensor(train_masks)
    validation_masks = torch.tensor(validation_masks)

    # Create Iterators of the datasets
    train_data = TensorDataset(train_inputs, train_masks, train_labels)
    train_sampler = RandomSampler(train_data)
    train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)
    validation_data = TensorDataset(validation_inputs, validation_masks, validation_labels)
    validation_sampler = SequentialSampler(validation_data)
    validation_dataloader = DataLoader(validation_data, sampler=validation_sampler, batch_size=batch_size)

    model = XLNetForSequenceClassification.from_pretrained('xlnet-base-cased', num_labels=2)
    # Loads model into GPU memory
    model.cuda()

    param_optimizer = list(model.named_parameters())
    no_decay = ['bias','gamma','beta']
    optimizer_grouped_parameters = [
        {'params':[p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
         'weight_decay_rate':0.01},
        {'params':[p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
         'weight_decay_rate':0.0}
    ]
    optimizer = AdamW(optimizer_grouped_parameters, lr=2e-5)

错误发生在 model = XLNetForSequenceClassification.from_pretrained('xlnet-base-cased', num_labels=2) 行。我正在使用的软件包：

from pandas import to_datetime
import torch
from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler
from keras.preprocessing.sequence import pad_sequences
from sklearn.model_selection import train_test_split
# MUST INSTALL PYTORCH-TRANSFORMERS
from pytorch_transformers import XLNetTokenizer, XLNetForSequenceClassification, AdamW
from tqdm import trange
from numpy import argmax, sum 
import nltk
nltk.download('punkt')

感谢所有试图提供帮助的人。

Answer 1

您可以尝试使用变形金刚代替 pytorch_transformers。

! pip install transformers（Google Colab）

在终端中，

pip install transformers

import torch

from transformers import XLNetForSequenceClassification

model = XLNetForSequenceClassification.from_pretrained('xlnet-base-cased', num_labels=2)

model.cuda()

param_optimizer = list(model.named_parameters())
no_decay = ['bias','gamma','beta']
optimizer_grouped_parameters = [
    {'params':[p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
      'weight_decay_rate':0.01},
    {'params':[p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
      'weight_decay_rate':0.0}
]

这是 google colab 中没有任何错误的代码：https://colab.research.google.com/drive/1A8edGYyFuE7d1Z-ZJusz5zU_7yCevF2L

我正在尝试使用 pytorch 的 XLNet 实现并在加载 XLNet 时得到 'Trying to create tensor with negative dimension -1: [-1, 768]'

I am trying to use pytorch's implementation of XLNet and got 'Trying to create tensor with negative dimension -1: [-1, 768]' when loading XLNet

transformer

pytorch

tensor