pytorch 钩子有问题吗？激活图始终为正
Problem with pytorch hooks? Activation maps allways positiv
我在看pytorch中vgg19的激活图。
我发现即使在我应用 ReLU 之前，地图的所有值都是正的。
这对我来说似乎很奇怪...如果这是正确的（可能是我没有正确使用 register_forward_hook 方法？）为什么还要应用 ReLu？
这是我生成此代码的代码：
import torch
import torchvision
import torchvision.models as models
import torchvision.transforms as transforms
from torchsummary import summary
import os, glob
import matplotlib.pyplot as plt
import numpy as np
# settings:
batch_size = 4
# load the model
model = models.vgg19(pretrained=True)
summary(model.cuda(), (3, 32, 32))
model.cpu()
# how to preprocess??? See here:
# https://discuss.pytorch.org/t/how-to-preprocess-input-for-pre-trained-networks/683/2
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225])
transform = transforms.Compose(
[transforms.ToTensor(),
normalize])
# build data loader
trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,
shuffle=True, num_workers=2)
# show one image
dataiter = iter(trainloader)
images, labels = dataiter.next()
# set a hook
activation = {}
def get_activation(name):
def hook(model, input, output):
activation[name] = output.detach()
return hook
# hook at the first conv layer
hook = model.features[0].register_forward_hook(get_activation("firstConv"))
model(images)
hook.remove()
# show results:
flatted_feat_maps = activation["firstConv"].detach().numpy().flatten()
print("All positiv??? --> ",np.all(flatted_feat_maps >= 0))
plt.hist(flatted_feat_maps)
plt.show()
----------------------------------------------------------------
Layer (type) Output Shape Param #
================================================================
Conv2d-1 [-1, 64, 32, 32] 1,792
ReLU-2 [-1, 64, 32, 32] 0
Conv2d-3 [-1, 64, 32, 32] 36,928
ReLU-4 [-1, 64, 32, 32] 0
MaxPool2d-5 [-1, 64, 16, 16] 0
Conv2d-6 [-1, 128, 16, 16] 73,856
ReLU-7 [-1, 128, 16, 16] 0
Conv2d-8 [-1, 128, 16, 16] 147,584
ReLU-9 [-1, 128, 16, 16] 0
MaxPool2d-10 [-1, 128, 8, 8] 0
Conv2d-11 [-1, 256, 8, 8] 295,168
ReLU-12 [-1, 256, 8, 8] 0
Conv2d-13 [-1, 256, 8, 8] 590,080
ReLU-14 [-1, 256, 8, 8] 0
Conv2d-15 [-1, 256, 8, 8] 590,080
ReLU-16 [-1, 256, 8, 8] 0
Conv2d-17 [-1, 256, 8, 8] 590,080
ReLU-18 [-1, 256, 8, 8] 0
MaxPool2d-19 [-1, 256, 4, 4] 0
Conv2d-20 [-1, 512, 4, 4] 1,180,160
ReLU-21 [-1, 512, 4, 4] 0
Conv2d-22 [-1, 512, 4, 4] 2,359,808
ReLU-23 [-1, 512, 4, 4] 0
Conv2d-24 [-1, 512, 4, 4] 2,359,808
ReLU-25 [-1, 512, 4, 4] 0
Conv2d-26 [-1, 512, 4, 4] 2,359,808
ReLU-27 [-1, 512, 4, 4] 0
MaxPool2d-28 [-1, 512, 2, 2] 0
Conv2d-29 [-1, 512, 2, 2] 2,359,808
ReLU-30 [-1, 512, 2, 2] 0
Conv2d-31 [-1, 512, 2, 2] 2,359,808
ReLU-32 [-1, 512, 2, 2] 0
Conv2d-33 [-1, 512, 2, 2] 2,359,808
ReLU-34 [-1, 512, 2, 2] 0
Conv2d-35 [-1, 512, 2, 2] 2,359,808
ReLU-36 [-1, 512, 2, 2] 0
MaxPool2d-37 [-1, 512, 1, 1] 0
AdaptiveAvgPool2d-38 [-1, 512, 7, 7] 0
Linear-39 [-1, 4096] 102,764,544
ReLU-40 [-1, 4096] 0
Dropout-41 [-1, 4096] 0
Linear-42 [-1, 4096] 16,781,312
ReLU-43 [-1, 4096] 0
Dropout-44 [-1, 4096] 0
Linear-45 [-1, 1000] 4,097,000
================================================================
Total params: 143,667,240
Trainable params: 143,667,240
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.01
Forward/backward pass size (MB): 5.25
Params size (MB): 548.05
Estimated Total Size (MB): 553.31
----------------------------------------------------------------
难道是我没有正确使用register_forward_hook？
你应该clone输出
def get_activation(name):
def hook(model, input, output):
activation[name] = output.detach().clone() #
return hook
请注意，Tensor.detach 仅将张量从图中分离，但两个张量仍将共享相同的底层存储。
Returned Tensor shares the same storage with the original one. In-place modifications on either of them will be seen, and may trigger errors in correctness checks. IMPORTANT NOTE: Previously, in-place size / stride / storage changes (such as resize_ / resize_as_ / set_ / transpose_) to the returned tensor also update the original tensor. Now, these in-place changes will not update the original tensor anymore, and will instead trigger an error. For sparse tensors: In-place indices / values changes (such as zero_ / copy_ / add_) to the returned tensor will not update the original tensor anymore, and will instead trigger an error.
我在看pytorch中vgg19的激活图。 我发现即使在我应用 ReLU 之前，地图的所有值都是正的。
这对我来说似乎很奇怪...如果这是正确的（可能是我没有正确使用 register_forward_hook 方法？）为什么还要应用 ReLu？
这是我生成此代码的代码：
import torch
import torchvision
import torchvision.models as models
import torchvision.transforms as transforms
from torchsummary import summary
import os, glob
import matplotlib.pyplot as plt
import numpy as np
# settings:
batch_size = 4
# load the model
model = models.vgg19(pretrained=True)
summary(model.cuda(), (3, 32, 32))
model.cpu()
# how to preprocess??? See here:
# https://discuss.pytorch.org/t/how-to-preprocess-input-for-pre-trained-networks/683/2
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225])
transform = transforms.Compose(
[transforms.ToTensor(),
normalize])
# build data loader
trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,
shuffle=True, num_workers=2)
# show one image
dataiter = iter(trainloader)
images, labels = dataiter.next()
# set a hook
activation = {}
def get_activation(name):
def hook(model, input, output):
activation[name] = output.detach()
return hook
# hook at the first conv layer
hook = model.features[0].register_forward_hook(get_activation("firstConv"))
model(images)
hook.remove()
# show results:
flatted_feat_maps = activation["firstConv"].detach().numpy().flatten()
print("All positiv??? --> ",np.all(flatted_feat_maps >= 0))
plt.hist(flatted_feat_maps)
plt.show()
----------------------------------------------------------------
Layer (type) Output Shape Param #
================================================================
Conv2d-1 [-1, 64, 32, 32] 1,792
ReLU-2 [-1, 64, 32, 32] 0
Conv2d-3 [-1, 64, 32, 32] 36,928
ReLU-4 [-1, 64, 32, 32] 0
MaxPool2d-5 [-1, 64, 16, 16] 0
Conv2d-6 [-1, 128, 16, 16] 73,856
ReLU-7 [-1, 128, 16, 16] 0
Conv2d-8 [-1, 128, 16, 16] 147,584
ReLU-9 [-1, 128, 16, 16] 0
MaxPool2d-10 [-1, 128, 8, 8] 0
Conv2d-11 [-1, 256, 8, 8] 295,168
ReLU-12 [-1, 256, 8, 8] 0
Conv2d-13 [-1, 256, 8, 8] 590,080
ReLU-14 [-1, 256, 8, 8] 0
Conv2d-15 [-1, 256, 8, 8] 590,080
ReLU-16 [-1, 256, 8, 8] 0
Conv2d-17 [-1, 256, 8, 8] 590,080
ReLU-18 [-1, 256, 8, 8] 0
MaxPool2d-19 [-1, 256, 4, 4] 0
Conv2d-20 [-1, 512, 4, 4] 1,180,160
ReLU-21 [-1, 512, 4, 4] 0
Conv2d-22 [-1, 512, 4, 4] 2,359,808
ReLU-23 [-1, 512, 4, 4] 0
Conv2d-24 [-1, 512, 4, 4] 2,359,808
ReLU-25 [-1, 512, 4, 4] 0
Conv2d-26 [-1, 512, 4, 4] 2,359,808
ReLU-27 [-1, 512, 4, 4] 0
MaxPool2d-28 [-1, 512, 2, 2] 0
Conv2d-29 [-1, 512, 2, 2] 2,359,808
ReLU-30 [-1, 512, 2, 2] 0
Conv2d-31 [-1, 512, 2, 2] 2,359,808
ReLU-32 [-1, 512, 2, 2] 0
Conv2d-33 [-1, 512, 2, 2] 2,359,808
ReLU-34 [-1, 512, 2, 2] 0
Conv2d-35 [-1, 512, 2, 2] 2,359,808
ReLU-36 [-1, 512, 2, 2] 0
MaxPool2d-37 [-1, 512, 1, 1] 0
AdaptiveAvgPool2d-38 [-1, 512, 7, 7] 0
Linear-39 [-1, 4096] 102,764,544
ReLU-40 [-1, 4096] 0
Dropout-41 [-1, 4096] 0
Linear-42 [-1, 4096] 16,781,312
ReLU-43 [-1, 4096] 0
Dropout-44 [-1, 4096] 0
Linear-45 [-1, 1000] 4,097,000
================================================================
Total params: 143,667,240
Trainable params: 143,667,240
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.01
Forward/backward pass size (MB): 5.25
Params size (MB): 548.05
Estimated Total Size (MB): 553.31
----------------------------------------------------------------
难道是我没有正确使用register_forward_hook？
你应该clone输出
def get_activation(name):
def hook(model, input, output):
activation[name] = output.detach().clone() #
return hook
请注意，Tensor.detach 仅将张量从图中分离，但两个张量仍将共享相同的底层存储。
Returned Tensor shares the same storage with the original one. In-place modifications on either of them will be seen, and may trigger errors in correctness checks. IMPORTANT NOTE: Previously, in-place size / stride / storage changes (such as resize_ / resize_as_ / set_ / transpose_) to the returned tensor also update the original tensor. Now, these in-place changes will not update the original tensor anymore, and will instead trigger an error. For sparse tensors: In-place indices / values changes (such as zero_ / copy_ / add_) to the returned tensor will not update the original tensor anymore, and will instead trigger an error.