Tensorflow：GPU 加速仅在第一个运行之后发生

Question

我已经在我的机器上安装了 CUDA 和 CUDNN (Ubuntu 16.04) 以及 tensorflow-gpu。

使用的版本：CUDA 10.0、CUDNN 7.6、Python3.6、Tensorflow 1.14

这是 nvidia-smi 的输出，显示了显卡配置。

| NVIDIA-SMI 410.78       Driver Version: 410.78       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 960M    On   | 00000000:02:00.0 Off |                  N/A |
| N/A   44C    P8    N/A /  N/A |    675MiB /  4046MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1502      G   /usr/lib/xorg/Xorg                           363MiB |
|    0      3281      G   compiz                                        96MiB |
|    0      4375      G   ...uest-channel-token=14359313252217012722    69MiB |
|    0      5157      C   ...felipe/proj/venv/bin/python3.6            141MiB |
+-----------------------------------------------------------------------------+

这是 device_lib.list_local_devices() 的输出（显示它可以看到哪些设备的 tensorflow 辅助方法），表明我的 GPU 对 tensorflow 可见：

[name: "/device:CPU:0"
  device_type: "CPU"
  memory_limit: 268435456
  locality {
  }
  incarnation: 5096693727819965430, 
name: "/device:XLA_GPU:0"
  device_type: "XLA_GPU"
  memory_limit: 17179869184
  locality {
  }
  incarnation: 13415556283266501672
  physical_device_desc: "device: XLA_GPU device", 
name: "/device:XLA_CPU:0"
  device_type: "XLA_CPU"
  memory_limit: 17179869184
  locality {
  }
  incarnation: 14339781620792127180
  physical_device_desc: "device: XLA_CPU device", 
name: "/device:GPU:0"
  device_type: "GPU"
  memory_limit: 3464953856
  locality {
    bus_id: 1
    links {
    }
  }
  incarnation: 13743207545082600644
  physical_device_desc: "device: 0, name: GeForce GTX 960M, pci bus id: 0000:02:00.0, compute capability: 5.0"
]

现在实际使用 GPU 进行计算。我使用一小段代码在 CPU 和 GPU 上 运行一些虚拟矩阵乘法 ，以比较性能：

shapes = [(50, 50), (100, 100), (500, 500), (1000, 1000), (10000,10000), (15000,15000)]

devices = ['/device:CPU:0', '/device:XLA_GPU:0']

for device in devices:
    for shape in shapes:
        with tf.device(device):
            random_matrix = tf.random_uniform(shape=shape, minval=0, maxval=1)
            dot_operation = tf.matmul(random_matrix, tf.transpose(random_matrix))
            sum_operation = tf.reduce_sum(dot_operation)

        # Time the actual runtime of the operations
        start_time = datetime.now()
        with tf.Session(config=tf.ConfigProto(log_device_placement=True)) as session:
            result = session.run(sum_operation)
        elapsed_time = datetime.now() - start_time

        # PRINT ELAPSED TIME, SHAPE AND DEVICE USED

惊喜来了。我第一次运行包含此代码块的单元格（我在 jupyter 笔记本上）时，GPU 计算比 CPU 花费的时间长得多：

# output of first run: CPU is faster
----------------------------------------
Input shape: (50, 50) using Device: /device:CPU:0 took: 0.01
Input shape: (100, 100) using Device: /device:CPU:0 took: 0.01
Input shape: (500, 500) using Device: /device:CPU:0 took: 0.01
Input shape: (1000, 1000) using Device: /device:CPU:0 took: 0.02
Input shape: (10000, 10000) using Device: /device:CPU:0 took: 6.22
Input shape: (15000, 15000) using Device: /device:CPU:0 took: 21.23
----------------------------------------
Input shape: (50, 50) using Device: /device:XLA_GPU:0 took: 2.82
Input shape: (100, 100) using Device: /device:XLA_GPU:0 took: 0.17
Input shape: (500, 500) using Device: /device:XLA_GPU:0 took: 0.18
Input shape: (1000, 1000) using Device: /device:XLA_GPU:0 took: 0.20
Input shape: (10000, 10000) using Device: /device:XLA_GPU:0 took: 28.36
Input shape: (15000, 15000) using Device: /device:XLA_GPU:0 took: 93.73
----------------------------------------

惊喜#2：当我重新运行包含虚拟矩阵乘法代码的单元格时，GPU 版本要快得多（正如预期的那样）：

# output of reruns: GPU is faster
----------------------------------------
Input shape: (50, 50) using Device: /device:CPU:0 took: 0.02
Input shape: (100, 100) using Device: /device:CPU:0 took: 0.02
Input shape: (500, 500) using Device: /device:CPU:0 took: 0.02
Input shape: (1000, 1000) using Device: /device:CPU:0 took: 0.04
Input shape: (10000, 10000) using Device: /device:CPU:0 took: 6.78
Input shape: (15000, 15000) using Device: /device:CPU:0 took: 24.65
----------------------------------------
Input shape: (50, 50) using Device: /device:XLA_GPU:0 took: 0.14
Input shape: (100, 100) using Device: /device:XLA_GPU:0 took: 0.12
Input shape: (500, 500) using Device: /device:XLA_GPU:0 took: 0.13
Input shape: (1000, 1000) using Device: /device:XLA_GPU:0 took: 0.14
Input shape: (10000, 10000) using Device: /device:XLA_GPU:0 took: 1.64
Input shape: (15000, 15000) using Device: /device:XLA_GPU:0 took: 5.29
----------------------------------------

所以我的问题是：为什么只有我运行代码一次后GPU加速才真正发生？

我可以看到 GPU 设置正确（否则根本不会发生加速）。是由于某种初始开销吗？ GPU 需要预热才能真正使用它们吗？

P.S.: 在两个运行上（即 GPU 较慢的那个和下一个 GPU 更快的） , 我可以看到 GPU 使用率是 100%，所以它肯定被使用了。

P.S.: 只有在第一个运行 GPU 似乎没有被拾取。如果我然后运行它两次、三次或多次，则第一次之后的所有运行都成功（即 GPU 计算速度更快）。

Answer 1

让我研究了 XLA 的东西，这帮助我找到了解决方案。

事实证明，GPU 以两种方式映射到 Tensorflow 设备：作为 XLA 设备和作为普通 GPU。

这就是为什么有两台设备，一台名为 "/device:XLA_GPU:0"，另一台名为 "/device:GPU:0"。

我需要做的就是激活"/device:GPU:0"。现在 GPU 立即被 Tensorflow 接收。

Tensorflow：GPU 加速仅在第一个 运行 之后发生

Tensorflow: GPU Acceleration only happens after first run

python

gpu

nvidia

tensorflow

Tensorflow：GPU 加速仅在第一个运行之后发生