Tensorflow 停留在第一个 epoch
Tensorflow stuck on first epoch
我的执行卡在了下面这一步
我在 Python 3.7 和 windows 10 上安装了 GTX 3090。我按照此处的指示使用 conda 安装了 cuda、cudnn 和 tensorflow-gpu
conda install tensorflow-gpu=2.3 tensorflow=2.3=mkl_py37h936c3e2_0
有人能告诉我为什么卡在这一步吗？
2021-04-01 10:09:38.147949: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
Warning, `split` argument is replaced by `split_val`, please condider to change your source code.The `split` argument will be removed in future releases.
class 0, validation count: 68, train count: 159
class 5, validation count: 83, train count: 195
Total data: 2 classes for 354 files for train
Total data: 2 classes for 151 files for validation
2021-04-01 10:09:40.046207: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2021-04-01 10:09:40.763173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:44:00.0 name: NVIDIA GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.755GHz coreCount: 82 deviceMemorySize: 24.00GiB deviceMemoryBandwidth: 871.81GiB/s
2021-04-01 10:09:40.763755: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA Quadro P1000 computeCapability: 6.1
coreClock: 1.5185GHz coreCount: 4 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 89.53GiB/s
2021-04-01 10:09:40.764222: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2021-04-01 10:09:40.768538: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2021-04-01 10:09:40.772484: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2021-04-01 10:09:40.774033: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2021-04-01 10:09:40.778690: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2021-04-01 10:09:40.781972: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2021-04-01 10:09:40.793304: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2021-04-01 10:09:40.793720: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1843] Ignoring visible gpu device (device: 1, name: NVIDIA Quadro P1000, pci bus id: 0000:01:00.0, compute capability: 6.1) with core count: 4. The minimum required count is 8. You can adjust this requirement with the env var TF_MIN_GPU_MULTIPROCESSOR_COUNT.
2021-04-01 10:09:40.794339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-04-01 10:09:40.795239: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-01 10:09:40.808243: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2af71215370 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-04-01 10:09:40.808648: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-04-01 10:09:40.809155: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:44:00.0 name: NVIDIA GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.755GHz coreCount: 82 deviceMemorySize: 24.00GiB deviceMemoryBandwidth: 871.81GiB/s
2021-04-01 10:09:40.809789: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2021-04-01 10:09:40.810112: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2021-04-01 10:09:40.810416: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2021-04-01 10:09:40.810728: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2021-04-01 10:09:40.811035: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2021-04-01 10:09:40.811338: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2021-04-01 10:09:40.811624: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2021-04-01 10:09:40.811974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-04-01 10:09:42.137546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-01 10:09:42.137800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
2021-04-01 10:09:42.137935: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
2021-04-01 10:09:42.138228: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 19112 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:44:00.0, compute capability: 8.6)
2021-04-01 10:09:42.141792: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2af27c8f640 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-04-01 10:09:42.142075: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 3090, Compute Capability 8.6
Epoch 1/1000
2021-04-01 10:09:46.249137: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2021-04-01 10:09:47.132659: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
这是我的依赖关系
# Name Version Build Channel
_tflow_select 2.3.0 gpu
absl-py 0.12.0 pyhd8ed1ab_0 conda-forge
aiohttp 3.7.4 py37hcc03f2d_0 conda-forge
astor 0.8.1 pyh9f0ad1d_0 conda-forge
astunparse 1.6.3 pyhd8ed1ab_0 conda-forge
async-timeout 3.0.1 py_1000 conda-forge
attrs 20.3.0 pyhd3deb0d_0 conda-forge
blinker 1.4 py_1 conda-forge
brotlipy 0.7.0 py37hcc03f2d_1001 conda-forge
ca-certificates 2020.12.5 h5b45459_0 conda-forge
cachetools 4.2.1 pyhd8ed1ab_0 conda-forge
certifi 2020.12.5 py37h03978a9_1 conda-forge
cffi 1.14.5 py37hd8e9650_0 conda-forge
chardet 4.0.0 py37h03978a9_1 conda-forge
click 7.1.2 pyh9f0ad1d_0 conda-forge
cryptography 3.4.7 py37h20c650d_0 conda-forge
cudatoolkit 10.1.243 h3826478_8 conda-forge
cudnn 7.6.5.32 h36d860d_1 conda-forge
cycler 0.10.0 pypi_0 pypi
gast 0.3.3 py_0 conda-forge
google-auth 1.28.0 pyh44b312d_0 conda-forge
google-auth-oauthlib 0.4.1 py_2 conda-forge
google-pasta 0.2.0 pyh8c360ce_0 conda-forge
grpcio 1.36.1 py37h04d2302_0 conda-forge
h5py 2.10.0 nompi_py37h23cfb99_105 conda-forge
hdf5 1.10.6 nompi_h5268f04_1114 conda-forge
idna 2.10 pyh9f0ad1d_0 conda-forge
importlib-metadata 3.10.0 py37h03978a9_0 conda-forge
intel-openmp 2020.3 h57928b3_311 conda-forge
keras 2.4.3 py_0 conda-forge
keras-applications 1.0.8 py_1 conda-forge
keras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forge
keras-video-generators 1.0.14 pypi_0 pypi
kiwisolver 1.3.1 pypi_0 pypi
krb5 1.17.2 hbae68bd_0 conda-forge
libblas 3.9.0 8_mkl conda-forge
libcblas 3.9.0 8_mkl conda-forge
libcurl 7.76.0 hf1763fc_0 conda-forge
liblapack 3.9.0 8_mkl conda-forge
libprotobuf 3.15.6 h7755175_0 conda-forge
libssh2 1.9.0 h680486a_6 conda-forge
m2w64-gcc-libgfortran 5.3.0 6 conda-forge
m2w64-gcc-libs 5.3.0 7 conda-forge
m2w64-gcc-libs-core 5.3.0 7 conda-forge
m2w64-gmp 6.1.0 2 conda-forge
m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge
markdown 3.3.4 pyhd8ed1ab_0 conda-forge
matplotlib 3.4.1 pypi_0 pypi
mkl 2020.4 hb70f87d_311 conda-forge
msys2-conda-epoch 20160418 1 conda-forge
multidict 5.1.0 py37hcc03f2d_1 conda-forge
numpy 1.19.5 py37hd20adf4_1 conda-forge
oauthlib 3.0.1 py_0 conda-forge
opencv-python 4.5.1.48 pypi_0 pypi
openssl 1.1.1k h8ffe710_0 conda-forge
opt_einsum 3.3.0 py_0 conda-forge
pillow 8.1.2 pypi_0 pypi
pip 21.0.1 pyhd8ed1ab_0 conda-forge
protobuf 3.15.6 py37hf2a7229_0 conda-forge
pyasn1 0.4.8 py_0 conda-forge
pyasn1-modules 0.2.7 py_0 conda-forge
pycparser 2.20 pyh9f0ad1d_2 conda-forge
pyjwt 2.0.1 pyhd8ed1ab_1 conda-forge
pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge
pyparsing 2.4.7 pypi_0 pypi
pyreadline 2.1 py37h03978a9_1003 conda-forge
pysocks 1.7.1 py37h03978a9_3 conda-forge
python 3.7.10 h7840368_100_cpython conda-forge
python-dateutil 2.8.1 pypi_0 pypi
python_abi 3.7 1_cp37m conda-forge
pyyaml 5.4.1 pypi_0 pypi
requests 2.25.1 pyhd3deb0d_0 conda-forge
requests-oauthlib 1.3.0 pyh9f0ad1d_0 conda-forge
rsa 4.7.2 pyh44b312d_0 conda-forge
scipy 1.6.2 py37h924764e_0 conda-forge
setuptools 49.6.0 py37h03978a9_3 conda-forge
six 1.15.0 pyh9f0ad1d_0 conda-forge
sqlite 3.35.3 h8ffe710_0 conda-forge
tensorboard 2.4.1 pyhd8ed1ab_0 conda-forge
tensorboard-plugin-wit 1.8.0 pyh44b312d_0 conda-forge
tensorflow 2.3.0 mkl_py37h936c3e2_0
tensorflow-base 2.3.0 gpu_py37h18d21e4_0
tensorflow-estimator 2.4.0 pyh9656e83_0 conda-forge
tensorflow-gpu 2.3.0 he13fc11_0
termcolor 1.1.0 py_2 conda-forge
tk 8.6.10 h8ffe710_1 conda-forge
typing-extensions 3.7.4.3 0 conda-forge
typing_extensions 3.7.4.3 py_0 conda-forge
urllib3 1.26.4 pyhd8ed1ab_0 conda-forge
vc 14.2 hb210afc_4 conda-forge
vs2015_runtime 14.28.29325 h5e1d092_4 conda-forge
werkzeug 1.0.1 pyh9f0ad1d_0 conda-forge
wheel 0.36.2 pyhd3deb0d_0 conda-forge
win_inet_pton 1.1.0 py37h03978a9_2 conda-forge
wincertstore 0.2 py37h03978a9_1006 conda-forge
wrapt 1.12.1 py37hcc03f2d_3 conda-forge
yaml 0.2.5 he774522_0 conda-forge
yarl 1.6.3 py37hcc03f2d_1 conda-forge
zipp 3.4.1 pyhd8ed1ab_0 conda-forge
zlib 1.2.11 h62dcd97_1010 conda-forge
更新：
完全相同的代码在 2080TI 上工作，我将其与刚得到的 3090 切换并做了一个新环境并卸载了所有 2080TI 相关软件。
这是模型
Total params: 5,502,338
Trainable params: 5,500,418
Non-trainable params: 1,920
我还添加了 CUDA_CACHE_MAXSIZE=2147483648 到我的环境变量中。
RTX 2080 Ti 卡基于 Turing 兼容 CUDA version start with 10.x 的架构，其中 RTX 3090 卡基于 Ampere 兼容的架构CUDA version start with 11.x.
所以您的 gpu 卡的兼容 Tensorflow 版本是 2.4.0。更多详情可以参考here.
我的执行卡在了下面这一步 我在 Python 3.7 和 windows 10 上安装了 GTX 3090。我按照此处的指示使用 conda 安装了 cuda、cudnn 和 tensorflow-gpu
conda install tensorflow-gpu=2.3 tensorflow=2.3=mkl_py37h936c3e2_0
有人能告诉我为什么卡在这一步吗？
2021-04-01 10:09:38.147949: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
Warning, `split` argument is replaced by `split_val`, please condider to change your source code.The `split` argument will be removed in future releases.
class 0, validation count: 68, train count: 159
class 5, validation count: 83, train count: 195
Total data: 2 classes for 354 files for train
Total data: 2 classes for 151 files for validation
2021-04-01 10:09:40.046207: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2021-04-01 10:09:40.763173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:44:00.0 name: NVIDIA GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.755GHz coreCount: 82 deviceMemorySize: 24.00GiB deviceMemoryBandwidth: 871.81GiB/s
2021-04-01 10:09:40.763755: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA Quadro P1000 computeCapability: 6.1
coreClock: 1.5185GHz coreCount: 4 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 89.53GiB/s
2021-04-01 10:09:40.764222: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2021-04-01 10:09:40.768538: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2021-04-01 10:09:40.772484: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2021-04-01 10:09:40.774033: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2021-04-01 10:09:40.778690: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2021-04-01 10:09:40.781972: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2021-04-01 10:09:40.793304: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2021-04-01 10:09:40.793720: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1843] Ignoring visible gpu device (device: 1, name: NVIDIA Quadro P1000, pci bus id: 0000:01:00.0, compute capability: 6.1) with core count: 4. The minimum required count is 8. You can adjust this requirement with the env var TF_MIN_GPU_MULTIPROCESSOR_COUNT.
2021-04-01 10:09:40.794339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-04-01 10:09:40.795239: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-01 10:09:40.808243: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2af71215370 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-04-01 10:09:40.808648: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-04-01 10:09:40.809155: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:44:00.0 name: NVIDIA GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.755GHz coreCount: 82 deviceMemorySize: 24.00GiB deviceMemoryBandwidth: 871.81GiB/s
2021-04-01 10:09:40.809789: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2021-04-01 10:09:40.810112: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2021-04-01 10:09:40.810416: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2021-04-01 10:09:40.810728: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2021-04-01 10:09:40.811035: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2021-04-01 10:09:40.811338: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2021-04-01 10:09:40.811624: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2021-04-01 10:09:40.811974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-04-01 10:09:42.137546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-01 10:09:42.137800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
2021-04-01 10:09:42.137935: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
2021-04-01 10:09:42.138228: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 19112 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:44:00.0, compute capability: 8.6)
2021-04-01 10:09:42.141792: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2af27c8f640 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-04-01 10:09:42.142075: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 3090, Compute Capability 8.6
Epoch 1/1000
2021-04-01 10:09:46.249137: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2021-04-01 10:09:47.132659: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
这是我的依赖关系
# Name Version Build Channel
_tflow_select 2.3.0 gpu
absl-py 0.12.0 pyhd8ed1ab_0 conda-forge
aiohttp 3.7.4 py37hcc03f2d_0 conda-forge
astor 0.8.1 pyh9f0ad1d_0 conda-forge
astunparse 1.6.3 pyhd8ed1ab_0 conda-forge
async-timeout 3.0.1 py_1000 conda-forge
attrs 20.3.0 pyhd3deb0d_0 conda-forge
blinker 1.4 py_1 conda-forge
brotlipy 0.7.0 py37hcc03f2d_1001 conda-forge
ca-certificates 2020.12.5 h5b45459_0 conda-forge
cachetools 4.2.1 pyhd8ed1ab_0 conda-forge
certifi 2020.12.5 py37h03978a9_1 conda-forge
cffi 1.14.5 py37hd8e9650_0 conda-forge
chardet 4.0.0 py37h03978a9_1 conda-forge
click 7.1.2 pyh9f0ad1d_0 conda-forge
cryptography 3.4.7 py37h20c650d_0 conda-forge
cudatoolkit 10.1.243 h3826478_8 conda-forge
cudnn 7.6.5.32 h36d860d_1 conda-forge
cycler 0.10.0 pypi_0 pypi
gast 0.3.3 py_0 conda-forge
google-auth 1.28.0 pyh44b312d_0 conda-forge
google-auth-oauthlib 0.4.1 py_2 conda-forge
google-pasta 0.2.0 pyh8c360ce_0 conda-forge
grpcio 1.36.1 py37h04d2302_0 conda-forge
h5py 2.10.0 nompi_py37h23cfb99_105 conda-forge
hdf5 1.10.6 nompi_h5268f04_1114 conda-forge
idna 2.10 pyh9f0ad1d_0 conda-forge
importlib-metadata 3.10.0 py37h03978a9_0 conda-forge
intel-openmp 2020.3 h57928b3_311 conda-forge
keras 2.4.3 py_0 conda-forge
keras-applications 1.0.8 py_1 conda-forge
keras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forge
keras-video-generators 1.0.14 pypi_0 pypi
kiwisolver 1.3.1 pypi_0 pypi
krb5 1.17.2 hbae68bd_0 conda-forge
libblas 3.9.0 8_mkl conda-forge
libcblas 3.9.0 8_mkl conda-forge
libcurl 7.76.0 hf1763fc_0 conda-forge
liblapack 3.9.0 8_mkl conda-forge
libprotobuf 3.15.6 h7755175_0 conda-forge
libssh2 1.9.0 h680486a_6 conda-forge
m2w64-gcc-libgfortran 5.3.0 6 conda-forge
m2w64-gcc-libs 5.3.0 7 conda-forge
m2w64-gcc-libs-core 5.3.0 7 conda-forge
m2w64-gmp 6.1.0 2 conda-forge
m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge
markdown 3.3.4 pyhd8ed1ab_0 conda-forge
matplotlib 3.4.1 pypi_0 pypi
mkl 2020.4 hb70f87d_311 conda-forge
msys2-conda-epoch 20160418 1 conda-forge
multidict 5.1.0 py37hcc03f2d_1 conda-forge
numpy 1.19.5 py37hd20adf4_1 conda-forge
oauthlib 3.0.1 py_0 conda-forge
opencv-python 4.5.1.48 pypi_0 pypi
openssl 1.1.1k h8ffe710_0 conda-forge
opt_einsum 3.3.0 py_0 conda-forge
pillow 8.1.2 pypi_0 pypi
pip 21.0.1 pyhd8ed1ab_0 conda-forge
protobuf 3.15.6 py37hf2a7229_0 conda-forge
pyasn1 0.4.8 py_0 conda-forge
pyasn1-modules 0.2.7 py_0 conda-forge
pycparser 2.20 pyh9f0ad1d_2 conda-forge
pyjwt 2.0.1 pyhd8ed1ab_1 conda-forge
pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge
pyparsing 2.4.7 pypi_0 pypi
pyreadline 2.1 py37h03978a9_1003 conda-forge
pysocks 1.7.1 py37h03978a9_3 conda-forge
python 3.7.10 h7840368_100_cpython conda-forge
python-dateutil 2.8.1 pypi_0 pypi
python_abi 3.7 1_cp37m conda-forge
pyyaml 5.4.1 pypi_0 pypi
requests 2.25.1 pyhd3deb0d_0 conda-forge
requests-oauthlib 1.3.0 pyh9f0ad1d_0 conda-forge
rsa 4.7.2 pyh44b312d_0 conda-forge
scipy 1.6.2 py37h924764e_0 conda-forge
setuptools 49.6.0 py37h03978a9_3 conda-forge
six 1.15.0 pyh9f0ad1d_0 conda-forge
sqlite 3.35.3 h8ffe710_0 conda-forge
tensorboard 2.4.1 pyhd8ed1ab_0 conda-forge
tensorboard-plugin-wit 1.8.0 pyh44b312d_0 conda-forge
tensorflow 2.3.0 mkl_py37h936c3e2_0
tensorflow-base 2.3.0 gpu_py37h18d21e4_0
tensorflow-estimator 2.4.0 pyh9656e83_0 conda-forge
tensorflow-gpu 2.3.0 he13fc11_0
termcolor 1.1.0 py_2 conda-forge
tk 8.6.10 h8ffe710_1 conda-forge
typing-extensions 3.7.4.3 0 conda-forge
typing_extensions 3.7.4.3 py_0 conda-forge
urllib3 1.26.4 pyhd8ed1ab_0 conda-forge
vc 14.2 hb210afc_4 conda-forge
vs2015_runtime 14.28.29325 h5e1d092_4 conda-forge
werkzeug 1.0.1 pyh9f0ad1d_0 conda-forge
wheel 0.36.2 pyhd3deb0d_0 conda-forge
win_inet_pton 1.1.0 py37h03978a9_2 conda-forge
wincertstore 0.2 py37h03978a9_1006 conda-forge
wrapt 1.12.1 py37hcc03f2d_3 conda-forge
yaml 0.2.5 he774522_0 conda-forge
yarl 1.6.3 py37hcc03f2d_1 conda-forge
zipp 3.4.1 pyhd8ed1ab_0 conda-forge
zlib 1.2.11 h62dcd97_1010 conda-forge
更新： 完全相同的代码在 2080TI 上工作，我将其与刚得到的 3090 切换并做了一个新环境并卸载了所有 2080TI 相关软件。
这是模型
Total params: 5,502,338
Trainable params: 5,500,418
Non-trainable params: 1,920
我还添加了 CUDA_CACHE_MAXSIZE=2147483648 到我的环境变量中。
RTX 2080 Ti 卡基于 Turing 兼容 CUDA version start with 10.x 的架构，其中 RTX 3090 卡基于 Ampere 兼容的架构CUDA version start with 11.x.
所以您的 gpu 卡的兼容 Tensorflow 版本是 2.4.0。更多详情可以参考here.