当我在 EC2 上 运行 爬虫时，为什么它不能正常工作？
when i run crawler on EC2, why didn't it work well?
我在 ec2 上设置 docker 并上传爬虫 scrapy 代码。自从昨天我 运行 ec2，它可以很好地收集数据。但突然间它不能正常工作。我猜是 EC2 的问题，因为它可以正常工作到今天午餐时间。
然后我尝试检查 EC2 条件，输入 df -h，它显示
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 796M 19M 778M 3% /run
/dev/xvda1 7.7G 7.7G 0 100% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/loop0 33M 33M 0 100% /snap/snapd/11107
/dev/loop1 33M 33M 0 100% /snap/snapd/11402
/dev/loop2 56M 56M 0 100% /snap/core18/1988
/dev/loop3 132M 132M 0 100% /snap/docker/796
/dev/loop4 34M 34M 0 100% /snap/amazon-ssm-agent/3552
tmpfs 796M 0 796M 0% /run/user/1000
肯定是昨天 /dev/xvda1 这部分 ('used') 不是 100%。
所以我试图增加 EBS 上 /dev/xvda1 卷的大小。
希望这次尝试可以解决问题。
在 EBS 之后，当我在 EC2 终端上输入 lsblk 时，它显示了这个
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 32.3M 1 loop /snap/snapd/11107
loop1 7:1 0 33.3M 1 loop /snap/amazon-ssm-agent/3552
loop2 7:2 0 32.3M 1 loop /snap/snapd/11402
loop3 7:3 0 55.5M 1 loop /snap/core18/1988
loop4 7:4 0 131.6M 1 loop /snap/docker/796
xvda 202:0 0 16G 0 disk
└─xvda1 202:1 0 8G 0 part /
无论如何我想知道如何删除 /dev/xvda1size。因为我计划我的爬虫将定期公开工作，但如果我必须增加卷大小，它总是会让我变得混乱。
请帮我，
我的爬虫怎么能不担心 /dev/xvda1 工作。
当我输入时sudo find / -type f -size +10M -exec ls -lh {} \;
它显示了这个
sudo: unable to resolve host ip-172-31-79-98: Resource temporarily unavailable
-rwxr-xr-x 1 root root 19M Mar 8 10:47 /snap/snapd/11402/usr/bin/snap
-rwxr-xr-x 1 root root 13M Mar 8 10:47 /snap/snapd/11402/usr/lib/snapd/snap-bootstrap
-rwxr-xr-x 1 root root 12M Mar 8 10:47 /snap/snapd/11402/usr/lib/snapd/snap-preseed
-rwxr-xr-x 1 root root 22M Mar 8 10:47 /snap/snapd/11402/usr/lib/snapd/snapd
-rwxr-xr-x 1 root root 19M Feb 10 10:44 /snap/snapd/11107/usr/bin/snap
-rwxr-xr-x 1 root root 13M Feb 10 10:44 /snap/snapd/11107/usr/lib/snapd/snap-bootstrap
-rwxr-xr-x 1 root root 12M Feb 10 10:44 /snap/snapd/11107/usr/lib/snapd/snap-preseed
-rwxr-xr-x 1 root root 22M Feb 10 10:44 /snap/snapd/11107/usr/lib/snapd/snapd
-rwxr-xr-x 1 root root 51M Feb 5 15:56 /snap/docker/796/bin/containerd
-rwxr-xr-x 1 root root 26M Feb 5 15:56 /snap/docker/796/bin/ctr
-rwxr-xr-x 1 root root 29M Feb 5 16:02 /snap/docker/796/bin/dnet
-rwxr-xr-x 1 root root 82M Feb 5 15:58 /snap/docker/796/bin/docker
-rwxr-xr-x 1 root root 31M Feb 5 16:02 /snap/docker/796/bin/docker-machine
-rwxr-xr-x 1 root root 98M Feb 5 16:00 /snap/docker/796/bin/dockerd
-rwxr-xr-x 1 root root 13M Feb 5 16:03 /snap/docker/796/bin/runc
-rw-r--r-- 1 root root 26M Mar 16 2020 /snap/docker/796/usr/lib/x86_64-linux-gnu/libicudat a.so.60.2
-rwxr-xr-x 1 root root 24M Dec 21 16:37 /snap/amazon-ssm-agent/3552/amazon-ssm-agent
-rwxr-xr-x 1 root root 39M Dec 21 16:37 /snap/amazon-ssm-agent/3552/ssm-agent-worker
-rwxr-xr-x 1 root root 27M Dec 21 16:37 /snap/amazon-ssm-agent/3552/ssm-cli
-rwxr-xr-x 1 root root 36M Dec 21 16:37 /snap/amazon-ssm-agent/3552/ssm-document-worker
-rwxr-xr-x 1 root root 13M Dec 21 16:37 /snap/amazon-ssm-agent/3552/ssm-session-logger
-rwxr-xr-x 1 root root 34M Dec 21 16:37 /snap/amazon-ssm-agent/3552/ssm-session-worker
-rw-r--r-- 1 root root 21M Mar 22 07:02 /boot/initrd.img-5.4.0-1039-aws
-rw-r--r-- 1 root root 21M Mar 22 07:01 /boot/initrd.img-5.4.0-1038-aws
-rw------- 1 root root 16M Mar 23 07:14 /sys/devices/pci0000:00/0000:00:03.0/resource1
-rw------- 1 root root 16M Mar 23 07:14 /sys/devices/pci0000:00/0000:00:03.0/resource1_wc
-rw------- 1 root root 32M Mar 23 07:14 /sys/devices/pci0000:00/0000:00:02.0/resource0_wc
-rw------- 1 root root 32M Mar 23 07:14 /sys/devices/pci0000:00/0000:00:02.0/resource0
-r-------- 1 root root 128T Mar 23 07:14 /proc/kcore
find: ‘/proc/2374/task/2374/fdinfo/6’: No such file or directory
find: ‘/proc/2374/fdinfo/5’: No such file or directory
-rwxr-xr-x 1 root root 15M Mar 8 14:14 /usr/bin/runc
-rwxr-xr-x 1 root root 16M Mar 2 20:16 /usr/bin/rootlesskit
-rwxr-xr-x 1 root root 19M Feb 2 08:21 /usr/bin/snap
-rwxr-xr-x 1 root root 111M Mar 2 20:16 /usr/bin/dockerd
-rwxr-xr-x 1 root root 30M Mar 8 14:14 /usr/bin/ctr
-rwxr-xr-x 1 root root 55M Mar 8 14:14 /usr/bin/containerd
-rwxr-xr-x 1 root root 11M Nov 23 2018 /usr/bin/lxc
-rwxr-xr-x 1 root root 69M Mar 2 20:16 /usr/bin/docker
-rw-r--r-- 1 root root 20M Feb 27 15:10 /usr/lib/python2.7/config-x86_64-linux-gnu/libpytho n2.7.a
-rw-r--r-- 1 root root 20M Feb 27 15:10 /usr/lib/python2.7/config-x86_64-linux-gnu/libpytho n2.7-pic.a
-rw-r--r-- 1 root root 26M Mar 16 2020 /usr/lib/x86_64-linux-gnu/libicudata.so.60.2
-rwxr-xr-x 1 root root 18M Nov 23 2018 /usr/lib/lxd/lxd
-rwxr-xr-x 1 root root 13M Feb 2 08:21 /usr/lib/snapd/snap-bootstrap
-rwxr-xr-x 1 root root 22M Feb 2 08:21 /usr/lib/snapd/snapd
-rw-r--r-- 1 root root 36M Jan 26 15:33 /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/lib python3.6m-pic.a
-rw-r--r-- 1 root root 37M Jan 26 15:33 /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/lib python3.6m.a
-rwxr-xr-x 1 root root 25M Dec 4 2019 /usr/lib/gcc/x86_64-linux-gnu/7/cc1plus
-rwxr-xr-x 1 root root 22M Dec 4 2019 /usr/lib/gcc/x86_64-linux-gnu/7/lto1
-rwxr-xr-x 1 root root 23M Dec 4 2019 /usr/lib/gcc/x86_64-linux-gnu/7/cc1
-rwxr-xr-x 1 root root 62M Mar 2 20:16 /usr/libexec/docker/cli-plugins/docker-buildx
-rwxr-xr-x 1 root root 36M Mar 2 20:16 /usr/libexec/docker/cli-plugins/docker-app
-r--r--r-- 1 root root 21M Mar 22 07:05 /home/ubuntu/crawlab-1/.git/objects/pack/pack-6c0a8218eeb814173c677e3df54850423986d710.pack
-rw-r--r-- 1 root root 26M Apr 26 2018 /var/lib/apt/lists/us-east-1.ec2.archive.ubuntu.com_ubuntu_dists_bionic_universe_i18n_Translati on-en
-rw-r--r-- 1 root root 11M Mar 22 15:48 /var/lib/apt/lists/us-east-1.ec2.archive.ubuntu.com_ubuntu_dists_bionic-updates_main_binary-amd 64_Packages
-rw-r--r-- 1 root root 49M Apr 26 2018 /var/lib/apt/lists/us-east-1.ec2.archive.ubuntu.com_ubuntu_dists_bionic_universe_binary-amd64_P ackages
-rw------- 2 root root 132M Mar 22 07:14 /var/lib/snapd/cache/aa3d5fe426c66baa3261f1641b9b5fe284cd08a16761eb765438cb5396b1c0ba96abe1e29 6135543871dbe68423be6b1
-rw------- 2 root root 56M Feb 24 17:36 /var/lib/snapd/seed/snaps/core18_1988.snap
-rw------- 2 root root 34M Feb 24 17:36 /var/lib/snapd/seed/snaps/amazon-ssm-agent_3552.snap
-rw------- 1 root root 32M Feb 24 17:36 /var/lib/snapd/seed/snaps/snapd_11036.snap
-rw------- 1 root root 33M Mar 23 04:18 /var/lib/snapd/snaps/snapd_11402.snap
-rw------- 2 root root 56M Feb 24 17:36 /var/lib/snapd/snaps/core18_1988.snap
-rw------- 2 root root 34M Feb 24 17:36 /var/lib/snapd/snaps/amazon-ssm-agent_3552.snap
-rw------- 2 root root 132M Mar 22 07:14 /var/lib/snapd/snaps/docker_796.snap
-rw------- 1 root root 33M Mar 22 09:39 /var/lib/snapd/snaps/snapd_11107.snap
-rw-r----- 1 root root 14M Mar 23 06:45 /var/snap/docker/common/var-lib-docker/containers/5f951fb4ee5e008ad09cf3e5a86bbd5c25061a0a11237 8c1b6123a12d2c16c27/5f951fb4ee5e008ad09cf3e5a86bbd5c25061a0a112378c1b6123a12d2c16c27-json.log
-rw------- 1 999 docker 14M Mar 23 06:41 /var/snap/docker/common/var-lib-docker/volumes/b5276a9f65fc8a8995ce9a76c50bcd27ddd90043201d4a6 f609dda8aae79d462/_data/collection-10-5521340255836326071.wt
-rw------- 1 999 docker 100M Mar 23 06:45 /var/snap/docker/common/var-lib-docker/volumes/b5276a9f65fc8a8995ce9a76c50bcd27ddd90043201d4a 6f609dda8aae79d462/_data/journal/WiredTigerLog.0000000034
-rw------- 1 999 docker 100M Mar 23 06:35 /var/snap/docker/common/var-lib-docker/volumes/b5276a9f65fc8a8995ce9a76c50bcd27ddd90043201d4a 6f609dda8aae79d462/_data/journal/WiredTigerPreplog.0000000001
-rw------- 1 999 docker 100M Mar 23 06:35 /var/snap/docker/common/var-lib-docker/volumes/b5276a9f65fc8a8995ce9a76c50bcd27ddd90043201d4a 6f609dda8aae79d462/_data/journal/WiredTigerPreplog.0000000002
-rwxr-xr-x 1 root root 12M Mar 13 04:01 /var/snap/docker/common/var-lib-docker/overlay2/b1991d45bb2288ac16dcab9c1860d3eb821543b2e66a22f e261d9a159dd2019e/diff/usr/local/bin/redis-server
-rw-r--r-- 1 root root 27M Mar 17 2020 /var/snap/docker/common/var-lib-docker/overlay2/7a9d391c4792c7234e369212d9e4300b3279bb7a805a187 36a83f30428033492/diff/usr/lib/x86_64-linux-gnu/libicudata.so.66.1
-rwxr-xr-x 1 root root 26M Aug 8 2020 /var/snap/docker/common/var-lib-docker/overlay2/7a9d391c4792c7234e369212d9e4300b3279bb7a805a187 36a83f30428033492/diff/usr/lib/gcc/x86_64-linux-gnu/9/cc1plus
-rwxr-xr-x 1 root root 24M Aug 8 2020 /var/snap/docker/common/var-lib-docker/overlay2/7a9d391c4792c7234e369212d9e4300b3279bb7a805a187 36a83f30428033492/diff/usr/lib/gcc/x86_64-linux-gnu/9/lto1
-rwxr-xr-x 1 root root 25M Aug 8 2020 /var/snap/docker/common/var-lib-docker/overlay2/7a9d391c4792c7234e369212d9e4300b3279bb7a805a187 36a83f30428033492/diff/usr/lib/gcc/x86_64-linux-gnu/9/cc1
-rw-r--r-- 1 root root 18M Apr 23 2020 /var/snap/docker/common/var-lib-docker/overlay2/7a9d391c4792c7234e369212d9e4300b3279bb7a805a187 36a83f30428033492/diff/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal_universe_binary-amd64_Packages.lz4
-rw------- 1 root root 15M Mar 22 08:07 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef7411 d7a1a1c98f3e88add/diff/tmp/.com.google.Chrome.HNlE5E/Default/Cache/b5f47fbc4d23c5f6_0
-rw------- 1 root root 15M Mar 22 08:05 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef7411 d7a1a1c98f3e88add/diff/tmp/.com.google.Chrome.diyKvW/Default/Cache/b5f47fbc4d23c5f6_0
-rw-r--r-- 1 root root 73M Mar 23 06:42 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef7411 d7a1a1c98f3e88add/diff/tmp/chrome.deb
-rwxr-xr-x 1 root root 12M Jan 27 03:32 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef7411 d7a1a1c98f3e88add/diff/usr/local/bin/chromedriver
-rwxr-xr-x 1 root staff 11M Mar 22 07:27 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef741 1d7a1a1c98f3e88add/diff/usr/local/lib/python3.8/dist-packages/chromedriver_py/chromedriver_linux64
-rwxr-xr-x 1 root staff 16M Mar 22 07:27 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef741 1d7a1a1c98f3e88add/diff/usr/local/lib/python3.8/dist-packages/chromedriver_py/chromedriver_mac64
-rw------- 1 root root 19M Mar 22 07:27 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef7411 d7a1a1c98f3e88add/diff/root/.cache/pip/http/6/0/4/2/9/60429761adcdcb005ac1b1419b8c7b286bf36316aa97d5f62b7ccb34
-rw------- 1 root root 19M Mar 22 07:26 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef7411 d7a1a1c98f3e88add/diff/root/.cache/pip/http/e/f/1/7/2/ef17289f5b2b98078e36fd05289a14ec9579924ee0366839cdf7611a
-rwxr-xr-x 1 root root 165M Mar 11 23:57 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef741 1d7a1a1c98f3e88add/diff/opt/google/chrome/chrome
-rwxr-xr-x 1 ubuntu ubuntu 20M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongoexport
-rwxr-xr-x 1 ubuntu ubuntu 20M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongorestore
-rwxr-xr-x 1 root root 51M Dec 19 2013 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e5159971 c54372cccfc5004f9/diff/usr/bin/mongo
-rwxr-xr-x 1 root root 62M Dec 19 2013 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e5159971 c54372cccfc5004f9/diff/usr/bin/mongos
-rwxr-xr-x 1 ubuntu ubuntu 20M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongoimport
-rwxr-xr-x 1 ubuntu ubuntu 16M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/bsondump
-rwxr-xr-x 1 ubuntu ubuntu 19M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongotop
-rwxr-xr-x 1 ubuntu ubuntu 20M Jan 22 17:14 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongofiles
-rwxr-xr-x 1 ubuntu ubuntu 19M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongostat
-rwxr-xr-x 1 ubuntu ubuntu 20M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongodump
-rwxr-xr-x 1 root root 79M Dec 19 2013 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e5159971 c54372cccfc5004f9/diff/usr/bin/mongod
-rwxr-xr-x 1 root root 48M Dec 11 02:49 /var/snap/docker/common/var-lib-docker/overlay2/d06370e7f6ed90afffd39f640723116f2818ff7b96619d4 499e497870be4df5f/diff/opt/bin/crawlab
-rwxr-xr-x 1 root root 48M Dec 11 02:55 /var/snap/docker/common/var-lib-docker/overlay2/0d641b4b0ae1855a87a93c6f8d275be503912cf266ac69b 03be851e6a42b881b/diff/usr/local/bin/crawlab-server
-rw-r--r-- 1 root root 73M Mar 22 07:22 /var/snap/docker/common/var-lib-docker/overlay2/a033a3ea16c7b174830b9872fb6dea5a504363b42b9b9dc acd8427c4a9e475a2/diff/tmp/chrome.deb
-rwxr-xr-x 1 root root 12M Jan 27 03:32 /var/snap/docker/common/var-lib-docker/overlay2/a033a3ea16c7b174830b9872fb6dea5a504363b42b9b9dc acd8427c4a9e475a2/diff/usr/local/bin/chromedriver
-rwxr-xr-x 1 root staff 11M Mar 22 07:19 /var/snap/docker/common/var-lib-docker/overlay2/a033a3ea16c7b174830b9872fb6dea5a504363b42b9b9d cacd8427c4a9e475a2/diff/usr/local/lib/python3.8/dist-packages/chromedriver_py/chromedriver_linux64
-rwxr-xr-x 1 root staff 16M Mar 22 07:19 /var/snap/docker/common/var-lib-docker/overlay2/a033a3ea16c7b174830b9872fb6dea5a504363b42b9b9d cacd8427c4a9e475a2/diff/usr/local/lib/python3.8/dist-packages/chromedriver_py/chromedriver_mac64
-rw------- 1 root root 19M Mar 22 07:18 /var/snap/docker/common/var-lib-docker/overlay2/a033a3ea16c7b174830b9872fb6dea5a504363b42b9b9dc acd8427c4a9e475a2/diff/root/.cache/pip/http/6/0/4/2/9/60429761adcdcb005ac1b1419b8c7b286bf36316aa97d5f62b7ccb34
-rwxr-xr-x 1 root root 165M Mar 11 23:57 /var/snap/docker/common/var-lib-docker/overlay2/a033a3ea16c7b174830b9872fb6dea5a504363b42b9b9d cacd8427c4a9e475a2/diff/opt/google/chrome/chrome
-rw-r--r-- 1 root root 39M Mar 23 04:18 /var/cache/apt/srcpkgcache.bin
-rw-r--r-- 1 root root 39M Mar 23 04:18 /var/cache/apt/pkgcache.bin
-rw-r--r-- 1 root root 40M Mar 8 23:12 /var/cache/apt/archives/docker-ce-cli_5%3a20.10.5~3-0~ubuntu-bionic_amd64.deb
-rw-r--r-- 1 root root 24M Mar 8 23:12 /var/cache/apt/archives/docker-ce_5%3a20.10.5~3-0~ubuntu-bionic_amd64.deb
-rw-r--r-- 1 root root 27M Mar 9 00:39 /var/cache/apt/archives/containerd.io_1.4.4-1_amd64.deb
-rw-r--r-- 1 root root 40M Mar 8 23:12 /var/cache/apt/archives/docker-ce-cli_5%3a20.10.4~3-0~ubuntu-bionic_amd64.deb
-rw-r--r-- 1 root root 11M Mar 1 06:20 /var/cache/apt/archives/linux-aws-5.4-headers-5.4.0-1039_5.4.0-1039.41~18.04.1_all.deb
-rw-r--r-- 1 root root 14M Mar 1 06:20 /var/cache/apt/archives/linux-modules-5.4.0-1039-aws_5.4.0-1039.41~18.04.1_amd64.deb
-rw-r--r-- 1 root root 24M Mar 8 23:12 /var/cache/apt/archives/docker-ce_5%3a20.10.4~3-0~ubuntu-bionic_amd64.deb
-rw-r-----+ 1 root systemd-journal 16M Mar 22 11:18 /var/log/journal/ea09dcfc1f4741e1b257d2463c341f29/system@9cf46084d404438fb64d33dbd9 5ecc2a-0000000000000001-0005be1a8eeb62f4.journal
我创建了新实例，然后我设置了 docker 和 scrapy blah blah blah...
无论如何，在我爬了 1 次之后。当我输入 df -Th 时，它显示了这个。
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs tmpfs 796M 1.1M 795M 1% /run
/dev/xvda1 ext4 7.7G 5.7G 2.1G 74% /
tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/loop0 squashfs 34M 34M 0 100% /snap/amazon-ssm-agent/3552
/dev/loop1 squashfs 32M 32M 0 100% /snap/snapd/11036
/dev/loop2 squashfs 56M 56M 0 100% /snap/core18/1988
overlay overlay 7.7G 5.7G 2.1G 74% /var/lib/docker/overlay2/e7965025d305ee6d51b55dee17fe547b70b51ebea9750d22e1ff55337d54b3ea/merged
overlay overlay 7.7G 5.7G 2.1G 74% /var/lib/docker/overlay2/414d298f3057de6461ca2d5b9ceeb290e35215500cbfdaa3d000ad7afda46359/merged
tmpfs tmpfs 796M 0 796M 0% /run/user/1000
/dev/loop3 squashfs 33M 33M 0 100% /snap/snapd/11107
overlay overlay 7.7G 5.7G 2.1G 74% /var/lib/docker/overlay2/b03949ee74691b6ef3f1471c37ebf58d5cd6947b8e139430b4f1c0776c06016c/merged
overlay overlay 7.7G 5.7G 2.1G 74% /var/lib/docker/overlay2/02159a834b6c0c51d81686f024aab575da916d001d91a7027d2c3868f7b2696e/merged
在我爬了 9 次之后它显示了这个
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs tmpfs 796M 1.1M 795M 1% /run
/dev/xvda1 ext4 7.7G 7.0G 732M 91% /
tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/loop0 squashfs 34M 34M 0 100% /snap/amazon-ssm-agent/3552
/dev/loop1 squashfs 32M 32M 0 100% /snap/snapd/11036
/dev/loop2 squashfs 56M 56M 0 100% /snap/core18/1988
overlay overlay 7.7G 7.0G 732M 91% /var/lib/docker/overlay2/e7965025d305ee6d51b55dee17fe547b70b51ebea9750d22e1ff55337d54b3ea/merged
tmpfs tmpfs 796M 0 796M 0% /run/user/1000
/dev/loop3 squashfs 33M 33M 0 100% /snap/snapd/11107
overlay overlay 7.7G 7.0G 732M 91% /var/lib/docker/overlay2/b03949ee74691b6ef3f1471c37ebf58d5cd6947b8e139430b4f1c0776c06016c/merged
overlay overlay 7.7G 7.0G 732M 91% /var/lib/docker/overlay2/02159a834b6c0c51d81686f024aab575da916d001d91a7027d2c3868f7b2696e/merged
overlay overlay 7.7G 7.0G 732M 91% /var/lib/docker/overlay2/414d298f3057de6461ca2d5b9ceeb290e35215500cbfdaa3d000ad7afda46359/merged
在我抓取 1 次后，我在终端上写了 du -h --max-depth=3 /var/lib | sort -h，它显示了这个。
....
304M /var/lib/docker/volumes/b8a9c7ac497041f99b629fdb14e9dfc283f4108d73512d3f87a79e9a5af3d0fd
306M /var/lib/docker/volumes
350M /var/lib/docker/overlay2/daf820fd04371808cf9213038ab552fdec5fc659d1fbf3b1643529e8ba1049e9
430M /var/lib/docker/overlay2/3569c54068321f2973fcde4f80695f482840915df635105f6bdb68226a113741
437M /var/lib/docker/overlay2/414d298f3057de6461ca2d5b9ceeb290e35215500cbfdaa3d000ad7afda46359
2.1G /var/lib/docker/overlay2/b03949ee74691b6ef3f1471c37ebf58d5cd6947b8e139430b4f1c0776c06016c
2.5G /var/lib/docker/overlay2/02159a834b6c0c51d81686f024aab575da916d001d91a7027d2c3868f7b2696e
6.3G /var/lib/docker/overlay2
6.6G /var/lib/docker
6.9G /var/lib
9次后
306M /var/lib/docker/volumes/b8a9c7ac497041f99b629fdb14e9dfc283f4108d73512d3f87a79e9a5af3d0fd
308M /var/lib/docker/volumes
350M /var/lib/docker/overlay2/daf820fd04371808cf9213038ab552fdec5fc659d1fbf3b1643529e8ba1049e9
430M /var/lib/docker/overlay2/3569c54068321f2973fcde4f80695f482840915df635105f6bdb68226a113741
437M /var/lib/docker/overlay2/414d298f3057de6461ca2d5b9ceeb290e35215500cbfdaa3d000ad7afda46359
2.3G /var/lib/docker/overlay2/b03949ee74691b6ef3f1471c37ebf58d5cd6947b8e139430b4f1c0776c06016c
2.7G /var/lib/docker/overlay2/02159a834b6c0c51d81686f024aab575da916d001d91a7027d2c3868f7b2696e
6.7G /var/lib/docker/overlay2
7.0G /var/lib/docker
7.4G /var/lib
很明显您的 EC2 实例存在磁盘使用问题（磁盘已满）。
您已成功增加卷大小（如 lsblk 输出所示），但您还需要扩展文件系统，如 here.
所述
基本上，您应该 运行 sudo growpart /dev/xvda 1 和 sudo resize2fs /dev/xvda1。再次使用df -h进行验证。
为了优化性能，您应该考虑：
- 为您的工作负载使用专用卷（即在
/mnt/data 上安装一个新的更大的 EBS 卷，并将您的应用程序配置为使用此路径进行数据存储）
- 调整此卷的大小 and/or 更改 type（即使用配置的 IOPS SSD）
Scrapy也支持S3 for storage.
您可以configure scrapy logging使用特定的日志文件（--logfile FILE，稍后您可以将其删除）或完全禁用它（--nolog，不推荐）。
我在 ec2 上设置 docker 并上传爬虫 scrapy 代码。自从昨天我 运行 ec2，它可以很好地收集数据。但突然间它不能正常工作。我猜是 EC2 的问题，因为它可以正常工作到今天午餐时间。
然后我尝试检查 EC2 条件，输入 df -h，它显示
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 796M 19M 778M 3% /run
/dev/xvda1 7.7G 7.7G 0 100% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/loop0 33M 33M 0 100% /snap/snapd/11107
/dev/loop1 33M 33M 0 100% /snap/snapd/11402
/dev/loop2 56M 56M 0 100% /snap/core18/1988
/dev/loop3 132M 132M 0 100% /snap/docker/796
/dev/loop4 34M 34M 0 100% /snap/amazon-ssm-agent/3552
tmpfs 796M 0 796M 0% /run/user/1000
肯定是昨天 /dev/xvda1 这部分 ('used') 不是 100%。
所以我试图增加 EBS 上 /dev/xvda1 卷的大小。
希望这次尝试可以解决问题。
在 EBS 之后，当我在 EC2 终端上输入 lsblk 时，它显示了这个
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 32.3M 1 loop /snap/snapd/11107
loop1 7:1 0 33.3M 1 loop /snap/amazon-ssm-agent/3552
loop2 7:2 0 32.3M 1 loop /snap/snapd/11402
loop3 7:3 0 55.5M 1 loop /snap/core18/1988
loop4 7:4 0 131.6M 1 loop /snap/docker/796
xvda 202:0 0 16G 0 disk
└─xvda1 202:1 0 8G 0 part /
无论如何我想知道如何删除 /dev/xvda1size。因为我计划我的爬虫将定期公开工作，但如果我必须增加卷大小，它总是会让我变得混乱。
请帮我，
我的爬虫怎么能不担心 /dev/xvda1 工作。
当我输入时sudo find / -type f -size +10M -exec ls -lh {} \;
它显示了这个
sudo: unable to resolve host ip-172-31-79-98: Resource temporarily unavailable
-rwxr-xr-x 1 root root 19M Mar 8 10:47 /snap/snapd/11402/usr/bin/snap
-rwxr-xr-x 1 root root 13M Mar 8 10:47 /snap/snapd/11402/usr/lib/snapd/snap-bootstrap
-rwxr-xr-x 1 root root 12M Mar 8 10:47 /snap/snapd/11402/usr/lib/snapd/snap-preseed
-rwxr-xr-x 1 root root 22M Mar 8 10:47 /snap/snapd/11402/usr/lib/snapd/snapd
-rwxr-xr-x 1 root root 19M Feb 10 10:44 /snap/snapd/11107/usr/bin/snap
-rwxr-xr-x 1 root root 13M Feb 10 10:44 /snap/snapd/11107/usr/lib/snapd/snap-bootstrap
-rwxr-xr-x 1 root root 12M Feb 10 10:44 /snap/snapd/11107/usr/lib/snapd/snap-preseed
-rwxr-xr-x 1 root root 22M Feb 10 10:44 /snap/snapd/11107/usr/lib/snapd/snapd
-rwxr-xr-x 1 root root 51M Feb 5 15:56 /snap/docker/796/bin/containerd
-rwxr-xr-x 1 root root 26M Feb 5 15:56 /snap/docker/796/bin/ctr
-rwxr-xr-x 1 root root 29M Feb 5 16:02 /snap/docker/796/bin/dnet
-rwxr-xr-x 1 root root 82M Feb 5 15:58 /snap/docker/796/bin/docker
-rwxr-xr-x 1 root root 31M Feb 5 16:02 /snap/docker/796/bin/docker-machine
-rwxr-xr-x 1 root root 98M Feb 5 16:00 /snap/docker/796/bin/dockerd
-rwxr-xr-x 1 root root 13M Feb 5 16:03 /snap/docker/796/bin/runc
-rw-r--r-- 1 root root 26M Mar 16 2020 /snap/docker/796/usr/lib/x86_64-linux-gnu/libicudat a.so.60.2
-rwxr-xr-x 1 root root 24M Dec 21 16:37 /snap/amazon-ssm-agent/3552/amazon-ssm-agent
-rwxr-xr-x 1 root root 39M Dec 21 16:37 /snap/amazon-ssm-agent/3552/ssm-agent-worker
-rwxr-xr-x 1 root root 27M Dec 21 16:37 /snap/amazon-ssm-agent/3552/ssm-cli
-rwxr-xr-x 1 root root 36M Dec 21 16:37 /snap/amazon-ssm-agent/3552/ssm-document-worker
-rwxr-xr-x 1 root root 13M Dec 21 16:37 /snap/amazon-ssm-agent/3552/ssm-session-logger
-rwxr-xr-x 1 root root 34M Dec 21 16:37 /snap/amazon-ssm-agent/3552/ssm-session-worker
-rw-r--r-- 1 root root 21M Mar 22 07:02 /boot/initrd.img-5.4.0-1039-aws
-rw-r--r-- 1 root root 21M Mar 22 07:01 /boot/initrd.img-5.4.0-1038-aws
-rw------- 1 root root 16M Mar 23 07:14 /sys/devices/pci0000:00/0000:00:03.0/resource1
-rw------- 1 root root 16M Mar 23 07:14 /sys/devices/pci0000:00/0000:00:03.0/resource1_wc
-rw------- 1 root root 32M Mar 23 07:14 /sys/devices/pci0000:00/0000:00:02.0/resource0_wc
-rw------- 1 root root 32M Mar 23 07:14 /sys/devices/pci0000:00/0000:00:02.0/resource0
-r-------- 1 root root 128T Mar 23 07:14 /proc/kcore
find: ‘/proc/2374/task/2374/fdinfo/6’: No such file or directory
find: ‘/proc/2374/fdinfo/5’: No such file or directory
-rwxr-xr-x 1 root root 15M Mar 8 14:14 /usr/bin/runc
-rwxr-xr-x 1 root root 16M Mar 2 20:16 /usr/bin/rootlesskit
-rwxr-xr-x 1 root root 19M Feb 2 08:21 /usr/bin/snap
-rwxr-xr-x 1 root root 111M Mar 2 20:16 /usr/bin/dockerd
-rwxr-xr-x 1 root root 30M Mar 8 14:14 /usr/bin/ctr
-rwxr-xr-x 1 root root 55M Mar 8 14:14 /usr/bin/containerd
-rwxr-xr-x 1 root root 11M Nov 23 2018 /usr/bin/lxc
-rwxr-xr-x 1 root root 69M Mar 2 20:16 /usr/bin/docker
-rw-r--r-- 1 root root 20M Feb 27 15:10 /usr/lib/python2.7/config-x86_64-linux-gnu/libpytho n2.7.a
-rw-r--r-- 1 root root 20M Feb 27 15:10 /usr/lib/python2.7/config-x86_64-linux-gnu/libpytho n2.7-pic.a
-rw-r--r-- 1 root root 26M Mar 16 2020 /usr/lib/x86_64-linux-gnu/libicudata.so.60.2
-rwxr-xr-x 1 root root 18M Nov 23 2018 /usr/lib/lxd/lxd
-rwxr-xr-x 1 root root 13M Feb 2 08:21 /usr/lib/snapd/snap-bootstrap
-rwxr-xr-x 1 root root 22M Feb 2 08:21 /usr/lib/snapd/snapd
-rw-r--r-- 1 root root 36M Jan 26 15:33 /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/lib python3.6m-pic.a
-rw-r--r-- 1 root root 37M Jan 26 15:33 /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/lib python3.6m.a
-rwxr-xr-x 1 root root 25M Dec 4 2019 /usr/lib/gcc/x86_64-linux-gnu/7/cc1plus
-rwxr-xr-x 1 root root 22M Dec 4 2019 /usr/lib/gcc/x86_64-linux-gnu/7/lto1
-rwxr-xr-x 1 root root 23M Dec 4 2019 /usr/lib/gcc/x86_64-linux-gnu/7/cc1
-rwxr-xr-x 1 root root 62M Mar 2 20:16 /usr/libexec/docker/cli-plugins/docker-buildx
-rwxr-xr-x 1 root root 36M Mar 2 20:16 /usr/libexec/docker/cli-plugins/docker-app
-r--r--r-- 1 root root 21M Mar 22 07:05 /home/ubuntu/crawlab-1/.git/objects/pack/pack-6c0a8218eeb814173c677e3df54850423986d710.pack
-rw-r--r-- 1 root root 26M Apr 26 2018 /var/lib/apt/lists/us-east-1.ec2.archive.ubuntu.com_ubuntu_dists_bionic_universe_i18n_Translati on-en
-rw-r--r-- 1 root root 11M Mar 22 15:48 /var/lib/apt/lists/us-east-1.ec2.archive.ubuntu.com_ubuntu_dists_bionic-updates_main_binary-amd 64_Packages
-rw-r--r-- 1 root root 49M Apr 26 2018 /var/lib/apt/lists/us-east-1.ec2.archive.ubuntu.com_ubuntu_dists_bionic_universe_binary-amd64_P ackages
-rw------- 2 root root 132M Mar 22 07:14 /var/lib/snapd/cache/aa3d5fe426c66baa3261f1641b9b5fe284cd08a16761eb765438cb5396b1c0ba96abe1e29 6135543871dbe68423be6b1
-rw------- 2 root root 56M Feb 24 17:36 /var/lib/snapd/seed/snaps/core18_1988.snap
-rw------- 2 root root 34M Feb 24 17:36 /var/lib/snapd/seed/snaps/amazon-ssm-agent_3552.snap
-rw------- 1 root root 32M Feb 24 17:36 /var/lib/snapd/seed/snaps/snapd_11036.snap
-rw------- 1 root root 33M Mar 23 04:18 /var/lib/snapd/snaps/snapd_11402.snap
-rw------- 2 root root 56M Feb 24 17:36 /var/lib/snapd/snaps/core18_1988.snap
-rw------- 2 root root 34M Feb 24 17:36 /var/lib/snapd/snaps/amazon-ssm-agent_3552.snap
-rw------- 2 root root 132M Mar 22 07:14 /var/lib/snapd/snaps/docker_796.snap
-rw------- 1 root root 33M Mar 22 09:39 /var/lib/snapd/snaps/snapd_11107.snap
-rw-r----- 1 root root 14M Mar 23 06:45 /var/snap/docker/common/var-lib-docker/containers/5f951fb4ee5e008ad09cf3e5a86bbd5c25061a0a11237 8c1b6123a12d2c16c27/5f951fb4ee5e008ad09cf3e5a86bbd5c25061a0a112378c1b6123a12d2c16c27-json.log
-rw------- 1 999 docker 14M Mar 23 06:41 /var/snap/docker/common/var-lib-docker/volumes/b5276a9f65fc8a8995ce9a76c50bcd27ddd90043201d4a6 f609dda8aae79d462/_data/collection-10-5521340255836326071.wt
-rw------- 1 999 docker 100M Mar 23 06:45 /var/snap/docker/common/var-lib-docker/volumes/b5276a9f65fc8a8995ce9a76c50bcd27ddd90043201d4a 6f609dda8aae79d462/_data/journal/WiredTigerLog.0000000034
-rw------- 1 999 docker 100M Mar 23 06:35 /var/snap/docker/common/var-lib-docker/volumes/b5276a9f65fc8a8995ce9a76c50bcd27ddd90043201d4a 6f609dda8aae79d462/_data/journal/WiredTigerPreplog.0000000001
-rw------- 1 999 docker 100M Mar 23 06:35 /var/snap/docker/common/var-lib-docker/volumes/b5276a9f65fc8a8995ce9a76c50bcd27ddd90043201d4a 6f609dda8aae79d462/_data/journal/WiredTigerPreplog.0000000002
-rwxr-xr-x 1 root root 12M Mar 13 04:01 /var/snap/docker/common/var-lib-docker/overlay2/b1991d45bb2288ac16dcab9c1860d3eb821543b2e66a22f e261d9a159dd2019e/diff/usr/local/bin/redis-server
-rw-r--r-- 1 root root 27M Mar 17 2020 /var/snap/docker/common/var-lib-docker/overlay2/7a9d391c4792c7234e369212d9e4300b3279bb7a805a187 36a83f30428033492/diff/usr/lib/x86_64-linux-gnu/libicudata.so.66.1
-rwxr-xr-x 1 root root 26M Aug 8 2020 /var/snap/docker/common/var-lib-docker/overlay2/7a9d391c4792c7234e369212d9e4300b3279bb7a805a187 36a83f30428033492/diff/usr/lib/gcc/x86_64-linux-gnu/9/cc1plus
-rwxr-xr-x 1 root root 24M Aug 8 2020 /var/snap/docker/common/var-lib-docker/overlay2/7a9d391c4792c7234e369212d9e4300b3279bb7a805a187 36a83f30428033492/diff/usr/lib/gcc/x86_64-linux-gnu/9/lto1
-rwxr-xr-x 1 root root 25M Aug 8 2020 /var/snap/docker/common/var-lib-docker/overlay2/7a9d391c4792c7234e369212d9e4300b3279bb7a805a187 36a83f30428033492/diff/usr/lib/gcc/x86_64-linux-gnu/9/cc1
-rw-r--r-- 1 root root 18M Apr 23 2020 /var/snap/docker/common/var-lib-docker/overlay2/7a9d391c4792c7234e369212d9e4300b3279bb7a805a187 36a83f30428033492/diff/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal_universe_binary-amd64_Packages.lz4
-rw------- 1 root root 15M Mar 22 08:07 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef7411 d7a1a1c98f3e88add/diff/tmp/.com.google.Chrome.HNlE5E/Default/Cache/b5f47fbc4d23c5f6_0
-rw------- 1 root root 15M Mar 22 08:05 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef7411 d7a1a1c98f3e88add/diff/tmp/.com.google.Chrome.diyKvW/Default/Cache/b5f47fbc4d23c5f6_0
-rw-r--r-- 1 root root 73M Mar 23 06:42 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef7411 d7a1a1c98f3e88add/diff/tmp/chrome.deb
-rwxr-xr-x 1 root root 12M Jan 27 03:32 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef7411 d7a1a1c98f3e88add/diff/usr/local/bin/chromedriver
-rwxr-xr-x 1 root staff 11M Mar 22 07:27 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef741 1d7a1a1c98f3e88add/diff/usr/local/lib/python3.8/dist-packages/chromedriver_py/chromedriver_linux64
-rwxr-xr-x 1 root staff 16M Mar 22 07:27 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef741 1d7a1a1c98f3e88add/diff/usr/local/lib/python3.8/dist-packages/chromedriver_py/chromedriver_mac64
-rw------- 1 root root 19M Mar 22 07:27 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef7411 d7a1a1c98f3e88add/diff/root/.cache/pip/http/6/0/4/2/9/60429761adcdcb005ac1b1419b8c7b286bf36316aa97d5f62b7ccb34
-rw------- 1 root root 19M Mar 22 07:26 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef7411 d7a1a1c98f3e88add/diff/root/.cache/pip/http/e/f/1/7/2/ef17289f5b2b98078e36fd05289a14ec9579924ee0366839cdf7611a
-rwxr-xr-x 1 root root 165M Mar 11 23:57 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef741 1d7a1a1c98f3e88add/diff/opt/google/chrome/chrome
-rwxr-xr-x 1 ubuntu ubuntu 20M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongoexport
-rwxr-xr-x 1 ubuntu ubuntu 20M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongorestore
-rwxr-xr-x 1 root root 51M Dec 19 2013 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e5159971 c54372cccfc5004f9/diff/usr/bin/mongo
-rwxr-xr-x 1 root root 62M Dec 19 2013 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e5159971 c54372cccfc5004f9/diff/usr/bin/mongos
-rwxr-xr-x 1 ubuntu ubuntu 20M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongoimport
-rwxr-xr-x 1 ubuntu ubuntu 16M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/bsondump
-rwxr-xr-x 1 ubuntu ubuntu 19M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongotop
-rwxr-xr-x 1 ubuntu ubuntu 20M Jan 22 17:14 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongofiles
-rwxr-xr-x 1 ubuntu ubuntu 19M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongostat
-rwxr-xr-x 1 ubuntu ubuntu 20M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongodump
-rwxr-xr-x 1 root root 79M Dec 19 2013 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e5159971 c54372cccfc5004f9/diff/usr/bin/mongod
-rwxr-xr-x 1 root root 48M Dec 11 02:49 /var/snap/docker/common/var-lib-docker/overlay2/d06370e7f6ed90afffd39f640723116f2818ff7b96619d4 499e497870be4df5f/diff/opt/bin/crawlab
-rwxr-xr-x 1 root root 48M Dec 11 02:55 /var/snap/docker/common/var-lib-docker/overlay2/0d641b4b0ae1855a87a93c6f8d275be503912cf266ac69b 03be851e6a42b881b/diff/usr/local/bin/crawlab-server
-rw-r--r-- 1 root root 73M Mar 22 07:22 /var/snap/docker/common/var-lib-docker/overlay2/a033a3ea16c7b174830b9872fb6dea5a504363b42b9b9dc acd8427c4a9e475a2/diff/tmp/chrome.deb
-rwxr-xr-x 1 root root 12M Jan 27 03:32 /var/snap/docker/common/var-lib-docker/overlay2/a033a3ea16c7b174830b9872fb6dea5a504363b42b9b9dc acd8427c4a9e475a2/diff/usr/local/bin/chromedriver
-rwxr-xr-x 1 root staff 11M Mar 22 07:19 /var/snap/docker/common/var-lib-docker/overlay2/a033a3ea16c7b174830b9872fb6dea5a504363b42b9b9d cacd8427c4a9e475a2/diff/usr/local/lib/python3.8/dist-packages/chromedriver_py/chromedriver_linux64
-rwxr-xr-x 1 root staff 16M Mar 22 07:19 /var/snap/docker/common/var-lib-docker/overlay2/a033a3ea16c7b174830b9872fb6dea5a504363b42b9b9d cacd8427c4a9e475a2/diff/usr/local/lib/python3.8/dist-packages/chromedriver_py/chromedriver_mac64
-rw------- 1 root root 19M Mar 22 07:18 /var/snap/docker/common/var-lib-docker/overlay2/a033a3ea16c7b174830b9872fb6dea5a504363b42b9b9dc acd8427c4a9e475a2/diff/root/.cache/pip/http/6/0/4/2/9/60429761adcdcb005ac1b1419b8c7b286bf36316aa97d5f62b7ccb34
-rwxr-xr-x 1 root root 165M Mar 11 23:57 /var/snap/docker/common/var-lib-docker/overlay2/a033a3ea16c7b174830b9872fb6dea5a504363b42b9b9d cacd8427c4a9e475a2/diff/opt/google/chrome/chrome
-rw-r--r-- 1 root root 39M Mar 23 04:18 /var/cache/apt/srcpkgcache.bin
-rw-r--r-- 1 root root 39M Mar 23 04:18 /var/cache/apt/pkgcache.bin
-rw-r--r-- 1 root root 40M Mar 8 23:12 /var/cache/apt/archives/docker-ce-cli_5%3a20.10.5~3-0~ubuntu-bionic_amd64.deb
-rw-r--r-- 1 root root 24M Mar 8 23:12 /var/cache/apt/archives/docker-ce_5%3a20.10.5~3-0~ubuntu-bionic_amd64.deb
-rw-r--r-- 1 root root 27M Mar 9 00:39 /var/cache/apt/archives/containerd.io_1.4.4-1_amd64.deb
-rw-r--r-- 1 root root 40M Mar 8 23:12 /var/cache/apt/archives/docker-ce-cli_5%3a20.10.4~3-0~ubuntu-bionic_amd64.deb
-rw-r--r-- 1 root root 11M Mar 1 06:20 /var/cache/apt/archives/linux-aws-5.4-headers-5.4.0-1039_5.4.0-1039.41~18.04.1_all.deb
-rw-r--r-- 1 root root 14M Mar 1 06:20 /var/cache/apt/archives/linux-modules-5.4.0-1039-aws_5.4.0-1039.41~18.04.1_amd64.deb
-rw-r--r-- 1 root root 24M Mar 8 23:12 /var/cache/apt/archives/docker-ce_5%3a20.10.4~3-0~ubuntu-bionic_amd64.deb
-rw-r-----+ 1 root systemd-journal 16M Mar 22 11:18 /var/log/journal/ea09dcfc1f4741e1b257d2463c341f29/system@9cf46084d404438fb64d33dbd9 5ecc2a-0000000000000001-0005be1a8eeb62f4.journal
我创建了新实例，然后我设置了 docker 和 scrapy blah blah blah...
无论如何，在我爬了 1 次之后。当我输入 df -Th 时，它显示了这个。
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs tmpfs 796M 1.1M 795M 1% /run
/dev/xvda1 ext4 7.7G 5.7G 2.1G 74% /
tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/loop0 squashfs 34M 34M 0 100% /snap/amazon-ssm-agent/3552
/dev/loop1 squashfs 32M 32M 0 100% /snap/snapd/11036
/dev/loop2 squashfs 56M 56M 0 100% /snap/core18/1988
overlay overlay 7.7G 5.7G 2.1G 74% /var/lib/docker/overlay2/e7965025d305ee6d51b55dee17fe547b70b51ebea9750d22e1ff55337d54b3ea/merged
overlay overlay 7.7G 5.7G 2.1G 74% /var/lib/docker/overlay2/414d298f3057de6461ca2d5b9ceeb290e35215500cbfdaa3d000ad7afda46359/merged
tmpfs tmpfs 796M 0 796M 0% /run/user/1000
/dev/loop3 squashfs 33M 33M 0 100% /snap/snapd/11107
overlay overlay 7.7G 5.7G 2.1G 74% /var/lib/docker/overlay2/b03949ee74691b6ef3f1471c37ebf58d5cd6947b8e139430b4f1c0776c06016c/merged
overlay overlay 7.7G 5.7G 2.1G 74% /var/lib/docker/overlay2/02159a834b6c0c51d81686f024aab575da916d001d91a7027d2c3868f7b2696e/merged
在我爬了 9 次之后它显示了这个
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs tmpfs 796M 1.1M 795M 1% /run
/dev/xvda1 ext4 7.7G 7.0G 732M 91% /
tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/loop0 squashfs 34M 34M 0 100% /snap/amazon-ssm-agent/3552
/dev/loop1 squashfs 32M 32M 0 100% /snap/snapd/11036
/dev/loop2 squashfs 56M 56M 0 100% /snap/core18/1988
overlay overlay 7.7G 7.0G 732M 91% /var/lib/docker/overlay2/e7965025d305ee6d51b55dee17fe547b70b51ebea9750d22e1ff55337d54b3ea/merged
tmpfs tmpfs 796M 0 796M 0% /run/user/1000
/dev/loop3 squashfs 33M 33M 0 100% /snap/snapd/11107
overlay overlay 7.7G 7.0G 732M 91% /var/lib/docker/overlay2/b03949ee74691b6ef3f1471c37ebf58d5cd6947b8e139430b4f1c0776c06016c/merged
overlay overlay 7.7G 7.0G 732M 91% /var/lib/docker/overlay2/02159a834b6c0c51d81686f024aab575da916d001d91a7027d2c3868f7b2696e/merged
overlay overlay 7.7G 7.0G 732M 91% /var/lib/docker/overlay2/414d298f3057de6461ca2d5b9ceeb290e35215500cbfdaa3d000ad7afda46359/merged
在我抓取 1 次后，我在终端上写了 du -h --max-depth=3 /var/lib | sort -h，它显示了这个。
....
304M /var/lib/docker/volumes/b8a9c7ac497041f99b629fdb14e9dfc283f4108d73512d3f87a79e9a5af3d0fd
306M /var/lib/docker/volumes
350M /var/lib/docker/overlay2/daf820fd04371808cf9213038ab552fdec5fc659d1fbf3b1643529e8ba1049e9
430M /var/lib/docker/overlay2/3569c54068321f2973fcde4f80695f482840915df635105f6bdb68226a113741
437M /var/lib/docker/overlay2/414d298f3057de6461ca2d5b9ceeb290e35215500cbfdaa3d000ad7afda46359
2.1G /var/lib/docker/overlay2/b03949ee74691b6ef3f1471c37ebf58d5cd6947b8e139430b4f1c0776c06016c
2.5G /var/lib/docker/overlay2/02159a834b6c0c51d81686f024aab575da916d001d91a7027d2c3868f7b2696e
6.3G /var/lib/docker/overlay2
6.6G /var/lib/docker
6.9G /var/lib
9次后
306M /var/lib/docker/volumes/b8a9c7ac497041f99b629fdb14e9dfc283f4108d73512d3f87a79e9a5af3d0fd
308M /var/lib/docker/volumes
350M /var/lib/docker/overlay2/daf820fd04371808cf9213038ab552fdec5fc659d1fbf3b1643529e8ba1049e9
430M /var/lib/docker/overlay2/3569c54068321f2973fcde4f80695f482840915df635105f6bdb68226a113741
437M /var/lib/docker/overlay2/414d298f3057de6461ca2d5b9ceeb290e35215500cbfdaa3d000ad7afda46359
2.3G /var/lib/docker/overlay2/b03949ee74691b6ef3f1471c37ebf58d5cd6947b8e139430b4f1c0776c06016c
2.7G /var/lib/docker/overlay2/02159a834b6c0c51d81686f024aab575da916d001d91a7027d2c3868f7b2696e
6.7G /var/lib/docker/overlay2
7.0G /var/lib/docker
7.4G /var/lib
很明显您的 EC2 实例存在磁盘使用问题（磁盘已满）。
您已成功增加卷大小（如 lsblk 输出所示），但您还需要扩展文件系统，如 here.
基本上，您应该 运行 sudo growpart /dev/xvda 1 和 sudo resize2fs /dev/xvda1。再次使用df -h进行验证。
为了优化性能，您应该考虑：
- 为您的工作负载使用专用卷（即在
/mnt/data上安装一个新的更大的 EBS 卷，并将您的应用程序配置为使用此路径进行数据存储） - 调整此卷的大小 and/or 更改 type（即使用配置的 IOPS SSD）
Scrapy也支持S3 for storage.
您可以configure scrapy logging使用特定的日志文件（--logfile FILE，稍后您可以将其删除）或完全禁用它（--nolog，不推荐）。