How to handle Attribute Error: 'NoneType' object has no attribute 'findAll'

Question

在使用以下功能扫描大量网站时，我收到错误消息（见下文）。我可以在下面的函数中添加任何 except 步骤来处理此类错误吗？

async def scrape(url):
    try:
        r = requests.get(url, timeout=(3, 6))
        r.raise_for_status()
        soup = BeautifulSoup(r.content, 'html.parser')
        data = {
"coming soon": soup.body.findAll(text = re.compile("coming soon", re.I)),
"Opening Soon": soup.body.findAll(text = re.compile("Opening Soon", re.I)),
"Under Construction": soup.body.findAll(text = re.compile("Under Construction", re.I)),
"Currently Unavailable": soup.body.findAll(text = re.compile("Currently Unavailable", re.I)),
"button": soup.findAll(text = re.compile('button2.js'))}
        results[url] = data
    except (requests.exceptions.ConnectionError, requests.exceptions.Timeout, requests.exceptions.MissingSchema):
        status[url] = "Connection Error"
    except (requests.exceptions.HTTPError):
        status[url] = "Http Error"
    except (requests.exceptions.TooManyRedirects):
        status[url] = "Redirects"
    except (requests.exceptions.RequestException) as err:
        status[url] = "Fatal Error: " + err + url
    else:
        status[url] = "OK"

错误：

Task exception was never retrieved
future: <Task finished name='Task-4782' coro=<scrape() done, defined at crawler.py:47>  exception=AttributeError("'NoneType' object has no attribute 'findAll'")>
Traceback (most recent call last):
  File "crawler.py", line 53, in scrape
    "coming soon": soup.body.findAll(text = re.compile("coming soon", re.I)),
AttributeError: 'NoneType' object has no attribute 'findAll'

Answer 1

发生这种情况是因为soup.body是None，我们可以简单地用if条件处理这种情况。

async def scrape(url):
        try:
            r = requests.get(url, timeout=(3, 6))
            r.raise_for_status()
            soup = BeautifulSoup(r.content, 'html.parser')
            if soup.body:
               data = {
               "coming soon": soup.body.findAll(text = re.compile("coming soon", re.I)),
               "Opening Soon": soup.body.findAll(text = re.compile("Opening Soon", re.I)),
               "Under Construction": soup.body.findAll(text = re.compile("Under Construction", re.I)),
               "Currently Unavailable": soup.body.findAll(text = re.compile("Currently Unavailable", re.I)),
               "button": soup.findAll(text = re.compile('button2.js'))}
               results[url] = data
        except (requests.exceptions.ConnectionError, requests.exceptions.Timeout, requests.exceptions.MissingSchema):
            status[url] = "Connection Error"
        except (requests.exceptions.HTTPError):
            status[url] = "Http Error"
        except (requests.exceptions.TooManyRedirects):
            status[url] = "Redirects"
        except (requests.exceptions.RequestException) as err:
            status[url] = "Fatal Error: " + err + url
        else:
            status[url] = "OK"

How to handle Attribute Error: 'NoneType' object has no attribute 'findAll'

How to handle Attribute Error: 'NoneType' object has no attribute 'findAll'

python

beautifulsoup

web-crawler

web-scraping