Beautiful Soup 和 Python 的属性错误

Question

我有一段可用的代码，然后我今天运行它坏了。我已经删除了给我带来问题的相关部分。

from bs4 import BeautifulSoup
import requests

webpage = requests.get('http://www.bbcgoodfood.com/search/recipes?query=')

soup = BeautifulSoup(webpage.content) 
links = soup.find("div",{"class":"main row grid-padding"}).find_all("h2",{"class":"node-title"})

for link in links:
    print(link.a["href"])

这给我一个错误"Attribute Error: 'NoneType' object has no attribute 'find_all'"

这个错误到底告诉我什么？

find_all() 是美丽汤文档中的有效命令。查看网页的源代码，我找到我想要的对象的路径似乎是有道理的。

我认为网站一定发生了变化，因为我不知道我的代码怎么会停止工作。但是我不太理解错误信息...

感谢您的帮助！

Answer 1

这是因为当您尝试访问该页面时，它给您 permission denied，所以 soup.find() returns 什么都没有 None，None没有 find_all() 的属性，这给你一个 AttributeError.

from bs4 import BeautifulSoup
import requests

webpage = requests.get('http://www.bbcgoodfood.com/search/recipes?query=')


print webpage.content
<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>

You don't have permission to access "http&#58;&#47;&#47;www&#46;bbcgoodfood&#46;com&#47;search&#47;recipes&#63;" on this server.<P>
Reference&#32;&#35;18&#46;4fa9cd17&#46;1428789762&#46;680369dc
</BODY>
</HTML>

如果你通过添加一个 header 和正确的用户代理来解决这个问题，就像@Vader 建议的那样，你的代码将运行正常：

...
headers = {'User-agent': 'Mozilla/5.0'}
webpage = requests.get('http://www.bbcgoodfood.com/search/recipes?query=', headers=headers)

soup = BeautifulSoup(webpage.content) 
links = soup.find("div",{"class":"main row grid-padding"}).find_all("h2",{"class":"node-title"})

for link in links:
    print(link.a["href"])

/recipes/4942/lemon-drizzle-cake
/recipes/3092/ultimate-chocolate-cake
/recipes/3228/chilli-con-carne
/recipes/3229/yummy-scrummy-carrot-cake
/recipes/1223/bestever-brownies
/recipes/1167651/chicken-and-chorizo-jambalaya
/recipes/2089/spiced-carrot-and-lentil-soup
/recipes/1521/summerinwinter-chicken
/recipes/1364/spicy-root-and-lentil-casserole
/recipes/4814/mustardstuffed-chicken
/recipes/4622/classic-scones-with-jam-and-clotted-cream
/recipes/333614/red-lentil-chickpea-and-chilli-soup
/recipes/5605/falafel-burgers
/recipes/11695/raspberry-bakewell-cake
/recipes/4686/chicken-biryani

Answer 2

您尝试解析的站点没有 "like" 您的用户代理和 returns 403 错误，然后解析器失败，因为它找不到 div。尝试将用户代理更改为其中一种浏览器的用户代理：

webpage = requests.get('http://www.bbcgoodfood.com/search/recipes?query=', headers = {'user-agent':'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'})

Beautiful Soup 和 Python 的属性错误

Attribute Error With Beautiful Soup And Python

python

beautifulsoup

web-scraping

python-2.7