在 div class 上迭代并提取信息

Question

import requests
from bs4 import BeautifulSoup

url = "https://boulder.noshdelivery.co/restaurants"
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")

restaurant_wrapper = soup.find(class_ = "dd_rest_list")

restaurants = restaurant_wrapper.find_all(class_="menu__vendor-name")
restaurant_street_address = restaurant_wrapper.find("span", itemprop="streetAddress")
restaurant_address_locality = restaurant_wrapper.find("span", itemprop="addressLocality")

def extract_restaurant_data(restaurant):
    restaurant_title = restaurant_wrapper.find(class_="menu__vendor-name") 
    return  {
        "title" : restaurant_title.text.strip(),
        "streetAddress": restaurant_street_address.text.strip(),
        "addressLocality": restaurant_address_locality.text.strip()

    }

results = [extract_restaurant_data(restaurant) for restaurant in restaurants]
print(results)

我想知道为什么这段代码打印完全相同的信息而不遍历餐厅列表。

我的输出是这样的

{'title': '5280 Cafe At Rallysport', 'streetAddress': '2727 29th St.', 
'addressLocality': 'Boulder'},  
{'title': '5280 Cafe At Rallysport', 'streetAddress': '2727 29th St.', '   addressLocality': 'Boulder'}........

信息相同。我不知道为什么我的代码没有遍历“餐厅”列表中的不同名称

Answer 1

你只做了一个find数据。在每个部分做一个 find_all 然后 zip 他们在一起！

restaurant_details = zip(
    restaurant_wrapper.find_all(class_="menu__vendor-name"),
    restaurant_wrapper.find_all("span", itemprop="streetAddress"),
    restaurant_wrapper.find_all("span", itemprop="addressLocality"),
)
results = [
    {
        "title": title.text.strip(),
        "streetAddress": street_address.text.strip(),
        "addressLocality": address_locality.text.strip()
    }
    for title, street_address, address_locality in restaurant_details
]
print(results)

Answer 2

您的函数中写入了 restaurant_wrapper.find(class_="menu__vendor-name")，因此每次运行时它只会打印第一次出现的 class menu__vendor-name.

要在每次迭代中打印新餐厅的详细信息，您必须单独访问每个 Web 元素。

您可以使用以下代码获取所有餐厅的详细信息。

restwords = restaurant_wrapper.find_all("div", {"class": "dd_restwords"})

def extract_restaurant_data(restaurant):
  title = restaurant.find("div", {"class": "menu__vendor-name"}).text
  streetAddress = restaurant.find("span", {"itemprop": "streetAddress"}).text
  addressLocality = restaurant.find("span", {"itemprop": "addressLocality"}).text

  rest_data = {
      "title": title, 
      "streetAddress": streetAddress, 
      "addressLocality": addressLocality
      }
  
  return rest_data

for restaurant in restwords:
  print(extract_restaurant_data(restaurant))

在 div class 上迭代并提取信息

Iterate and extract info over div class

python

beautifulsoup

web-scraping