为什么我在开头和之后得到一大块 html 代码而不是 href 链接，然后有一个链接列表被正确提取

Question

`

from selenium import webdriver
from bs4 import BeautifulSoup as bs
import re

class Scraper:
    def __init__(self):
        self.driver = webdriver.Chrome(r'C:\Users\gkhat\Downloads\chromedriver.exe')
        self.url = "http://www.carwale.com/"
        self.href = []

    def load_url(self):
        self.driver.get(self.url + 'used/cars-for-sale/#sc=-1&so=-1&pn=1')
    def scroll_down(self):
        self.driver.execute_script('window.scrollTo(0, document.body.scrollHeight)')
    def read_data(self):
        main = self.driver.find_element_by_xpath('/html/body/div[12]/form/section[2]/div[1]/div[4]/div[1]/div[3]/div[2]')
        soup = bs(main.get_attribute("innerHTML"), "html.parser")
        print(soup)
        for elem in soup.findAll('h2', {'class': 'card-detail-block__title'}):
            print(elem.a['href'])
            self.href.append(str(elem.a['href']))`

Answer 1

这个代码

from selenium import webdriver
from bs4 import BeautifulSoup as bs
import re
driver = webdriver.Chrome(executable_path='C:/chromedriver.exe')
url = "http://www.carwale.com/used/cars-for-sale/#sc=-1&so=-1&pn=1"
driver.get(url)
driver.execute_script('window.scrollTo(0, document.body.scrollHeight)')
main = driver.find_element_by_xpath('/html/body/div[12]/form/section[2]/div[1]/div[4]/div[1]/div[3]/div[2]')
soup = bs(main.get_attribute("innerHTML"), "html.parser")

for elem in soup.findAll('h2', {'class': 'card-detail-block__title'}):
    print(elem.a['href'])

给这个

/used/cars-in-delhi/marutisuzuki-ciaz-2017-2018-d2251253/?slot=1&rk=1&isP=true
/used/cars-in-bangalore/hyundai-i10-2010-2017-d2295085/?slot=2&rk=2&isP=true
/used/cars-in-faridabad/marutisuzuki-alto-k10-d2332673/?slot=3&rk=3&isP=true
/used/cars-in-mumbai/renault-kwid-2015-2019-d2333781/?slot=4&rk=4&isP=true
/used/cars-in-mumbai/hyundai-i10-2010-2017-d2308033/?slot=5&rk=5&isP=true
/used/cars-in-mumbai/toyota-innova-crysta-2016-2020-d2327063/?slot=6&rk=6&isP=true
/used/cars-in-kolkata/landrover-freelander-2-2009-2011-d2321851/?slot=7&rk=7&isP=true
/used/cars-in-pune/hyundai-eon-2011-2019-d2295279/?slot=8&rk=8&isP=true
/used/cars-in-navi-mumbai/renault-kwid-2015-2019-d2303909/?slot=0&rk=9&isP=false
/used/cars-in-navi-mumbai/marutisuzuki-vitara-brezza-2016-2020-d2303835/?slot=0&rk=10&isP=false
/used/cars-in-navi-mumbai/marutisuzuki-swift-2014-2018-d2332235/?slot=0&rk=11&isP=false
/used/cars-in-navi-mumbai/volkswagen-vento-2015-2019-d2285395/?slot=0&rk=12&isP=false
/used/cars-in-navi-mumbai/jeep-compass-2017-2021-d2290491/?slot=0&rk=13&isP=false
/used/cars-in-navi-mumbai/honda-wr-v-2017-2020-d2332263/?slot=0&rk=14&isP=false
/used/cars-in-navi-mumbai/renault-kwid-2015-2019-d2332249/?slot=0&rk=15&isP=false
/used/cars-in-navi-mumbai/volkswagen-polo-2012-2014-d2310489/?slot=0&rk=16&isP=false
/used/cars-in-navi-mumbai/marutisuzuki-a-star-2008-2012-d2324317/?slot=0&rk=17&isP=false
/used/cars-in-navi-mumbai/hyundai-elite-i20-2016-2017-d2281903/?slot=0&rk=18&isP=false
/used/cars-in-navi-mumbai/hyundai-i20-active-2015-2018-d2316481/?slot=0&rk=19&isP=false
/used/cars-in-navi-mumbai/toyota-innova-crysta-2016-2020-d2324333/?slot=0&rk=20&isP=false
/used/cars-in-delhi/ford-figo-2015-2019-d2241843/?slot=0&rk=21&isP=false
/used/cars-in-vadodara/bmw-x3-2011-2014-d2198006/?slot=0&rk=22&isP=false
/used/cars-in-vadodara/volkswagen-jetta20082011-d2197994/?slot=0&rk=23&isP=false
/used/cars-in-vadodara/hyundai-santafe20112014-d2227084/?slot=0&rk=24&isP=false
/used/cars-in-vadodara/skoda-octavia-2013-2015-d2227082/?slot=0&rk=25&isP=false
/used/cars-in-vadodara/bmw-5-series-2007-2010-d2227083/?slot=0&rk=26&isP=false
/used/cars-in-mumbai/honda-cr-v-2013-2018-d2309393/?slot=0&rk=27&isP=false
/used/cars-in-mumbai/bmw-3-series-2007-2009-d2309385/?slot=0&rk=28&isP=false
/used/cars-in-mumbai/skoda-rapid-2014-2015-d2306945/?slot=0&rk=29&isP=false
/used/cars-in-mumbai/audi-a6-2015-2019-d2309389/?slot=0&rk=30&isP=false
/used/cars-in-mumbai/mercedesbenz-e-class-2013-2015-d2321627/?slot=0&rk=31&isP=false
/used/cars-in-delhi/volkswagen-vento-2010-2012-d2287775/?slot=0&rk=32&isP=false

我刚刚删除了 print(soup)。

为什么我在开头和之后得到一大块 html 代码而不是 href 链接，然后有一个链接列表被正确提取

why am I getting a big chunk of html code instead of href links in the beginning and after that there is a list of links which are properly extracted

python

selenium

beautifulsoup

hyperlink

web-scraping