下一页未加载

Question

我的代码无法加载下一页。此外，当我手动刷新时，网页显示“拒绝访问”。

options = ChromeOptions()
options.add_argument("headless") # to hide window in 'background'
driver = Chrome(executable_path="C:/Users/samira.zade/AppData/Local/Programs/Python/Driver/chromedriver_win32/chromedriver.exe")
driver.get("https://www.connection.com/IPA/Shop/Product/Search?SearchType=1&term=tp- 
link#1st+Matches~12~List")# here change your link
driver.maximize_window()
time.sleep(5)
wait=WebDriverWait(driver,10)
pagenum = 10
data_connection = []
i = 0
for i in range(pagenum):
     driver.get(f"https://www.connection.com/IPA/Shop/Product/Search?SearchType=1&term=tp-link#{i+1}~Best+Matches~12~List")
     time.sleep(5)
     wait=WebDriverWait(driver,10)

Answer 1

我通常尽量避免使用 Selenium（或者换句话说，将其用作最后的手段）来抓取网站。产品数据来源来自https://www.connection.com/product/searchpage。您可以将页面查询作为参数传入。

我刚刚用 pandas 解析了 table 以快速向您展示。如果你想从页面中提取 other/more，你可以使用 BeautifulSoup.

import pandas as pd
import requests

url = "https://www.connection.com/product/searchpage"
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}

page = 1
continueLoop = True
result_df = pd.DataFrame()
while continueLoop == True:
    
    payload = {
        'SearchType': '1',
        'term': 'tp-link',
        '1st Matches~12~List': '',
        'pageNumber': page,
        'pageSize': '36',
        'url': 'https://www.connection.com/IPA/Shop/Product/Search',
        'mode': 'List'}
    
    response = requests.get(url, headers=headers, params=payload)
    if 'Pagination limit reached.' in response.text:
        continueLoop = False
        print('Pagination limit reached.')
        continue
    
    df = pd.read_html(response.text)[0]
    result_df = result_df.append(df).reset_index(drop=True)
    print(f'Collected page: {page}')
    page+=1

输出：

print(result_df)
           Product Image  ...                                        Price
0    Compare  Image Link  ...   .70 Qty:  Add To Cart  Add to Quicklist
1    Compare  Image Link  ...   .42 Qty:  Add To Cart  Add to Quicklist
2    Compare  Image Link  ...   .04 Qty:  Add To Cart  Add to Quicklist
3    Compare  Image Link  ...   .03 Qty:  Add To Cart  Add to Quicklist
4    Compare  Image Link  ...  4.07 Qty:  Add To Cart  Add to Quicklist
..                   ...  ...                                          ...
175  Compare  Image Link  ...   .98 Qty:  Add To Cart  Add to Quicklist
176  Compare  Image Link  ...   .77 Qty:  Add To Cart  Add to Quicklist
177  Compare  Image Link  ...   .99 Qty:  Add To Cart  Add to Quicklist
178  Compare  Image Link  ...   .99 Qty:  Add To Cart  Add to Quicklist
179  Compare  Image Link  ...   .99 Qty:  Add To Cart  Add to Quicklist

[180 rows x 4 columns]

下一页未加载

Next page doesn't load

python

selenium

beautifulsoup

web-scraping