无法使用 Selenium 从多个 URL 打印数据 Python

Question

据说这段代码有效，但我只面临一个问题 url 它随后通过错误抓取数据，如下图所示，帮助我解决了这个问题。它通过 未创建会话 错误

后仅打印一个 link

import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
import pandas as pd 
import time 

url = ["https://www.skechers.com/women/shoes/ultra-flex---twilight-twinkle/149173.html",
"https://www.skechers.com/women/shoes/ultra-flex---statements/12841.html",
"https://www.skechers.com/women/shoes/ultra-flex/12843.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-hyper-burst---extreme-outlook/124580.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-hyper-burst---lunar-mars/124577.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-hyper-burst/124575.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-stability---coco-jazz/124603.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-stability---magnificent-glow/124602.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-joy/124191.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-joy---paradise/15601.html",
"https://www.skechers.com/women/shoes/skech-air-element-2.0---vivid-blush/149408.html",
"https://www.skechers.com/women/shoes/skech-air-element-2.0---kaleidoscope-splash/149134.html",
"https://www.skechers.com/women/shoes/skech-air-element-2.0/149062.html",
"https://www.skechers.com/women/shoes/skech-air-element-2.0---rising-gaze/149611.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-air---twirl/124073.html",
"https://www.skechers.com/women/shoes/gowalk-air---whirl/124074.html",
"https://www.skechers.com/women/shoes/skechers-arch-fit-refine---her-best/104092.html",
"https://www.skechers.com/women/shoes/skechers-arch-fit-refine/104090.html",
"https://www.skechers.com/women/shoes/glide-step---stepping-up/104086.html",
"https://www.skechers.com/women/shoes/relaxed-fit-empire-dlux---sharp-witted/149007.html",
"https://www.skechers.com/women/shoes/relaxed-fit-empire-dlux---paradise-sky/149274.html",
"https://www.skechers.com/women/shoes/skechers-gorun-smart/128230.html"]
for endpoint in url:
    options = webdriver.ChromeOptions()
    options.add_experimental_option("excludeSwitches", ['enable-automation'])
    options.add_argument('--disable-blink-features=AutomationControlled')
    options.add_argument(
        "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36")
    options.add_argument("--remote-debugging-port=9222")

    driver = webdriver.Chrome('F:/chromedriver.exe',options=options)
    driver.get(endpoint)
    time.sleep(5)
    image= None
    page_Source = driver.page_source
    soup = BeautifulSoup(page_Source, 'html.parser')
    content= soup.find_all('div',class_='c-product-description-features s-product-description-features row') 

    pid=soup.find('span',class_='product-id').text
    links = soup.find_all('img', srcset=True)
    for link in links:
        img= link['srcset']
        alt=link['alt']

    print(pid ,img,alt)

Answer 1

在 loop.I 之外定义 chrome 驱动程序实例还没有睾丸，但这应该可行。

options = webdriver.ChromeOptions()
options.add_experimental_option("excludeSwitches", ['enable-automation'])
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument(
        "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36")
options.add_argument("--remote-debugging-port=9222")

driver = webdriver.Chrome('F:/chromedriver.exe',options=options)


url = ["https://www.skechers.com/women/shoes/ultra-flex---twilight-twinkle/149173.html",
"https://www.skechers.com/women/shoes/ultra-flex---statements/12841.html",
"https://www.skechers.com/women/shoes/ultra-flex/12843.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-hyper-burst---extreme-outlook/124580.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-hyper-burst---lunar-mars/124577.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-hyper-burst/124575.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-stability---coco-jazz/124603.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-stability---magnificent-glow/124602.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-joy/124191.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-joy---paradise/15601.html",
"https://www.skechers.com/women/shoes/skech-air-element-2.0---vivid-blush/149408.html",
"https://www.skechers.com/women/shoes/skech-air-element-2.0---kaleidoscope-splash/149134.html",
"https://www.skechers.com/women/shoes/skech-air-element-2.0/149062.html",
"https://www.skechers.com/women/shoes/skech-air-element-2.0---rising-gaze/149611.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-air---twirl/124073.html",
"https://www.skechers.com/women/shoes/gowalk-air---whirl/124074.html",
"https://www.skechers.com/women/shoes/skechers-arch-fit-refine---her-best/104092.html",
"https://www.skechers.com/women/shoes/skechers-arch-fit-refine/104090.html",
"https://www.skechers.com/women/shoes/glide-step---stepping-up/104086.html",
"https://www.skechers.com/women/shoes/relaxed-fit-empire-dlux---sharp-witted/149007.html",
"https://www.skechers.com/women/shoes/relaxed-fit-empire-dlux---paradise-sky/149274.html",
"https://www.skechers.com/women/shoes/skechers-gorun-smart/128230.html"]
for endpoint in url:    
    driver.get(endpoint)
    time.sleep(5)
    image= None
    page_Source = driver.page_source
    soup = BeautifulSoup(page_Source, 'html.parser')
    content= soup.find_all('div',class_='c-product-description-features s-product-description-features row') 

    pid=soup.find('span',class_='product-id').text
    links = soup.find_all('img', srcset=True)
    for link in links:
        img= link['srcset']
        alt=link['alt']

    print(pid ,img,alt)

或在每个 url

完成后退出（）会话

url = ["https://www.skechers.com/women/shoes/ultra-flex---twilight-twinkle/149173.html",
"https://www.skechers.com/women/shoes/ultra-flex---statements/12841.html",
"https://www.skechers.com/women/shoes/ultra-flex/12843.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-hyper-burst---extreme-outlook/124580.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-hyper-burst---lunar-mars/124577.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-hyper-burst/124575.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-stability---coco-jazz/124603.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-stability---magnificent-glow/124602.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-joy/124191.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-joy---paradise/15601.html",
"https://www.skechers.com/women/shoes/skech-air-element-2.0---vivid-blush/149408.html",
"https://www.skechers.com/women/shoes/skech-air-element-2.0---kaleidoscope-splash/149134.html",
"https://www.skechers.com/women/shoes/skech-air-element-2.0/149062.html",
"https://www.skechers.com/women/shoes/skech-air-element-2.0---rising-gaze/149611.html",
"https://www.skechers.com/women/shoes/skechers-gowalk-air---twirl/124073.html",
"https://www.skechers.com/women/shoes/gowalk-air---whirl/124074.html",
"https://www.skechers.com/women/shoes/skechers-arch-fit-refine---her-best/104092.html",
"https://www.skechers.com/women/shoes/skechers-arch-fit-refine/104090.html",
"https://www.skechers.com/women/shoes/glide-step---stepping-up/104086.html",
"https://www.skechers.com/women/shoes/relaxed-fit-empire-dlux---sharp-witted/149007.html",
"https://www.skechers.com/women/shoes/relaxed-fit-empire-dlux---paradise-sky/149274.html",
"https://www.skechers.com/women/shoes/skechers-gorun-smart/128230.html"]
for endpoint in url:
    options = webdriver.ChromeOptions()
    options.add_experimental_option("excludeSwitches", ['enable-automation'])
    options.add_argument('--disable-blink-features=AutomationControlled')
    options.add_argument(
        "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36")
    options.add_argument("--remote-debugging-port=9222")

    driver = webdriver.Chrome('F:/chromedriver.exe',options=options)
    driver.get(endpoint)
    time.sleep(5)
    image= None
    page_Source = driver.page_source
    soup = BeautifulSoup(page_Source, 'html.parser')
    content= soup.find_all('div',class_='c-product-description-features s-product-description-features row') 

    pid=soup.find('span',class_='product-id').text
    links = soup.find_all('img', srcset=True)
    for link in links:
        img= link['srcset']
        alt=link['alt']

    print(pid ,img,alt)
    driver.quit()

Answer 2

import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys

import time

url = ["https://www.skechers.com/women/shoes/ultra-flex/12843.html",
    "https://www.skechers.com/women/shoes/ultra-flex---twilight-twinkle/149173.html",
       "https://www.skechers.com/women/shoes/ultra-flex---statements/12841.html",
       "https://www.skechers.com/women/shoes/ultra-flex/12843.html",
       "https://www.skechers.com/women/shoes/skechers-gowalk-hyper-burst---extreme-outlook/124580.html",
       "https://www.skechers.com/women/shoes/skechers-gowalk-hyper-burst---lunar-mars/124577.html",
       "https://www.skechers.com/women/shoes/skechers-gowalk-hyper-burst/124575.html",
       "https://www.skechers.com/women/shoes/skechers-gowalk-stability---coco-jazz/124603.html",
       "https://www.skechers.com/women/shoes/skechers-gowalk-stability---magnificent-glow/124602.html",
       "https://www.skechers.com/women/shoes/skechers-gowalk-joy/124191.html",
       "https://www.skechers.com/women/shoes/skechers-gowalk-joy---paradise/15601.html",
       "https://www.skechers.com/women/shoes/skech-air-element-2.0---vivid-blush/149408.html",
       "https://www.skechers.com/women/shoes/skech-air-element-2.0---kaleidoscope-splash/149134.html",
       "https://www.skechers.com/women/shoes/skech-air-element-2.0/149062.html",
       "https://www.skechers.com/women/shoes/skech-air-element-2.0---rising-gaze/149611.html",
       "https://www.skechers.com/women/shoes/skechers-gowalk-air---twirl/124073.html",
       "https://www.skechers.com/women/shoes/gowalk-air---whirl/124074.html",
       "https://www.skechers.com/women/shoes/skechers-arch-fit-refine---her-best/104092.html",
       "https://www.skechers.com/women/shoes/skechers-arch-fit-refine/104090.html",
       "https://www.skechers.com/women/shoes/glide-step---stepping-up/104086.html",
       "https://www.skechers.com/women/shoes/relaxed-fit-empire-dlux---sharp-witted/149007.html",
       "https://www.skechers.com/women/shoes/relaxed-fit-empire-dlux---paradise-sky/149274.html",
       "https://www.skechers.com/women/shoes/skechers-gorun-smart/128230.html"]
count=0
for endpoint in url:
    print(count)
    count+=1
    options = webdriver.ChromeOptions()
    options.add_experimental_option("excludeSwitches", ['enable-automation'])
    options.add_argument('--disable-blink-features=AutomationControlled')
    options.add_argument(
        "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36")
    #options.add_argument("--remote-debugging-port=9222")

    driver = webdriver.Chrome('./chromedriver.exe', options=options)
    driver.get(endpoint)

问题：

问题出在远程调试端口上，您已经在一个实例中使用了它，

解决方案：

删除那个

无法使用 Selenium 从多个 URL 打印数据 Python

unable to print data from multiple urls using Selenium Python

python

selenium

webdriver

beautifulsoup

问题：

解决方案：