使用不更改的选项卡抓取网页 url

Question

我正在尝试抓取纳斯达克网页，但在定位元素时遇到一些问题：

我的代码：

from selenium import webdriver
import time
import pandas as pd

driver.get('http://www.nasdaqomxnordic.com/shares/microsite?Instrument=CSE32679&symbol=ALK%20B&name=ALK-Abell%C3%B3%20B')

time.sleep(5)
btn_overview = driver.find_element_by_xpath('//*[@id="tabarea"]/section/nav/ul/li[2]/a')
btn_overview.click()
time.sleep(5) 
employees = driver.find_element_by_xpath('//*[@id="CompanyProfile"]/div[6]')

最后一次调用后，我收到以下错误：

NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="CompanyProfile"]/div[6]"}

一般情况下问题是错误的'xpath'，但我试了几个项目，也是'id'。我怀疑它与选项卡有关（在我的例子中导航到“概述”）。从视觉上看，网页发生了变化，但例如，如果我抓取 table，它会从第一页获取：

table_test = pd.read_html(driver.page_source)[0]

我错过了什么或做错了什么？

Answer 1

overview 页面在 iframe 下

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

from selenium.webdriver.chrome.options import Options

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC


option = webdriver.ChromeOptions()
option.add_argument("start-maximized")

#chrome to stay open
option.add_experimental_option("detach", True)

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=option)
driver.get('http://www.nasdaqomxnordic.com/shares/microsite?Instrument=CSE32679&symbol=ALK%20B&name=ALK-Abell%C3%B3%20B')


WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="tabarea"]/section/nav/ul/li[2]/a'))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="cookieConsentOK"]'))).click()

WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe#MorningstarIFrame")))
employees=WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, '//*[@id="CompanyProfile"]/div[6]'))).text.split()[1]
print(employees)

输出：

2,537

webdriverManager

Answer 2

您确定需要 Selenium 吗？

import requests
from bs4 import BeautifulSoup

url = 'http://lt.morningstar.com/gj8uge2g9k/stockreport/default.aspx'
payload = {
'SecurityToken': '0P0000A5LL]3]1]E0EXG$XCSE_3060'}

response = requests.get(url, params=payload)
soup = BeautifulSoup(response.text, 'html.parser')

employees = soup.find('h3', text='Employees').next_sibling.text
print(employees)

输出：

2,537

使用不更改的选项卡抓取网页 url

Scraping webpage with tabs that do not change url

python

selenium

web-scraping