只有最后一页的输出被写入 CSV

Question

我正在尝试从页面中提取链接，但只提取了最后一页的链接。我如何在浏览页面时提取所有内容？

for var in range(1, 1001):
    page = driver.find_element_by_xpath('//a[contains(@href,"pageNbre=%r")]' % var)
    driver.execute_script("arguments[0].click();", page)
    print('Navigating to page %r ' % var)
    time.sleep(3)

elem = driver.find_elements_by_xpath('//*[contains(@href, "/c/")]')
url_list = []
for link in elem:
    print(link.get_attribute('href'))
    url_list.append(link.get_attribute('href'))

df = pd.DataFrame(url_list,columns=['url'])
df.to_csv('C://users//admin//desktop//urls.csv', index=False)

Answer 1

I'm failing to understand how it should look like indented on one level.

Jonah，看看应该如何：

for var in range(1, 1001):
    page = driver.find_element_by_xpath('//a[contains(@href,"pageNbre=%r")]' % var)
    driver.execute_script("arguments[0].click();", page)
    print('Navigating to page %r ' % var)
    time.sleep(3)

    elem = driver.find_elements_by_xpath('//*[contains(@href, "/c/")]')
    url_list = []
    for link in elem:
        print(link.get_attribute('href'))
        url_list.append(link.get_attribute('href'))

df = pd.DataFrame(url_list,columns=['url'])
df.to_csv('C://users//admin//desktop//urls.csv', index=False)

只有最后一页的输出被写入 CSV

Only last page's output gets written down to CSV

python

csv

selenium

screen-scraping

output