Pandas 只写入 CSV 文件的最后一行

Question

我正在从 txt 文件中抓取 urls 并将其导出到 csv 文件。但在所有过程之后，我的代码只写入了最后 url 的信息。我的猜测是我忘记了一个循环。但是哪里？这是我的代码：

import requests
from bs4 import BeautifulSoup
import pandas as pd
from urllib import urlopen

file = open('urls.txt', 'r')
filelines = (line.strip() for line in file)
for code in filelines:
    site = urlopen(code)
    soup = BeautifulSoup(site, "html.parser")
    final = soup.find_all("span", {"class": "bd js-title-main-info"})
    print final

records = []
for pagetxt in final:
    print pagetxt.text
    records.append((pagetxt.text))
df = pd.DataFrame(records, columns=['product name'])  
df.to_csv('test.csv', index=False, encoding='utf-8')

谢谢

Answer 1

当您从文件中获取数据时，您只会在变量 final 中保留最后一个值。尝试更早地附加数据（我用 ##### 标记了更改）：

import requests
from bs4 import BeautifulSoup
import pandas as pd
from urllib import urlopen

file = open('urls.txt', 'r')
filelines = (line.strip() for line in file)
records = []                         ######
for code in filelines:
    site = urlopen(code)
    soup = BeautifulSoup(site, "html.parser")
    final = soup.find_all("span", {"class": "bd js-title-main-info"})
    print final

    for pagetxt in final:               ######
       print pagetxt.text               ######
       records.append((pagetxt.text))   ######   

df = pd.DataFrame(records, columns=['product name'])  
df.to_csv('test.csv', index=False, encoding='utf-8')

Pandas 只写入 CSV 文件的最后一行

Pandas writes only the last line in a CSV File

dataframe

web-scraping

python-2.7

export-to-csv

python-requests