lxml 网络抓取，特定词提取

Question

我正在使用我的脚本自动化来从 lan-website 上抓取计数器，我现在很着急。

代码如下所示

<TR><td><p align="left" style="margin-left: 30;"><b>title</b></p></td><td><p>   </p></td>
</TR>
<TR><td><p align="left" style="margin-left: 40;">table one</p></td><td><p> Task&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;average </p></td>
</TR>
<TR><td><p align="left" style="margin-left: 40;"></p></td><td><p> number&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;number </p></td>
</TR>
    <TR><td><p align="left" style="margin-left: 40;">1-1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;C</p></td><td><p> 6490&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1 </p></td>
    </TR>
    <TR><td><p align="left" style="margin-left: 40;">2-4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;C</p></td><td><p> 442&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;2 </p></td>
    </TR>
    <TR><td><p align="left" style="margin-left: 40;">5-10&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;C</p></td><td><p> 44&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6 </p></td>
    </TR>
    <TR><td><p align="left" style="margin-left: 40;">11-20&nbsp;&nbsp;&nbsp;&nbsp;C</p></td><td><p> 3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;15 </p></td>
    </TR>
    <TR><td><p align="left" style="margin-left: 40;">21-30&nbsp;&nbsp;&nbsp;&nbsp;C</p></td><td><p> 2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;25 </p></td>
    </TR>
    <TR><td><p align="left" style="margin-left: 40;">31-50&nbsp;&nbsp;&nbsp;&nbsp;C</p></td><td><p> 1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;40 </p></td>
    </TR>
    <TR><td><p align="left" style="margin-left: 40;">sum</p></td><td><p> 6982&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1 </p></td>
    </TR>

所以在每个站点中我都有重复的相同单词，如 1-2、2-4、5-10 等，我想提取数字 "below it"，如 6490、442 按照特定的顺序，所以它应该看起来像

task - counter
1-1 = 6490
2-4 = 442

为此我使用

import requests
from lxml import html

pageContent=requests.get(
 'http://x.html')
tree = html.fromstring(pageContent.content)
scraped = tree.xpath('//p/text()')
print scraped

显然打印出这样的东西 \xa0\xa0\xa0\xa0\xa0task',你'1-1\xa0\xa0\xa0\xa0\xa0\xa0counter',你'6490

我卡住了伙计们...尝试使用其他方法但我失败了。

Answer 1

试试这个。它将为您获取上面提到的确切输出。这里 content 是上面粘贴的 html 元素的容器。

from lxml.html import fromstring
root = fromstring(content)
for items in root.cssselect("tr")[3:]:
    data = [' '.join(item.text_content().split()).split(" ")[0] for item in items.cssselect("td")]
    print(' = '.join(data))

输出：

1-1 = 6490
2-4 = 442
5-10 = 44
11-20 = 3
21-30 = 2
31-50 = 1
sum = 6982

Answer 2

这应该可以正常工作。我为您的输出想出了一个 dict，您可以轻松地将其用于各种目的 -

text = """<TR><td><p align="left" style="margin-left: 30;"><b>title</b></p></td><td><p>   
</p></td>
</TR>
<TR><td><p align="left" style="margin-left: 40;">table one</p></td>
<td><p> Task&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;average </p></td>
</TR>
<TR><td><p align="left" style="margin-left: 40;"></p></td><td><p> number&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;number </p></td>
</TR>
    <TR><td><p align="left" style="margin-left: 40;">1-1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;C</p></td><td><p> 6490&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1 </p></td>
    </TR>
    <TR><td><p align="left" style="margin-left: 40;">2-4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;C</p></td><td><p> 442&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;2 </p></td>
    </TR>
    <TR><td><p align="left" style="margin-left: 40;">5-10&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;C</p></td><td><p> 44&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6 </p></td>
    </TR>
    <TR><td><p align="left" style="margin-left: 40;">11-20&nbsp;&nbsp;&nbsp;&nbsp;C</p></td><td><p> 3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;15 </p></td>
    </TR>
    <TR><td><p align="left" style="margin-left: 40;">21-30&nbsp;&nbsp;&nbsp;&nbsp;C</p></td><td><p> 2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;25 </p></td>
    </TR>
    <TR><td><p align="left" style="margin-left: 40;">31-50&nbsp;&nbsp;&nbsp;&nbsp;C</p></td><td><p> 1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;40 </p></td>
    </TR>
    <TR><td><p align="left" style="margin-left: 40;">sum</p></td><td><p> 6982&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1 </p></td>
    </TR>"""
from bs4 import BeautifulSoup

soup = BeautifulSoup(text, "lxml")
data = {}
for tr in soup.find_all('tr')[3:-1]:
    p = tr.find_all('td')
    task = p[0].text.split()[0].strip()
    counter = p[1].text.split()[0].strip()
    data[task] = counter
print(data)

输出

{'1-1': '6490', '2-4': '442', '5-10': '44', '11-20': '3', '21-30': '2', '31-50': '1'}

lxml 网络抓取，特定词提取

lxml web-scraping, specific word extraction

python

lxml

html-parsing

web-scraping