BeautifulSoup不returns所有数据

Question

我正在尝试使用 Python 的库 BeautifulSoup 解析一些月相数据。

from bs4 import BeautifulSoup
import urllib2

moon_url = "http://www.moongiant.com/phase/today/"


try:
    rqest =  urllib2.urlopen(moon_url)
    moon_Soup = BeautifulSoup(rqest, 'lxml')
    moon_angle = 0
    moon_illumination = 0
    main_data = moon_Soup.find('div', {'id' : 'moonDetails'})
    print main_data

except urllib2.URLError:
    print "Error"

但输出不是这样的：

<div id="moonDetails">        
      Phase: <span>Waxing Crescent</span><br>Illumination: <span>36%
</span><br>Moon Age: <span>6.00 days</span><br>Moon Angle: <span>0.55</span><br>Moon Distance: <span>364,</span>434.78 km<br>Sun Angle: <span>0.53</span><br>Sun Distance: <span>149,</span>571,918.47 km<br>
</div>

只有这个：

<div id="moonDetails">
</div>

有什么想法吗？

Answer 1

如 RaminNietzsche 在评论中所述，您应该在此特定 script 标记中提取脚本文本。您可以使用 regex 或 built-in methods（例如 split()、strip() 和 replace()。

代码：

from bs4 import BeautifulSoup
import requests
import re
import json

moon_url = "http://www.moongiant.com/phase/today/"
html_source =  requests.get(moon_url).text

moon_soup = BeautifulSoup(html_source, 'html.parser')

data = moon_soup.find_all('script', {'type' : 'text/javascript'})

for d in data:
    d = d.text
    if 'var jArray=' in d:
        jArray = re.search('\{(.*?)\}', d).group()
        moon_data = json.loads(jArray)
        print(moon_data)

        #if you want mArray data too, you just have to:
        # 1. add `'var mArray=' in d` in the if clause, and
        # 2. uncomment the following lines
        #mArray = re.search('\[+(.*?)\];', d).group()
        #print(mArray)

输出：

{'3': ['<b>April 4</b>', '58%\n', 'Sun Angle: 0.53291621763825', 'Sun Distance: 149657950.85286', 'Moon Distance: 369697.55153449', 'Moon Age: 8.1316595947356', 'Moon Angle: 0.53870564539409', 'Waxing Gibbous', 'April 4'], '2': ["<span style='color:#c7b699'><b>April 3</b></span>", 'Illumination: <span>47%\n</span>', 'Sun Angle: <span>0.53', 'Sun Distance: <span>149,</span>614,</span>943.28', 'Moon Distance: <span>366,</span>585.35', 'Moon Age: <span>7.08', 'Moon Angle: <span>0.54', 'First Quarter', '<b>Monday, April 3, 2017</b>', 'April', 'Phase: <span>First Quarter</span>', 'April 3'], '1': ['<b>April 2</b>', '36%\n', 'Sun Angle: 0.53322274612254', 'Sun Distance: 149571918.46739', 'Moon Distance: 364434.77975454', 'Moon Age: 6.002888839693', 'Moon Angle: 0.54648504798072', 'Waxing Crescent', 'April 2'], '4': ['<b>April 5</b>', '69%\n', 'Sun Angle: 0.53276322269153', 'Sun Distance: 149700928.5008', 'Moon Distance: 373577.14506795', 'Moon Age: 9.1657967733025', 'Moon Angle: 0.53311119464703', 'Waxing Gibbous', 'April 5'], '0': ['<b>April 1</b>', '25%\n', 'Sun Angle: 0.53337618944887', 'Sun Distance: 149528889.15122', 'Moon Distance: 363387.67496992', 'Moon Age: 4.9078487808877', 'Moon Angle: 0.54805974945761', 'Waxing Crescent', 'April 1']}

由于它是作为 JSON 加载的，您可以像这样浏览它：

示例代码：

print(moon_data['4'])
print('-')*5
print(moon_data['4'][2])

输出：

['<b>April 5</b>', '69%\n', 'Sun Angle: 0.53276322269153', 'Sun Distance: 149700928.5008', 'Moon Distance: 373577.14506795', 'Moon Age: 9.1657967733025', 'Moon Angle: 0.53311119464703', 'Waxing Gibbous', 'April 5']
-----
Sun Angle: 0.53276322269153

Answer 2

我从 access Chrome DOM.

的 root 回答中抄袭了其要点的另一种方式

我们的想法是，您可以同时使用 selenium 和 lxml 来访问页面的 DOM已由其 javascript.

加载和处理

>>> moon_url = "http://www.moongiant.com/phase/today/"
>>> import selenium.webdriver as webdriver
>>> import lxml.html as html
>>> import lxml.html.clean as clean
>>> 
>>> browser = webdriver.Chrome()
>>> browser.get(moon_url)
>>> content = browser.page_source
>>> cleaner = clean.Cleaner()
>>> content = cleaner.clean_html(content)
>>> doc = html.fromstring(content)
>>> type(doc)
<class 'lxml.html.HtmlElement'>
>>> type(content)
<class 'str'>
>>> open('c:/scratch/content.htm','w').write(content)
27070

完成后，如上面最后几条语句所示，您可以将 DOM either/both 作为 HTML 或作为适合使用 lxml 处理的树来访问。在你的情况下，你可能更喜欢用 HTML 做汤；这意味着将 BeautifulSoup 应用到 content.

顺便说一句，当我保存 content 时，我确实在 HTML 中找到了以下结构，正如人们所期望的那样。

<div id="moonDetails">
    Phase: <span>First Quarter</span><br>
    Illumination: <span>47%</span><br>
    Moon Age: <span>7.08 days</span><br>
    Moon Angle: <span>0.54</span><br>
    Moon Distance: <span>366,</span>585.35 km<br>
    Sun Angle: <span>0.53</span><br>
    Sun Distance: <span>149,</span>614,943.28 km<br>
</div>

Answer 3

实际上在 RaminNietzsche 的评论之后我使用了 dryscrape 库。

from bs4 import BeautifulSoup
import urllib2
import dryscrape

    moon_url = "http://www.moongiant.com/phase/today/"

try:
    rqest =  urllib2.urlopen(moon_url)
    session = dryscrape.Session()
    session.visit(moon_url)
    response = session.body()
    soup = BeautifulSoup(response, 'lxml')

    moon_data = soup.findAll('div', {'id':'moonDetails'})
    print moon_data

因此现在的输出是：

<div id="moonDetails">        
      Phase: <span>Waxing Crescent</span><br>Illumination: <span>36%
</span><br>Moon Age: <span>6.00 days</span><br>Moon Angle: <span>0.55</span><br>Moon Distance: <span>364,</span>434.78 km<br>Sun Angle: <span>0.53</span><br>Sun Distance: <span>149,</span>571,918.47 km<br>
</div>

谢谢大家的解答！

BeautifulSoup不returns所有数据

BeautifulSoup does not returns all data

python

parsing

beautifulsoup

html-parsing