从网页中提取单个 href

Question

我正在编写一个代码，其中我必须提取一个 href link，我面临的问题是它提取了两个 links，除了最后一个 ID 部分，我有一个 ID，我只想从 link 中提取另一个 ID。这是我的代码：-

    import requests,re
    from bs4 import BeautifulSoup
    url="http://www.barneys.com/band-of-outsiders-oxford-sport-shirt-500758921.html"
    r=requests.get(url)
    soup=BeautifulSoup(r.content)
    g_1=soup.find_all("div",{"class":"color-scroll"})
    for item in g_1:
          a_1=soup.find_all('a', href=re.compile('^/on/demandware.store/Sites-BNY-Site/default/Product-Variation'))
          for elem in a_1:
               print elem['href']

我得到的输出是：-

         /on/demandware.store/Sites-BNY-Site/default/Product-Variation?pid=500758921
         /on/demandware.store/Sites-BNY-Site/default/Product-Variation?pid=500758910

我有第一个ID，即500758921，我想提取另一个。请帮忙。提前致谢！

Answer 1

运行每个 link

这个正则表达式

^/on/demandware.store/Sites-BNY-Site/default/Product-Variation\?pid=([0-9]+)

获取最后一个正则表达式组的结果。

Answer 2

如果您需要除第一个以外的所有 link，只需将 find_all() 的结果切片：

links = soup.find_all('a', href=re.compile('^/on/demandware.store/Sites-BNY-Site/default/Product-Variation'))
for link in links[1:]:
    print link['href']

切片起作用的原因是 find_all() returns 一个 ResultSet 内部基于常规 Python 列表的实例：

class ResultSet(list):
    """A ResultSet is just a list that keeps track of the SoupStrainer
    that created it."""
    def __init__(self, source, result=()):
        super(ResultSet, self).__init__(result)
        self.source = source

要从您获得的 link 中提取 pid，您可以使用正则表达式搜索将 pid 值保存在 捕获组中:

import re

pattern = re.compile("pid=(\w+)")
for item in g_1:
    links = soup.find_all('a', href=re.compile('^/on/demandware.store/Sites-BNY-Site/default/Product-Variation'))
    for link in links[1:]:
        match = pattern.search(link["href"])
        if match:
            print match.group(1)

Answer 3

这可能会：

import requests,re
from bs4 import BeautifulSoup
def getPID(url):
    return re.findall('(\d+)',url.rstrip('.html'))
url="http://www.barneys.com/band-of-outsiders-oxford-sport-shirt-500758921.html"
having_pid = getPID(url)
print(having_pid)
r=requests.get(url)
soup=BeautifulSoup(r.content)
g_1=soup.find_all("div",{"class":"color-scroll"})
for item in g_1:
    a_1=soup.find_all('a', href=re.compile('^/on/demandware.store/Sites-BNY-Site/default/Product-Variation'))
    for elem in a_1:
        if (getPID(elem['href'])[0] not in having_pid):
           print elem['href']

从网页中提取单个 href

Extract single href from a web page

python

regex

beautifulsoup

web-scraping