使用 cheerio 抓取网页

Question

我正在尝试使用 cheerio 抓取网页，但我的请求被某种机器人检测软件阻止了。响应如下所示：

<body>
<div class="content">
  ...
  <div class="main">
    <h1>Suspicious activity detected</h1>
 <div class="box">
   <p>Due to suspicious activity from your computer, we have   blocked your access to http://www.something.com. After completing the form, your information will be evaulauted and you may be unblocked.  </p>
      <p><span>Note</span>: This website may require JavaScript. If you have JavaScript disabled, please enable it before attempting to return to http://www.something.com.</p>
    <div class="block">
  <form id="distilUnblockForm" method="post" action="http://verify.distil.it/distil_blocked.php">
 <div id="dUF_first_name">
  <label for="dUF_input_first_name">First Name:</label>
  <input type="text" id="dUF_input_first_name" name="first_name" value="" />
 </div>
 <div id="dUF_last_name">
  <label for="dUF_input_last_name">Last Name:</label>
  <input type="text" id="dUF_input_last_name" name="last_name" value="" />
 </div>
 <div id="dUF_email">
  <label for="dUF_input_email">E-mail:</label>
  <input type="text" id="dUF_input_email" name="email" value="" />
 </div>
 <div id="dUF_city" style="display: none">
  <label for="dUF_input_city">City (Leave Blank):</label>
  <input type="text" id="dUF_input_city" name="city" value="" />
 </div>
 <div id="dUF_unblock">
  <input  id="dUF_input_unblock" name="unblock" type="submit" value="Request Unblock" />
 </div>
 <div id="dUF_unblock_text">
  You reached this page when attempting to access https://someWebsite from myIPAddress on someDateInISOFormat.
 </div>
 <div id="dUF_form_fields" style="display: none">
  <input type="hidden" name="B" value="someNumbersAndLetters" />         
  <input type="hidden" name="P" value="someMoreNumbersAndLetters" />         
  <input type="hidden" name="I" value="" />         
  <input type="hidden" name="U" value="" />         
  <input type="hidden" name="V" value="###" />
  <input type="hidden" name="O" value="" />
  <input type="hidden" name="D" value="###" />
  <input type="hidden" name="A" value="###" />
  <input type="hidden" name="LOADED" value="someDate" />
  <input type="hidden" name="Q" value='someUrl' />
  <input type="hidden" id="distil_block_identity_info" name="XX" value="" />
 </div>
</form>

...

</body>

我想我可以通过添加一个带有 post 函数的回调来解决这个问题，但它似乎不起作用。我的代码如下：

var url = someUrl;

request(url, function (error, response, html) {
 console.log("html", html); //where i am getting the above html
  
  request.post({
    uri: 'hhttp://verify.distil.it/distil_blocked.php',
 headers: { 'content-type': 'application/x-www-form-urlencoded' },
 body: require('querystring').stringify(credentials)
 }, function(err, res, body){
   if(err) {
   callback.call(null, new Error('Login failed'));
   return;
   } else {
        var $ = cheerio.load(html);
        var parsedResults = [];
        $('#someSelector').each(function(i, element){
          var node = $(this).children();
          // Get all the children
          var something = $(node).eq(0).text();
          var anotherThing = $(node).eq(1).text();
          var oneMoreThing = $(node).eq(2).text();
          // Make it into an object
          var metadata = {
         something: something,
         anotherThing: anotherThing,
           oneMoreThing: oneMoreThing
          };
          // Push meta-data into parsedResults array
          parsedResults.push(metadata);
        });
        // Log our finished parse results in the terminal
        console.log(parsedResults);
   }
  });
});

这在 post 请求上失败了，我不确定是因为我回调错误还是 post 请求不是绕过机器人的有效方法.非常感谢任何帮助，谢谢。

Answer 1

您应该尽可能将 header 设置为 browser-like。例如，您没有设置 user-agent、cache-control、accept 等。这使得网站很容易检测到您不是浏览器。

根据正常请求检查 header（在 Chrome 使用更多工具-> 开发人员工具、网络和 select HTML 文件。更多信息 on this thread) 并尽可能发送类似的邮件。

Answer 2

我意识到我遇到了问题，因为 cheerio 不加载 javacsript。我尝试抓取的网站使用 javacsript 加载数据，因此我需要使用不同的工具。可能会使用 (PhantomJS)[http://phantomjs.org/].

使用 cheerio 抓取网页

Scraping a webpage with cheerio

javascript

screen-scraping

node.js

web-scraping