如何在 HTMLUnit 中从 Javascript Link 下载文件

Question

正如标题所说，我正在尝试从 javascript link.

下载带有 HTMLUnit 的文件

我开始的页面是 https://ppair.uspto.gov/TruePassWebStart/AuthenticationChooser.html。当我在浏览器中单击 link "Authenticate with Java Web Start (new method)" 时，将下载一个 .jnlp 文件，然后运行该文件以打开一个 Java 程序 window，该程序要求提供身份验证信息。身份验证成功后，原始浏览器 window 会加载包含我将要抓取的信息的页面。

起始页的 link 源代码片段是：

<tr>
<!-- onClick="return launchWebStart('authenticate');" -->
    <td><a href="javascript:void(0)" id="webstart-authenticate" ><font size="5">Authenticate with Java Web Start (new method)</font></a>
</tr>

用于此过程的 javascript 文件位于 https://ppair.uspto.gov/TruePassWebStart/js/WebStart.js。基本上，javascript 获取一个 cookie，对其进行编码，并将其附加到 URL 以请求 jnlp 文件。我考虑过只模拟这个过程，但 HTMLUnit 文档不鼓励这样做。（他们说 "It is much better to manipulate the page as a user would by clicking on elements and shifting the focus around" 所以我正在尝试这样做。）

我在 HTMLUnit 中遇到的问题是，在我单击 () 适当的锚点后，我无法接收预期的 jnlp 文件。我尝试了从本网站其他问题中发现的几种不同的方法，包括：

HtmlUnit and JavaScript in links 和 HtmlUnit to invoke javascript from href to download a file

这是我使用的代码：

import java.io.IOException;
import java.io.InputStream;
import java.net.MalformedURLException;

import com.gargoylesoftware.htmlunit.*;
import com.gargoylesoftware.htmlunit.html.HtmlAnchor;
import com.gargoylesoftware.htmlunit.html.HtmlPage;

public class Test {

    public static void main(String[] args) throws FailingHttpStatusCodeException, MalformedURLException, IOException {
        WebClient webClient = new WebClient(BrowserVersion.FIREFOX_45);

        // open starting webpage
        HtmlPage page = webClient.getPage("https://ppair.uspto.gov/TruePassWebStart/AuthenticationChooser.html");

        // id of the element where the link is
        String linkID = "webstart-authenticate";

        // identify the appropriate anchor
        HtmlAnchor anchor = (HtmlAnchor) page.getElementById(linkID);

        // click the anchor
        Page p = anchor.click();

        // get the InputStream for the response; print it out
        InputStream is = p.getWebResponse().getContentAsStream();
        int b = 0;
        while ((b = is.read()) != -1) {
            System.out.print((char)b);
        }
        webClient.close();
    }
}

从上面的代码打印出来的是起始网页的 html 而不是预期的 jnlp 文件。控制台还每 3 秒打印一次来自 javascript WebConsole 的状态更新（至少如果我让代码等待足够长的时间），所以我知道 javascript（函数 launchWebStart 和followMediator 在单独的 javascript 文件中 WebStart.js):

Nov 21, 2016 2:53:25 PM com.gargoylesoftware.htmlunit.WebConsole info
INFO: launchWebStart

Nov 21, 2016 2:53:25 PM com.gargoylesoftware.htmlunit.WebConsole info
INFO: followMediator

Nov 21, 2016 2:53:25 PM com.gargoylesoftware.htmlunit.WebConsole info
INFO: responseReceived:200
WAIT

Nov 21, 2016 2:53:25 PM com.gargoylesoftware.htmlunit.WebConsole info
INFO: mediatorCallback: next wait

我还尝试使用 CollectingAttachmentHandler object，如 downloading files behind javascript button with htmlunit 所述：

import java.io.IOException;
import java.net.MalformedURLException;
import java.util.List;

import com.gargoylesoftware.htmlunit.*;
import com.gargoylesoftware.htmlunit.attachment.Attachment;
import com.gargoylesoftware.htmlunit.attachment.CollectingAttachmentHandler;
import com.gargoylesoftware.htmlunit.html.HtmlAnchor;
import com.gargoylesoftware.htmlunit.html.HtmlPage;

public class Test2 {

    public static void main(String[] args) throws FailingHttpStatusCodeException, MalformedURLException, IOException {
        WebClient webClient = new WebClient(BrowserVersion.FIREFOX_45);

        // open starting webpage
        HtmlPage page = webClient.getPage("https://ppair.uspto.gov/TruePassWebStart/AuthenticationChooser.html");

        // id of the element where the link is
        String linkID = "webstart-authenticate";

        // identify the appropriate anchor
        HtmlAnchor anchor = (HtmlAnchor) page.getElementById(linkID);

        CollectingAttachmentHandler attachmentHandler = new CollectingAttachmentHandler();
        webClient.setAttachmentHandler(attachmentHandler);
        attachmentHandler.handleAttachment(anchor.click());
        List<Attachment> attachments = attachmentHandler.getCollectedAttachments();

        int i = 0;
        while (i < attachments.size()) {
            Attachment attachment = attachments.get(i);
            Page attachedPage = attachment.getPage();
            WebResponse attachmentResponse = attachedPage.getWebResponse();
            String content = attachmentResponse.getContentAsString();
            System.out.println(content);
            i++;
        }
        webClient.close();
    }
}

此代码还打印出起始网页的内容。所以 none 的其他解决方案似乎对我有用。我不知道我做错了什么。我运行不知道如何让它工作（我认为这很容易！）非常感谢任何建议！

Answer 1

这是一个基于你的 Test2 的有效版本

    WebClient webClient = new WebClient(BrowserVersion.FIREFOX_45);

    // open starting webpage
    HtmlPage page = webClient.getPage("https://ppair.uspto.gov/TruePassWebStart/AuthenticationChooser.html");

    // id of the element where the link is
    String linkID = "webstart-authenticate";

    // identify the appropriate anchor
    HtmlAnchor anchor = (HtmlAnchor) page.getElementById(linkID);

    CountDownLatch latch = new CountDownLatch(1);
    webClient.setWebStartHandler(new WebStartHandler(){

        @Override
        public void handleJnlpResponse(WebResponse webResponse)
        {
            System.out.println("downloading...");
            try (FileOutputStream fos = new FileOutputStream("/Users/Franklyn/Downloads/uspto-auth.authenticate2.jnlp"))
            {
                IOUtils.copy(webResponse.getContentAsStream(),fos);
            } catch (IOException e)
            {
                throw new RuntimeException(e);
            }
            System.out.println("downloaded");
            latch.countDown();
        }
    });

    anchor.click();
    latch.await();//wait downloading to finish

    webClient.close();

那么为什么你的 Test2 不工作？因为响应Content-Type对应的下载文件是application/x-java-jnlp-file，所以你需要改用WebStartHandler。如果响应 headers 包含一个名为 'Content-Disposition' 的 header 并且它的值以 'attachment 开头',那么您的 Test2 可能会正常工作。

如何在 HTMLUnit 中从 Javascript Link 下载文件

How to Download File from Javascript Link in HTMLUnit

javascript

java

htmlunit