在 Java 中并行或异步下载多个文件

Question

这里我尝试一个接一个地下载多个文件：

环境 - Java1.6

public List<Attachment> download(List<Attachment> attachments)
{
  for(Attachment attachment : attachments) {
    attachment.setDownStatus("Failed");
    String destLocation = "C:\Users\attachments";
    try {
        String attUrl = attachment.getUrl();
        String fileName = attachment.getFileName();            
        URL url = new URL(attUrl);
        File fileLocation = new File(destLoc, fileName);
        FileUtils.copyURLToFile(url, fileLocation);
        if(fileLocation.exists()) {
           attachment.setDownStatus("Completed");
         }
       } catch(Exception e) {
          attachment.setDownStatus("Failed");
       } finally {
          attachment.setDestLocation(destLocation);
       }
   }
  return attachments;
}

我正在从提供的 URL (http://cdn.octafinance.com/wp-content/uploads/2015/07/google-hummingbird.jpg) 下载文件。

FileUtils.copyURLToFile(url, fileLocation);

以上代码完美地完成了下载工作，没有任何问题。

我的问题：
如果附件列表更多，则需要更多时间，所以我想将其设为异步或并行进程，而不是顺序下载。

Answer 1

public List<Attachment> download(List<Attachment> attachments)
{
  ExecutorService executorService = Executors.newCachedThreadPool();
  for(final Attachment attachment : attachments){
    executorService.submit(new Runnable() {

        @Override
        public void run() {
          try{
            String attUrl = attachment.getUrl();
            String fileName = attachment.getFileName();
            String destLocation = "C:\Users\attachments";
            URL url = new URL(attUrl);
            String fileLocation = new File(destLoc, fileName);
            FileUtils.copyURLToFile(url, fileLocation);
            if(fileLocation.exists()) {
              attachment.setDownStatus("Completed");
            }
          }
          catch(Exception e){
            attachment.setDownStatus("Failed");
          }
        }
    });
 }
 executorService.shutdown();
 return attachments;
}

Answer 2

结合使用 Java 8 个流和 ForkJoinPool

public List<Attachment> download(List<Attachment> attachments) throws InterruptedException, ExecutionException {

    ForkJoinPool forkJoinPool = new ForkJoinPool(attachments.size());

    return forkJoinPool.submit(() -> processAttachments(attachments)).get();
}

private List<Attachment> processAttachments(List<Attachment> attachments) {
    return attachments.stream().parallel().map(attachment -> processSingleAttachment(attachment)).collect(Collectors.toList());
}

private Attachment processSingleAttachment(Attachment attachment){
     //business logic to download single attachment
    .
    .
}

Answer 3

其实仔细看了看，Boris的代码是有问题的，确实有时候不会设置一些东西。这是修复该问题的更好版本：

public List<Attachment> download(List<Attachment> attachments) {
  ExecutorService executorService = Executors.newCachedThreadPool();
  List<Future<Attachment>> futures = new ArrayList<Future<Attachment>>();
  for (final Attachment attachment : attachments) {
    futures.add(executorService.submit(new Callable<Attachment>() {
      @Override
      public Attachment call() throws Exception {
        return doDownload(attachment);
      }
    }));
  }
  for (Future<Attachment> future: futures) {
    try {
      future.get();
    } catch (Exception ex) {
      // Do something
    }
  }
  return attachments;
}

private Attachment doDownload(Attachment attachment) throws Exception {
  attachment.setDownStatus("Failed");
  attachment.setDestLocation("C:\Users\attachments");
  String attUrl = attachment.getUrl();
  String fileName = attachment.getFileName();
  URL url = new URL(attUrl);
  File fileLocation = new File(attachment.getDestLocation(), fileName);
  FileUtils.copyURLToFile(url, fileLocation);
  if (fileLocation.exists()) {
    attachment.setDownStatus("Completed");
  }
  return attachment;
}

但是，考虑到您 Attachment 的结构及其使用方式，这绝对不是最佳选择。我没有解决这个问题：我只回答了问题。

在 Java 中并行或异步下载多个文件

Downloading Multiple Files Parallelly or Asynchronously in Java

java

multithreading

asynchronous

file

java-6