通过 Java Webapp 从查询中下载非常大的结果

Question

我一直在思考一个问题，想知道有没有人有什么建议。我有一个 Web 应用程序，我需要添加一个用于报告的部分。它将有一个基于用户 gui 输入构建的 t-sql 查询，执行并允许用户通过浏览器将文件作为 csv 下载到他们的计算机。就其本身而言 none 这太复杂了。我以前做过这些事情。然而，有一个问题我以前没有解决过。结果可能最终成为价值数 GB 的数据。我会做些什么（就代码而言）与正常情况不同，以防止它使浏览器过载或速度极慢？显然不能一次全部保存在内存中？我该如何防止这种情况发生？

Answer 1

您可以将 csv 数据直接写入传入请求的 OutputStream：

package example;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import java.io.IOException;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.nio.charset.StandardCharsets;

public class MyServlet extends HttpServlet {

    private final CSVCreator csvCreator;

    private MyServlet(CSVCreator csvCreator) {
        this.csvCreator = csvCreator;
    }

    @Override
    protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
        // be nice to your consumer, tell them what they get
        resp.setHeader("Content-Type", "text/comma-separated-values; charset=UTF-8");

        try (Writer writer = new OutputStreamWriter(resp.getOutputStream(), StandardCharsets.UTF_8)) {
            this.csvCreator.createCSV("some params you need to create the correct csv", writer);
        }
    }
}

您必须更改您的 csv 生成才能写入 Writer ... 例如：

package example;

import java.io.IOException;

public class CSVCreator {

    public void createCSV(String someParamsYouNeedToCreateTheCSV, Appendable appendable) throws IOException {
        appendable.append("Column1,Column2,Column3\n");
        appendable.append("Row1,Row1,Row1\n");
        appendable.append("Row2,Row2,Row2\n");
    }
}

例如，您还可以使用 apache-commons-csv 将 csv 写入给定的 Appendable（参见：https://javadoc.io/doc/org.apache.commons/commons-csv/latest/org/apache/commons/csv/CSVPrinter.html）

package example;

import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVPrinter;

import java.io.IOException;

public class CSVCreator {

    public void createCSV(String someParamsYouNeedToCreateTheCSV, Appendable appendable) throws IOException {
        try (CSVPrinter printer = new CSVPrinter(appendable, CSVFormat.DEFAULT)) {
            printer.printRecord("Column1", "Column2", "Column3");
            printer.printRecord("Row1", "Row1", "Row1");
            printer.printRecord("Row2", "Row2", "Row2");
        }
    }
}

编辑：

您可以像这样从结果集中创建 CSV 文件：

package example;

import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVPrinter;

import java.io.IOException;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;

public class ResultSetToCSV {

    public void createCSV(ResultSet rs, Appendable appendable) throws IOException, SQLException {
        final ResultSetMetaData metaData = rs.getMetaData();
        final int columnCount = metaData.getColumnCount();

        try (CSVPrinter printer = new CSVPrinter(appendable, CSVFormat.DEFAULT)) {
            printHeaders(metaData, columnCount, printer);
            printValues(rs, columnCount, printer);
        }
    }

    private void printHeaders(ResultSetMetaData metaData, int columnCount, CSVPrinter printer) throws IOException, SQLException {
        for (int i = 1; i <= columnCount; i++) {
            printer.print(metaData.getColumnLabel(i));
        }

        printer.println();
    }

    private void printValues(ResultSet rs, int columnCount, CSVPrinter printer) throws IOException, SQLException {
        while (rs.next()) {
            printValue(rs, columnCount, printer);
        }
    }

    private void printValue(ResultSet rs, int columnCount, CSVPrinter printer) throws IOException, SQLException {
        for (int i = 1; i <= columnCount; i++) {
            printer.print(rs.getString(i));
        }

        printer.println();
    }
}

通过 Java Webapp 从查询中下载非常大的结果

Downloading very large results from query via Java Webapp

java

performance

servlets