java.lang.OutOfMemoryError: Java heap space during bytea download

Question

我正在使用此代码从 PostgreSQL 下载 bytea 对象：

public void initFileDBData() throws SQLException, IOException
{
    if (ds == null)
    {
        throw new SQLException("Can't get data source");
    }
    Connection conn = ds.getConnection();

    if (conn == null)
    {
        throw new SQLException("Can't get database connection");
    }

    PreparedStatement ps = null;

    try
    {
        conn.setAutoCommit(false);
        ps = conn.prepareStatement("SELECT * FROM PROCEDURE_FILES WHERE ID = ?");

        ps.setInt(1, id);
        ResultSet rs = ps.executeQuery();
        while (rs.next())
        {
            String file_name = rs.getString("FILE_NAME");
            InputStream binaryStreasm = rs.getBinaryStream("FILE");
            FacesContext fc = FacesContext.getCurrentInstance();
            ExternalContext ec = fc.getExternalContext();

            ec.responseReset();
            ec.setResponseContentLength(binaryStreasm.available());
            ec.setResponseHeader("Content-Disposition", "attachment; filename=\"" + file_name + "\"");

            byte[] buf;

                buf = new byte[binaryStreasm.available()];
                int offset = 0;
                int numRead = 0;
                while ((offset < buf.length) && ((numRead = binaryStreasm.read(buf, offset, buf.length - offset)) >= 0))
                {
                    offset += numRead;
                }

            HttpServletResponse response
                = (HttpServletResponse) FacesContext.getCurrentInstance()
                .getExternalContext().getResponse();

            response.setContentType("application/octet-stream");
            response.setHeader("Content-Disposition", "attachment;filename=" + file_name);
            response.getOutputStream().write(buf);
            response.getOutputStream().flush();
            response.getOutputStream().close();
            FacesContext.getCurrentInstance().responseComplete();
        }

    }
    finally
    {
        if (ps != null)
        {
            ps.close();
        }
        conn.close();
    }
}

但是当我开始下载代码时，我在这一行得到 java.lang.OutOfMemoryError: Java heap space:

 buf = new byte[binaryStreasm.available()];

我能否以某种方式优化代码以消耗更少的内存？

更新代码：

public void initFileDBData() throws SQLException, IOException
    {
        if (ds == null)
        {
            throw new SQLException("Can't get data source");
        }
        Connection conn = ds.getConnection();

        if (conn == null)
        {
            throw new SQLException("Can't get database connection");
        }

        PreparedStatement ps = null;

        try
        {
            conn.setAutoCommit(false);
            ps = conn.prepareStatement("SELECT *, octet_length(FILE) as file_length FROM PROCEDURE_FILES WHERE ID = ?");

            ps.setInt(1, id);
            ResultSet rs = ps.executeQuery();
            while (rs.next())
            {
                String file_name = rs.getString("FILE_NAME");
                FacesContext fc = FacesContext.getCurrentInstance();
                ExternalContext ec = fc.getExternalContext();

                ec.responseReset();
                ec.setResponseContentLength(rs.getInt("file_length"));
                ec.setResponseHeader("Content-Disposition", "attachment; filename=\"" + file_name + "\"");

                HttpServletResponse response
                    = (HttpServletResponse) FacesContext.getCurrentInstance()
                    .getExternalContext().getResponse();

                byte[] buffer = new byte[4096];

                try (InputStream input = rs.getBinaryStream("FILE");
                    OutputStream output = response.getOutputStream())
                {
                    int numRead = 0;

                    while ((numRead = input.read(buffer)) != -1)
                    {
                        output.write(buffer, 0, numRead);
                    }
                }

                response.setContentType("application/octet-stream");
                response.setHeader("Content-Disposition", "attachment;filename=" + file_name);
                response.getOutputStream().write(buffer);
                response.getOutputStream().flush();
                response.getOutputStream().close();
                FacesContext.getCurrentInstance().responseComplete();
            }
        }
        finally
        {
            if (ps != null)
            {
                ps.close();
            }
            conn.close();
        }
    }

Answer 1

这可能是因为 binaryStreasm.available() returns 是一个很大的值，因此它会尝试创建一个无法放入内存的字节数组，请尝试设置较小的值，例如 512 或 1024.

这里的另一个问题是您尝试将 bytea 对象的全部内容加载到内存中，这不是正确的方法，尤其是当您必须像这里一样处理大的二进制内容时。你应该把内容写入response.getOutputStream()或者先写入一个临时文件，然后再把文件的内容写入response.getOutputStream()

Answer 2

您对 InputStream::available 的依赖是错误的。它的 documentation 表示：

Returns an estimate of the number of bytes that can be read (or skipped over) from this input stream without blocking by the next invocation of a method for this input stream. The next invocation might be the same thread or another thread. A single read or skip of this many bytes will not block, but may read or skip fewer bytes.

Note that while some implementations of InputStream will return the total number of bytes in the stream, many will not. It is never correct to use the return value of this method to allocate a buffer intended to hold all data in this stream.

（强调我的）

所以你面临两个问题：

如果您没有输出流的确切大小，要在 Content-Length header 中传递什么数字？
如何在不将所有数据同时保存在内存中的情况下将流从结果集传递到响应。

我会通过稍微更改查询来解决第一个问题，使其 returns bytea 字段的实际大小。

ps = conn.prepareStatement("SELECT *,octet_length(FILE) as file_length FROM PROCEDURE_FILES WHERE ID = ?");

PostgreSQL 函数 octet_length 给出 bytea 列的长度（以字节为单位）。

一旦你有了它，你就可以使用

ec.setResponseContentLength(rs.getInt("file_length"));

现在，对于第二个问题，您应该避免将所有内容都读入大缓冲区。如果您使用 rs.getInt("file_length') 中的数字分配缓冲区，您将运行陷入同样的内存问题。你应该逐渐复制流。

如果您有 Apache Commons IO，您可以使用 IOUtils.copy() 将流从结果集中复制到响应的输出流。在设置响应内容类型和长度之前避免获取二进制流，然后执行：

IOUtils.copy( rs.getBinaryStream("FILE"), response.getOutputStream() );

如果您不想使用 Apache Commons IO，您可以编写自己的循环 - 使用 small 缓冲区。同样，首先设置响应内容类型和长度，然后执行类似

的操作

byte[] buffer = new byte[4096];

try ( InputStream input = rs.getBinaryStream("FILE");
      OutputStream output = response.getOutputStream() ) {

    int numRead = 0;

    while ( ( numRead = input.read( buffer ) ) != -1 ) {
        output.write(buffer, 0, numRead );
    }

}

然后完成响应，关闭结果集，就大功告成了。我使用了 try-with-resources 语法，它会在完成后自动关闭流。

顺便说一下，没有理由使用 while 来读取单行，如果查询 returns 超过一行，您的代码将无法工作。您可以使用简单的 if (rs.next()) 并在 else.

中抛出一些异常或向用户显示一些错误

java.lang.OutOfMemoryError: Java heap space during bytea download

java.lang.OutOfMemoryError: Java heap space during bytea download

java

postgresql

postgresql-9.3