将 3GB 文件转换为字节数组

Question

我正在尝试将 3GB 文件转换为字节数组，但出现 OOM(OutOfMemoryError)。

我们试过了

RandomAccessFile randomAccessFile = new RandomAccessFile(sourceLocation.getAbsolutePath(), "r");
MappedByteBuffer mappedByteBuffer = randomAccessFile.getChannel().map(FileChannel.MapMode.READ_ONLY, 0, randomAccessFile.length()); //while it is large file, it threw 'mmap failed: ENOMEM (Out of memory)' exception.

byte[] data = new byte[1024 * 1024];  // 1MB read at time
while (mappedByteBuffer.hasRemaining()) {
    int remaining = data.length;

    if (mappedByteBuffer.remaining() < remaining)
        remaining = mappedByteBuffer.remaining();
        mappedByteBuffer.get(data, 0, remaining);
    }

    mappedByteBuffer.rewind();
    byte fileContent[] = mappedByteBuffer.array(); //while it is small file, it threw 'java.nio.ReadOnlyBufferException' exception.
    randomAccessFile.close();
}

我的自定义请求正文：自定义请求正文class我的请求正文准备的地方

import android.os.Looper;

import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;

import okhttp3.MediaType;
import okhttp3.RequestBody;
import okio.BufferedSink;

public class ProgressRequestBodyTemp extends RequestBody
{
    private static final int DEFAULT_BUFFER_SIZE = 2048;
    private File mFile;
    private String mPath;
    private String mFileType;
    private int mItemIndex;
    private UploadCallbacks mListener;

    private byte[] encryptedData;
    private ByteArrayInputStream bis;

    public ProgressRequestBodyTemp(final String fileType, final File file, byte[] encryptedData, final int itemIndex, final UploadCallbacks listener)
    {
        this.mFile = file;
        this.mFileType = fileType;
        this.mItemIndex = itemIndex;
        this.mListener = listener;

        this.encryptedData = encryptedData;

        try
        {
            bis = new ByteArrayInputStream(encryptedData); // Convert byte array into input stream for send data to server
        }
        catch (Exception ex)
        {
            ex.printStackTrace();
        }
    }

    @Override
    public MediaType contentType()
    {
        try
        {
            return MediaType.parse(mFileType);
        }
        catch (Exception ex)
        {
            return MediaType.parse("");
        }
    }

    @Override
    public long contentLength() throws IOException
    {
        return encryptedData.length;
    }

    @Override
    public void writeTo(BufferedSink sink) throws IOException
    {
        long fileLength = mFile.length();
        byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];

        long uploaded = 0;
        try
        {
            int read;
            android.os.Handler handler = new android.os.Handler(Looper.getMainLooper());
            while ((read = bis.read(buffer)) != -1)
            {

                // update progress on UI thread
                handler.post(new ProgressUpdater(uploaded, encryptedData.length));

                uploaded += read;
                sink.write(buffer, 0, read);
            }
        }
        finally
        {
            bis.close();
        }
    }

    public interface UploadCallbacks
    {
        void onProgressUpdate(int itemIndex, int percentage);
    }

    private class ProgressUpdater implements Runnable
    {
        private long mUploaded;
        private long mTotal;

        public ProgressUpdater(long uploaded, long total)
        {
            mUploaded = uploaded;
            mTotal = total;
        }

        @Override
        public void run()
        {
            if (mListener != null)
            {
                mListener.onProgressUpdate(mItemIndex, (int) (100 * mUploaded / mTotal));
            }
        }
    }
}

我的请求：请求上传文件

File sourceLocation = new File(".....");
byte fileContent[] = .... // byte array

ProgressRequestBody requestFile  = new ProgressRequestBody(fileType, sourceLocation, fileContent, itemIndex, this);
MultipartBody.Part body = MultipartBody.Part.createFormData("file", sourceLocation.getName(), requestFile); // MultipartBody.Part is used to send also the actual file name
Call<String> mediaCall = getRetrofit().addMedia("SOME STRING DATA", body);

正在给 OutOfMemoryError，请建议将大文件转换为字节数组的最佳方法。

非常感谢任何帮助。提前致谢。

Answer 1

使用InputStream和OutputStream。

它们适用于大量数据，例如，当您处理 3GB 的数据并且无法将其加载到内存中时。

如果您尝试上传文件，请使用 FileInputStream。创建一个 File 对象，将其传递给 FileOutputStream 构造函数并开始从其 InputStream 读取字节到 byte[] 缓冲区，然后将带有 OutputStream 的字节发送到服务器。

这种方法不会导致 OutOfMemoryError，因为您读取的字节数足以填满缓冲区，缓冲区应该约为 2KB - 大小为 8KB。缓冲区满后，将字节写入服务器。缓冲区写入服务器后，你再次读取缓冲区，这个过程一直持续到整个文件上传。

使用 FileInputStream 的示例

        File file = new File("yourfile.txt");
        FileInputStream fis = null;
        OutputStream outputStream = null;

        url = new URL(desiredUrl);
        HttpURLConnection connection = (HttpURLConnection) url.openConnection();


        try {
            fis = new FileInputStream(file);
            connection.setDoOutput(true);
            outputStream = connection.getOutputStream();

            int actuallyRead;
            byte[] buffer = new byte[2048];
            while ((actuallyRead = fis.read(buffer)) != -1) {
                //do something with bytes, for example, write to the server
            outputStream.write(buffer);

            }

        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                if (fis != null)
                    fis.close();
            } catch (IOException ex) {
                ex.printStackTrace();
            }
            try {
                if (outputStream != null)
                    outputStream.close();
            } catch (IOException ex) {
                ex.printStackTrace();
            }

        }

Note: This approach does not mean you need to reconnect to the server each time a buffer is filled. It will continuously write to the server, until it is done processing the file. This will happen all under the same, single connection.

将 3GB 文件转换为字节数组

Converting a 3GB file into a byte array

java

android

out-of-memory

使用 FileInputStream 的示例