使用密码解密大文件时出现内存不足异常
Out of memory exception when decrypt large file using Cipher
我正在尝试使用 javax.crypto 下的 类 和 input/output 的文件流来实现 encrypt/decrypt 程序。
为了限制内存使用，我 运行 加上 -Xmx256m 参数。
它适用于较小文件的加密和解密。
但是解密超大文件（1G大小）时，出现内存不足异常：
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at com.sun.crypto.provider.GaloisCounterMode.decrypt(GaloisCounterMode.java:505)
at com.sun.crypto.provider.CipherCore.update(CipherCore.java:782)
at com.sun.crypto.provider.CipherCore.update(CipherCore.java:667)
at com.sun.crypto.provider.AESCipher.engineUpdate(AESCipher.java:380)
at javax.crypto.Cipher.update(Cipher.java:1831)
at javax.crypto.CipherOutputStream.write(CipherOutputStream.java:166)
解密代码如下：
private final int _readSize = 0x10000;//64k
...
GCMParameterSpec gcmParameterSpec = new GCMParameterSpec(gcmTagSize, iv);
Key keySpec = new SecretKeySpec(key, keyParts[0]);
Cipher decCipher = Cipher.getInstance("AES/GCM/PKCS5Padding");
decCipher.init(Cipher.DECRYPT_MODE, keySpec, gcmParameterSpec);
try (InputStream fileInStream = Files.newInputStream(inputEncryptedFile);
OutputStream fileOutStream = Files.newOutputStream(outputDecryptedFile)) {
try (CipherOutputStream cipherOutputStream = new CipherOutputStream(fileOutStream, decCipher)) {
long count = 0L;
byte[] buffer = new byte[_readSize];
int n;
for (; (n = fileInStream.read(buffer)) != -1; count += (long) n) {
cipherOutputStream.write(buffer, 0, n);
}
}
}
gcmTagSize 和 iv 等关键参数是从关键文件中读取的，它适用于较小的文件，例如 50M 左右的文件。
据我了解，每次只有 64k 数据传递给解密，为什么它 运行 堆内存不足？
我怎样才能避免这种情况？
编辑：
实际上我尝试过使用 4k 作为缓冲区大小，失败并出现同样的异常。
编辑 2：
经过更多测试，它可以处理的最大文件大小约为堆大小的 1/4。例如，如果您设置 -Xmx256m，大于 64M 的文件将无法解密。
这似乎是 GCM 模式实施的问题。我不确定您是否可以解决它。
如果您查看堆栈跟踪：
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at com.sun.crypto.provider.GaloisCounterMode.decrypt(GaloisCounterMode.java:505)
从 GaloisCounterMode 中写入 ByteArrayOutputStream 时发生内存不足错误。您使用 FileOutputStream，所以要么您没有显示正确的代码，要么这个 ByteArrayStream 在内部使用。
如果您查看 source for GaloisCounterMode，您会发现它定义了一个内部 ByteArrayOutputStream（它实际上定义了两个，但我认为这就是问题所在）：
// buffer for storing input in decryption, not used for encryption
private ByteArrayOutputStream ibuffer = null;
然后，稍后，它将字节写入此流。注意代码注释。
int decrypt(byte[] in, int inOfs, int len, byte[] out, int outOfs) {
processAAD();
if (len > 0) {
// store internally until decryptFinal is called because
// spec mentioned that only return recovered data after tag
// is successfully verified
ibuffer.write(in, inOfs, len);
}
return 0;
}
该缓冲区直到 decryptFinal() 才被重置。
编辑：查看 this CSx answer 看起来 GCM 需要缓冲整个流。如果您有大文件而内存不足，那将是一个非常糟糕的选择。
我认为你最好的解决办法是切换到 CBC 模式。
坏消息是：恕我直言，该错误是由本机 Java 中 AES GCM 模式的错误实施引起的。即使你可以让它工作，你也会发现解密一个大文件（1 GB 左右）会花费很多时间（可能是几个小时？）。
但是有 好消息：您 could/should 使用 BouncyCastle 作为解密任务的服务提供者 - 这样解密就会工作并且速度要快得多。
以下完整示例将创建一个 1 GB 大小的示例文件，使用 BouncyCastle 对其进行加密，然后对其进行解密。最后有一个文件比较，表明明文和解密文件内容相同，文件将被删除。 您的设备上 space 需要临时总共超过 3 GB 的可用空间才能 运行 此示例。
使用 64 KB 的缓冲区，我运行使用以下数据在这个例子中：
加密毫秒数：14295 |解密：16249
1 KB 的缓冲区在加密方面有点慢，但在解密任务方面慢得多：
加密毫秒数：15250 |解密：21952
关于您的密码的最后一句话 - "AES/GCM/PKCS5Padding" 在某些实现中不存在并且 "available" 但实际使用的算法是 "AES/GCM/NoPadding"（有关更多详细信息，请参阅 Can PKCS5Padding be in AES/GCM mode? ).
import org.bouncycastle.jce.provider.BouncyCastleProvider;
import javax.crypto.BadPaddingException;
import javax.crypto.Cipher;
import javax.crypto.IllegalBlockSizeException;
import javax.crypto.NoSuchPaddingException;
import javax.crypto.spec.GCMParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.io.*;
import java.nio.file.Files;
import java.security.*;
import java.util.Arrays;
public class GcmTestBouncyCastle {
public static void main(String[] args) throws IOException, NoSuchPaddingException, InvalidAlgorithmParameterException,
NoSuchAlgorithmException, IllegalBlockSizeException, BadPaddingException, NoSuchProviderException, InvalidKeyException {
System.out.println("Encryption & Decryption with BouncyCastle AES-GCM-Mode");
System.out.println("");
// you need bouncy castle, get version 1.65 here:
// https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15on/1.65
Security.addProvider(new BouncyCastleProvider());
// setup files
// filenames
String filenamePlain = "plain.dat";
String filenameEncrypt = "encrypt.dat";
String filenameDecrypt = "decrypt.dat";
// generate a testfile of 1024 byte | 1 gb
//createFileWithDefinedLength(filenamePlain, 1024);
createFileWithDefinedLength(filenamePlain, 1024 * 1024 * 1024); // 1 gb
// time measurement
long startMilli = 0;
long encryptionMilli = 0;
long decryptionMilli = 0;
// generate nonce/iv
int GCM_NONCE_LENGTH = 12; // for a nonce of 96 bit length
int GCM_TAG_LENGTH = 16;
int GCM_KEY_LENGTH = 32; // 32 = 256 bit keylength, 16 = 128 bit keylength
SecureRandom r = new SecureRandom();
byte[] nonce = new byte[GCM_NONCE_LENGTH];
r.nextBytes(nonce);
// key should be generated as random byte[]
byte[] key = new byte[GCM_KEY_LENGTH];
r.nextBytes(key);
// encrypt file
startMilli = System.currentTimeMillis();
encryptWithGcmBc(filenamePlain, filenameEncrypt, key, nonce, GCM_TAG_LENGTH);
encryptionMilli = System.currentTimeMillis() - startMilli;
startMilli = System.currentTimeMillis();
decryptWithGcmBc(filenameEncrypt, filenameDecrypt, key, nonce, GCM_TAG_LENGTH);
decryptionMilli = System.currentTimeMillis() - startMilli;
// check that plain and decrypted files are equal
System.out.println("SHA256-file compare " + filenamePlain + " | " + filenameDecrypt + " : "
+ Arrays.equals(sha256File(filenamePlain), sha256File(filenameDecrypt)));
System.out.println("Milliseconds for Encryption: " + encryptionMilli + " | Decryption: " + decryptionMilli);
// clean up with files
Files.deleteIfExists(new File(filenamePlain).toPath());
Files.deleteIfExists(new File(filenameEncrypt).toPath());
Files.deleteIfExists(new File(filenameDecrypt).toPath());
}
public static void encryptWithGcmBc(String filenamePlain, String filenameEnc, byte[] key, byte[] nonce, int gcm_tag_length)
throws IOException, NoSuchAlgorithmException, NoSuchPaddingException, InvalidKeyException,
InvalidAlgorithmParameterException, IllegalBlockSizeException, BadPaddingException, NoSuchProviderException {
Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding", "BC");
SecretKeySpec keySpec = new SecretKeySpec(key, "AES");
GCMParameterSpec gcmSpec = new GCMParameterSpec(gcm_tag_length * 8, nonce);
cipher.init(Cipher.ENCRYPT_MODE, keySpec, gcmSpec);
try (FileInputStream fis = new FileInputStream(filenamePlain);
BufferedInputStream in = new BufferedInputStream(fis);
FileOutputStream out = new FileOutputStream(filenameEnc);
BufferedOutputStream bos = new BufferedOutputStream(out)) {
//byte[] ibuf = new byte[1024];
byte[] ibuf = new byte[0x10000]; // = 65536
int len;
while ((len = in.read(ibuf)) != -1) {
byte[] obuf = cipher.update(ibuf, 0, len);
if (obuf != null)
bos.write(obuf);
}
byte[] obuf = cipher.doFinal();
if (obuf != null)
bos.write(obuf);
}
}
public static void decryptWithGcmBc(String filenameEnc, String filenameDec, byte[] key, byte[] nonce, int gcm_tag_length)
throws IOException, NoSuchAlgorithmException, NoSuchPaddingException, InvalidKeyException,
InvalidAlgorithmParameterException, IllegalBlockSizeException, BadPaddingException, NoSuchProviderException {
try (FileInputStream in = new FileInputStream(filenameEnc);
FileOutputStream out = new FileOutputStream(filenameDec)) {
//byte[] ibuf = new byte[1024];
byte[] ibuf = new byte[0x10000]; // = 65536
int len;
Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding", "BC");
SecretKeySpec keySpec = new SecretKeySpec(key, "AES");
GCMParameterSpec gcmSpec = new GCMParameterSpec(gcm_tag_length * 8, nonce);
cipher.init(Cipher.DECRYPT_MODE, keySpec, gcmSpec);
while ((len = in.read(ibuf)) != -1) {
byte[] obuf = cipher.update(ibuf, 0, len);
if (obuf != null)
out.write(obuf);
}
byte[] obuf = cipher.doFinal();
if (obuf != null)
out.write(obuf);
}
}
// just for creating a large file within seconds
private static void createFileWithDefinedLength(String filenameString, long sizeLong) throws IOException {
RandomAccessFile raf = new RandomAccessFile(filenameString, "rw");
try {
raf.setLength(sizeLong);
} finally {
raf.close();
}
}
// just for file comparing
public static byte[] sha256File(String filenameString) throws IOException, NoSuchAlgorithmException {
byte[] buffer = new byte[8192];
int count;
MessageDigest md = MessageDigest.getInstance("SHA-256");
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(filenameString));
while ((count = bis.read(buffer)) > 0) {
md.update(buffer, 0, count);
}
bis.close();
return md.digest();
}
}
我正在尝试使用 javax.crypto 下的 类 和 input/output 的文件流来实现 encrypt/decrypt 程序。 为了限制内存使用，我 运行 加上 -Xmx256m 参数。
它适用于较小文件的加密和解密。 但是解密超大文件（1G大小）时，出现内存不足异常：
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at com.sun.crypto.provider.GaloisCounterMode.decrypt(GaloisCounterMode.java:505)
at com.sun.crypto.provider.CipherCore.update(CipherCore.java:782)
at com.sun.crypto.provider.CipherCore.update(CipherCore.java:667)
at com.sun.crypto.provider.AESCipher.engineUpdate(AESCipher.java:380)
at javax.crypto.Cipher.update(Cipher.java:1831)
at javax.crypto.CipherOutputStream.write(CipherOutputStream.java:166)
解密代码如下：
private final int _readSize = 0x10000;//64k
...
GCMParameterSpec gcmParameterSpec = new GCMParameterSpec(gcmTagSize, iv);
Key keySpec = new SecretKeySpec(key, keyParts[0]);
Cipher decCipher = Cipher.getInstance("AES/GCM/PKCS5Padding");
decCipher.init(Cipher.DECRYPT_MODE, keySpec, gcmParameterSpec);
try (InputStream fileInStream = Files.newInputStream(inputEncryptedFile);
OutputStream fileOutStream = Files.newOutputStream(outputDecryptedFile)) {
try (CipherOutputStream cipherOutputStream = new CipherOutputStream(fileOutStream, decCipher)) {
long count = 0L;
byte[] buffer = new byte[_readSize];
int n;
for (; (n = fileInStream.read(buffer)) != -1; count += (long) n) {
cipherOutputStream.write(buffer, 0, n);
}
}
}
gcmTagSize 和 iv 等关键参数是从关键文件中读取的，它适用于较小的文件，例如 50M 左右的文件。
据我了解，每次只有 64k 数据传递给解密，为什么它 运行 堆内存不足？ 我怎样才能避免这种情况？
编辑：
实际上我尝试过使用 4k 作为缓冲区大小，失败并出现同样的异常。
编辑 2：
经过更多测试，它可以处理的最大文件大小约为堆大小的 1/4。例如，如果您设置 -Xmx256m，大于 64M 的文件将无法解密。
这似乎是 GCM 模式实施的问题。我不确定您是否可以解决它。
如果您查看堆栈跟踪：
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at com.sun.crypto.provider.GaloisCounterMode.decrypt(GaloisCounterMode.java:505)
从 GaloisCounterMode 中写入 ByteArrayOutputStream 时发生内存不足错误。您使用 FileOutputStream，所以要么您没有显示正确的代码，要么这个 ByteArrayStream 在内部使用。
如果您查看 source for GaloisCounterMode，您会发现它定义了一个内部 ByteArrayOutputStream（它实际上定义了两个，但我认为这就是问题所在）：
// buffer for storing input in decryption, not used for encryption
private ByteArrayOutputStream ibuffer = null;
然后，稍后，它将字节写入此流。注意代码注释。
int decrypt(byte[] in, int inOfs, int len, byte[] out, int outOfs) {
processAAD();
if (len > 0) {
// store internally until decryptFinal is called because
// spec mentioned that only return recovered data after tag
// is successfully verified
ibuffer.write(in, inOfs, len);
}
return 0;
}
该缓冲区直到 decryptFinal() 才被重置。
编辑：查看 this CSx answer 看起来 GCM 需要缓冲整个流。如果您有大文件而内存不足，那将是一个非常糟糕的选择。
我认为你最好的解决办法是切换到 CBC 模式。
坏消息是：恕我直言，该错误是由本机 Java 中 AES GCM 模式的错误实施引起的。即使你可以让它工作，你也会发现解密一个大文件（1 GB 左右）会花费很多时间（可能是几个小时？）。 但是有 好消息：您 could/should 使用 BouncyCastle 作为解密任务的服务提供者 - 这样解密就会工作并且速度要快得多。
以下完整示例将创建一个 1 GB 大小的示例文件，使用 BouncyCastle 对其进行加密，然后对其进行解密。最后有一个文件比较，表明明文和解密文件内容相同，文件将被删除。 您的设备上 space 需要临时总共超过 3 GB 的可用空间才能 运行 此示例。
使用 64 KB 的缓冲区，我运行使用以下数据在这个例子中：
加密毫秒数：14295 |解密：16249
1 KB 的缓冲区在加密方面有点慢，但在解密任务方面慢得多：
加密毫秒数：15250 |解密：21952
关于您的密码的最后一句话 - "AES/GCM/PKCS5Padding" 在某些实现中不存在并且 "available" 但实际使用的算法是 "AES/GCM/NoPadding"（有关更多详细信息，请参阅 Can PKCS5Padding be in AES/GCM mode? ).
import org.bouncycastle.jce.provider.BouncyCastleProvider;
import javax.crypto.BadPaddingException;
import javax.crypto.Cipher;
import javax.crypto.IllegalBlockSizeException;
import javax.crypto.NoSuchPaddingException;
import javax.crypto.spec.GCMParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.io.*;
import java.nio.file.Files;
import java.security.*;
import java.util.Arrays;
public class GcmTestBouncyCastle {
public static void main(String[] args) throws IOException, NoSuchPaddingException, InvalidAlgorithmParameterException,
NoSuchAlgorithmException, IllegalBlockSizeException, BadPaddingException, NoSuchProviderException, InvalidKeyException {
System.out.println("Encryption & Decryption with BouncyCastle AES-GCM-Mode");
System.out.println("");
// you need bouncy castle, get version 1.65 here:
// https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15on/1.65
Security.addProvider(new BouncyCastleProvider());
// setup files
// filenames
String filenamePlain = "plain.dat";
String filenameEncrypt = "encrypt.dat";
String filenameDecrypt = "decrypt.dat";
// generate a testfile of 1024 byte | 1 gb
//createFileWithDefinedLength(filenamePlain, 1024);
createFileWithDefinedLength(filenamePlain, 1024 * 1024 * 1024); // 1 gb
// time measurement
long startMilli = 0;
long encryptionMilli = 0;
long decryptionMilli = 0;
// generate nonce/iv
int GCM_NONCE_LENGTH = 12; // for a nonce of 96 bit length
int GCM_TAG_LENGTH = 16;
int GCM_KEY_LENGTH = 32; // 32 = 256 bit keylength, 16 = 128 bit keylength
SecureRandom r = new SecureRandom();
byte[] nonce = new byte[GCM_NONCE_LENGTH];
r.nextBytes(nonce);
// key should be generated as random byte[]
byte[] key = new byte[GCM_KEY_LENGTH];
r.nextBytes(key);
// encrypt file
startMilli = System.currentTimeMillis();
encryptWithGcmBc(filenamePlain, filenameEncrypt, key, nonce, GCM_TAG_LENGTH);
encryptionMilli = System.currentTimeMillis() - startMilli;
startMilli = System.currentTimeMillis();
decryptWithGcmBc(filenameEncrypt, filenameDecrypt, key, nonce, GCM_TAG_LENGTH);
decryptionMilli = System.currentTimeMillis() - startMilli;
// check that plain and decrypted files are equal
System.out.println("SHA256-file compare " + filenamePlain + " | " + filenameDecrypt + " : "
+ Arrays.equals(sha256File(filenamePlain), sha256File(filenameDecrypt)));
System.out.println("Milliseconds for Encryption: " + encryptionMilli + " | Decryption: " + decryptionMilli);
// clean up with files
Files.deleteIfExists(new File(filenamePlain).toPath());
Files.deleteIfExists(new File(filenameEncrypt).toPath());
Files.deleteIfExists(new File(filenameDecrypt).toPath());
}
public static void encryptWithGcmBc(String filenamePlain, String filenameEnc, byte[] key, byte[] nonce, int gcm_tag_length)
throws IOException, NoSuchAlgorithmException, NoSuchPaddingException, InvalidKeyException,
InvalidAlgorithmParameterException, IllegalBlockSizeException, BadPaddingException, NoSuchProviderException {
Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding", "BC");
SecretKeySpec keySpec = new SecretKeySpec(key, "AES");
GCMParameterSpec gcmSpec = new GCMParameterSpec(gcm_tag_length * 8, nonce);
cipher.init(Cipher.ENCRYPT_MODE, keySpec, gcmSpec);
try (FileInputStream fis = new FileInputStream(filenamePlain);
BufferedInputStream in = new BufferedInputStream(fis);
FileOutputStream out = new FileOutputStream(filenameEnc);
BufferedOutputStream bos = new BufferedOutputStream(out)) {
//byte[] ibuf = new byte[1024];
byte[] ibuf = new byte[0x10000]; // = 65536
int len;
while ((len = in.read(ibuf)) != -1) {
byte[] obuf = cipher.update(ibuf, 0, len);
if (obuf != null)
bos.write(obuf);
}
byte[] obuf = cipher.doFinal();
if (obuf != null)
bos.write(obuf);
}
}
public static void decryptWithGcmBc(String filenameEnc, String filenameDec, byte[] key, byte[] nonce, int gcm_tag_length)
throws IOException, NoSuchAlgorithmException, NoSuchPaddingException, InvalidKeyException,
InvalidAlgorithmParameterException, IllegalBlockSizeException, BadPaddingException, NoSuchProviderException {
try (FileInputStream in = new FileInputStream(filenameEnc);
FileOutputStream out = new FileOutputStream(filenameDec)) {
//byte[] ibuf = new byte[1024];
byte[] ibuf = new byte[0x10000]; // = 65536
int len;
Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding", "BC");
SecretKeySpec keySpec = new SecretKeySpec(key, "AES");
GCMParameterSpec gcmSpec = new GCMParameterSpec(gcm_tag_length * 8, nonce);
cipher.init(Cipher.DECRYPT_MODE, keySpec, gcmSpec);
while ((len = in.read(ibuf)) != -1) {
byte[] obuf = cipher.update(ibuf, 0, len);
if (obuf != null)
out.write(obuf);
}
byte[] obuf = cipher.doFinal();
if (obuf != null)
out.write(obuf);
}
}
// just for creating a large file within seconds
private static void createFileWithDefinedLength(String filenameString, long sizeLong) throws IOException {
RandomAccessFile raf = new RandomAccessFile(filenameString, "rw");
try {
raf.setLength(sizeLong);
} finally {
raf.close();
}
}
// just for file comparing
public static byte[] sha256File(String filenameString) throws IOException, NoSuchAlgorithmException {
byte[] buffer = new byte[8192];
int count;
MessageDigest md = MessageDigest.getInstance("SHA-256");
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(filenameString));
while ((count = bis.read(buffer)) > 0) {
md.update(buffer, 0, count);
}
bis.close();
return md.digest();
}
}