如何将 ArrayList 存储在文件中？

Question

我有一个 class 表示存储在文件中的 ArrayList，因为我需要一个包含数 GB 数据的 ArrayList，这显然太大而无法存储在内存中。数据由一个名为 Field 的 class 表示，函数 Field.parse() 仅用于将 Field 转换为 String 和其他方式。

字段 class 存储（奇怪的）棋子及其坐标的列表。

我的 class 工作正常，但是向文件添加元素需要很长时间，我需要我的程序尽快运行。有谁知道更多 efficient/faster 的做事方式？

此外，我不允许使用外部 libraries/apis。请记住这一点。

这是负责在临时文件中存储 Field 对象的 class：

private File file;
private BufferedReader reader;
private BufferedWriter writer;

public FieldSaver() {
    try {
        file = File.createTempFile("chess-moves-", ".temp");
        System.out.println(file.getAbsolutePath());
    } catch (IOException e) {
        e.printStackTrace();
    }
}

public void add(Field field) {
    try {
        File temp = File.createTempFile("chess-moves-", ".temp");
        writer = new BufferedWriter(new FileWriter(temp));
        reader = new BufferedReader(new FileReader(file));
        String line;

        while((line = reader.readLine()) != null ) {
            writer.write(line);
            writer.newLine();
        }

        reader.close();
        writer.write(field.parse());
        writer.close();
        file.delete();
        file = new File(temp.getAbsolutePath());
    } catch (IOException e) {
        e.printStackTrace();
    }
}

public Field get(int n) {
    try {
        reader = new BufferedReader(new FileReader(file));
        for (int i = 0; i < n; i++) {
            reader.readLine();
        }
        String line = reader.readLine();
        reader.close();
        return Field.parse(line);
    } catch (IOException e) {
        e.printStackTrace();
    }
    return null;
}

这是字段 class：

private WildBoar wildBoar;
private HuntingDog[] huntingDogs;
private Hunter hunter;

private int size;

@Override
public String toString() {
    String result = "Wildschwein: " + wildBoar.toString();
    for (HuntingDog dog : huntingDogs) {
        result += "; Hund: " + dog.toString();
    }
    return result + "; Jäger: " + hunter.toString();
}

@Override
public boolean equals(Object obj) {
    if (obj instanceof Field) {
        Field field = (Field) obj;
        HuntingDog[] dogs = field.getHuntingDogs();
        return wildBoar.equals(field.getWildBoar()) && hunter.equals(field.getHunter()) && huntingDogs[0].equals(dogs[0]) && huntingDogs[1].equals(dogs[1]) && huntingDogs[2].equals(dogs[2]);
    }
    return false;
}

public Field(int size, WildBoar wildBoar, HuntingDog[] huntingDogs, Hunter hunter) {
    this.size = size;
    this.wildBoar = wildBoar;
    this.huntingDogs = huntingDogs;
    this.hunter = hunter;
}

public WildBoar getWildBoar() {
    return wildBoar;
}

public HuntingDog[] getHuntingDogs() {
    return huntingDogs;
}

public Hunter getHunter() {
    return hunter;
}

public int getSize() {
    return size;
}

public static Field parse(String s) {
    String[] arr = s.split(",");
    WildBoar boar = WildBoar.parse(arr[0]);
    Hunter hunter = Hunter.parse(arr[1]);
    HuntingDog[] dogs = new HuntingDog[arr.length - 2];
    for(int i = 2; i < arr.length; i++) {
        dogs[i - 2] = HuntingDog.parse(arr[i]);
    }
    return new Field(8, boar, dogs, hunter);
}

public String parse() {
    String result = wildBoar.parse() + "," + hunter.parse();
    for(HuntingDog dog : huntingDogs) {
        result += "," + dog.parse();
    }
    return result;
}

Answer 1

使用 Map 实现，例如来自 ehcache 的 Cache。该库将为您进行优化，因此您不必处理写入和读取磁盘以及管理何时将其保存在内存或磁盘上。您可以将其用作法线贴图。您可能需要地图而不是列表来加快查找速度，以便库可以为您进行更多优化。

http://www.ehcache.org/

CacheManager cacheManager = CacheManagerBuilder.newCacheManagerBuilder()
      .withCache("preConfigured",
           CacheConfigurationBuilder.newCacheConfigurationBuilder(Long.class, String.class,
                                          ResourcePoolsBuilder.heap(100))
           .build())
      .build(true);

  Cache<Long, String> preConfigured
      = cacheManager.getCache("preConfigured", Long.class, String.class);

Answer 2

这是一个 MCVE，可以根据您提供的信息执行您想要的操作。

你可以运行看看它可以将 Field 保存到文件并通过索引非常快速地获得 Field。

Fields 是常量长度，因此您可以通过索引乘以字段长度（以字节为单位）的字节偏移量来按索引获得 Field。如果字段不是恒定长度，这将更加困难。

import java.io.Closeable;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.RandomAccessFile;

public class FieldSaver implements Closeable {

    public static void main(String[] args) throws IOException {
        File f = File.createTempFile("chess-moves-", ".temp");
        try (FieldSaver test = new FieldSaver(f);) {
            for (byte i = 0; i < 100; i++) {
                test.add(new Field(8, new WildBoar(i, i), new Hunter(i, i), new HuntingDog[] {
                        new HuntingDog(i, i),
                        new HuntingDog(i, i),
                        new HuntingDog(i, i) }));
            }

            // Get a few Fields by index
            System.out.println(test.get(0));
            System.out.println(test.get(50));
            System.out.println(test.get(99));

            // EOF exception, there is no Field 100
            // System.out.println(test.get(100));
        }
    }

    private final RandomAccessFile data;

    public FieldSaver(File f) throws FileNotFoundException {
        data = new RandomAccessFile(f, "rw");
    }

    public void add(Field field) throws IOException {
        data.seek(data.length());
        field.write(data);
    }


    public Field get(int index) throws IOException {
        data.seek(index * Field.STORAGE_LENGTH_BYTES);
        return Field.read(data);
    }

    public void close() throws IOException { data.close(); }


    static abstract class Piece {
        protected byte xPos;
        protected byte yPos;

        public Piece(DataInput data) throws IOException {
            xPos = data.readByte();
            yPos = data.readByte();
        }

        public Piece(byte xPos, byte yPos) {
            this.xPos = xPos;
            this.yPos = yPos;
        }

        public void write(DataOutput data) throws IOException {
            data.writeByte(xPos);
            data.writeByte(yPos);
        }

        public String toString() { return "[" + xPos + ", " + yPos + "]"; }
    }

    static class Hunter extends Piece {
        public Hunter(byte xPos, byte yPos) { super(xPos, yPos); }
        public Hunter(DataInput data) throws IOException { super(data); }
    }

    static class HuntingDog extends Piece {
        public HuntingDog(byte xPos, byte yPos) { super(xPos, yPos); }
        public HuntingDog(DataInput data) throws IOException { super(data); }
    }

    static class WildBoar extends Piece {
        public WildBoar(byte xPos, byte yPos) { super(xPos, yPos); }
        public WildBoar(DataInput data) throws IOException { super(data); }
    }

    static class Field {
        // size of boar + hunter + 3 dogs
        public static final int STORAGE_LENGTH_BYTES = 2 + 2 + (3 * 2);

        private int size;
        private WildBoar boar;
        private Hunter hunter;
        private final HuntingDog[] dogs;

        public Field(int size, WildBoar wildBoar, Hunter hunter, HuntingDog[] huntingDogs) {
            this.size = size;
            this.boar = wildBoar;
            this.hunter = hunter;
            this.dogs = huntingDogs;
        }

        public String toString() {
            String result = "Wildschwein: " + boar.toString();
            for (HuntingDog dog : dogs) {
                result += "; Hund: " + dog.toString();
            }
            return result + "; Jäger: " + hunter.toString();
        }

        public static Field read(DataInput data) throws IOException {
            WildBoar boar = new WildBoar(data);
            Hunter hunter = new Hunter(data);
            HuntingDog[] dogs = new HuntingDog[3];
            for (int i = 0; i < 3; i++) {
                dogs[i] = new HuntingDog(data);
            }
            return new Field(8, boar, hunter, dogs);
        }

        public void write(DataOutput data) throws IOException {
            boar.write(data);
            hunter.write(data);
            for (HuntingDog dog : dogs) {
                dog.write(data);
            }
        }
    }
}

如何将 ArrayList 存储在文件中？

How to store an ArrayList in a file?

java

file

arraylist

out-of-memory