我怎样才能使我的文件 reading/writing 程序更快？

Question

所以我有一个包含一堆天气数据的大文件。我必须将大文件中的每一行分配到其相应的状态文件中。因此总共将有 50 个具有自己数据的新状态文件。

大文件包含约 100 万行记录，如下所示：

COOP:166657,'NEW IBERIA AIRPORT ACADIANA REGIONAL LA US',200001,177,553

虽然电台的名称可能会有所不同并且有不同的字数。

目前我正在使用正则表达式查找模式并输出到文件，它必须按状态分组。如果我在不做任何修改的情况下读入整个文件，大约需要 46 秒。使用查找州缩写、创建文件并输出到该文件的代码，需要 10 多分钟。

这是我现在拥有的：

package climate;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.util.Arrays;
import java.util.Scanner;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

/**
 * This program will read in a large file containing many stations and states,
 * and output in order the stations to their corresponding state file.
 * 
 * Note: This take a long time depending on processor. It also appends data to
 * the files so you must remove all the state files in the current directory
 * before running for accuracy.
 * 
 * @author Marcus
 *
 */

public class ClimateCleanStates {

    public static void main(String[] args) throws IOException {

        Scanner in = new Scanner(System.in);
        System.out
                .println("Note: This program can take a long time depending on processor.");
        System.out
                .println("It is also not necessary to run as state files are in this directory.");
        System.out
                .println("But if you would like to see how it works, you may continue.");
        System.out.println("Please remove state files before running.");
        System.out.println("\nIs the States directory empty?");
        String answer = in.nextLine();

        if (answer.equals("N")) {
            System.exit(0);
            in.close();
        }
        System.out.println("Would you like to run the program?");
        String answer2 = in.nextLine();
        if (answer2.equals("N")) {
            System.exit(0);
            in.close();
        }

        String[] statesSpaced = new String[51];

        File statefile, dir, infile;

        // Create files for each states
        dir = new File("States");
        dir.mkdir();


        infile = new File("climatedata.csv");
        FileReader fr = new FileReader(infile);
        BufferedReader br = new BufferedReader(fr);

        String line;
        line = br.readLine(); 
        System.out.println();

        // Read in climatedata.csv
        final long start = System.currentTimeMillis();
        while ((line = br.readLine()) != null) {
            // Remove instances of -9999
            if (!line.contains("-9999")) {

                        String stateFileName = null;

                        Pattern p = Pattern.compile(".* ([A-Z][A-Z]) US"); 
                        Matcher m = p.matcher(line);
                        if (m.find()){
                            stateFileName = m.group(1); 
                            stateFileName = "States/" + stateFileName + ".csv";
                            statefile = new File(stateFileName);

                            FileWriter stateWriter = new FileWriter(statefile, true);
                            stateWriter.write(line + "\n");
                            // Progress reporting
                            //System.out.printf("Writing [%s] to file [%s]\n", line,
                            //      statefile);

                            stateWriter.flush();
                            stateWriter.close();

                        } 
            }
        }
        System.out.println("Elapsed " + (System.currentTimeMillis() - start) + " ms");
        br.close();
        fr.close();
        in.close();

    }

}

Answer 1

不要在循环中做任何你不需要做的事情。制作一张以文件名为关键字的文件地图，并让它们保持打开状态。根据需要给他们写信，在完成之前不要关闭任何一个。您也永远不需要冲洗它们。你正在击败所有好的缓冲。完成后，您可以考虑增大缓冲区。

Answer 2

您可以使用 Map 来跟踪状态文件而不是每次都关闭它们：

Map<String, FileWriter> fileMap = new HashMap<String, FileWriter>();

while ((line = br.readLine()) != null) {
    if (!line.contains("-9999")) {
        if (m.find()) {
            stateFileName = m.group(1); 
            stateFileName = "States/" + stateFileName + ".csv";
            FileWriter stateFileWriter = fileMap.get(stateFileName);
            if (stateFileWriter == null) {
                stateFileWriter = new FileWriter(stateFileName, true);
                fileMap.put(stateFileName, stateFileWriter);
            }

            stateFileWriter.write(line + "\n");
        }
    }
}

// flush the writers and close once you have parsed the entire file
for(Map.Entry<String, FileWriter> entry : fileMap.entrySet()) {
    FileWriter writer = entry.getValue();
    writer.flush();
    writer.close();
}

我怎样才能使我的文件 reading/writing 程序更快？

How can I make my file reading/writing program faster?

java

regex

io

file