在文本文件中轉換矩陣的有效方法是什麼？

Question

我有一個文本文件，其中包含一個2維矩陣。它看起來像下面。

01 02 03 04 05 
06 07 08 09 10 
11 12 13 14 15 
16 17 18 19 20 

正如您所看到的，每行由一個新行分隔，並且每列由空格分隔。我需要以有效的方式轉置這個矩陣。

01 06 11 16 
02 07 12 17 
03 08 04 05 
04 09 14 19 
05 10 15 20 

實際上，矩陣是10,000乘以14,000。單個元素是雙重/浮動的。如果不是不可能的話，試圖將這個文件/矩陣全部轉置到內存中將是昂貴的。

沒有人知道一個util API來做類似這樣的事情或有效的方法嗎？

我曾嘗試過：我的幼稚方法是爲每列（轉置矩陣）創建一個臨時文件。所以，有10,000行，我將有10,000個臨時文件。當我讀取每一行時，我將每個值標記，並將該值附加到相應的文件中。所以對於上面的例子，我會有類似以下的內容。

file-0: 01 06 11 16 
file-1: 02 07 12 17 
file-3: 03 08 13 18 
file-4: 04 09 14 19 
file-5: 05 10 15 20 

然後我重新讀入每個文件並將它們追加到一個文件中。我想知道是否有更聰明的方法，因爲我知道文件I/O操作將是一個痛點。

Answer 1

解決方案使用最少的存儲和極低的性能：

import org.apache.commons.io.FileUtils; 

import java.io.BufferedWriter; 
import java.io.File; 
import java.io.FileWriter; 
import java.io.IOException; 

public class MatrixTransposer { 

    private static final String TMP_DIR = System.getProperty("java.io.tmpdir") + "/"; 
    private static final String EXTENSION = ".matrix.tmp.result"; 
    private final String original; 
    private final String dst; 

    public MatrixTransposer(String original, String dst) { 
    this.original = original; 
    this.dst = dst; 
    } 

    public void transpose() throws IOException { 

    deleteTempFiles(); 

    int max = 0; 

    FileReader fileReader = null; 
    BufferedReader reader = null; 
    try { 
     fileReader = new FileReader(original); 
     reader = new BufferedReader(fileReader); 
     String row; 
     while((row = reader.readLine()) != null) { 

     max = appendRow(max, row, 0); 
     } 
    } finally { 
     if (null != reader) reader.close(); 
     if (null != fileReader) fileReader.close(); 
    } 


    mergeResultingRows(max); 
    } 

    private void deleteTempFiles() { 
    for (String tmp : new File(TMP_DIR).list()) { 
     if (tmp.endsWith(EXTENSION)) { 
     FileUtils.deleteQuietly(new File(TMP_DIR + "/" + tmp)); 
     } 
    } 
    } 

    private void mergeResultingRows(int max) throws IOException { 

    FileUtils.deleteQuietly(new File(dst)); 

    FileWriter writer = null; 
    BufferedWriter out = null; 

    try { 
     writer = new FileWriter(new File(dst), true); 
     out = new BufferedWriter(writer); 
     for (int i = 0; i <= max; i++) { 
     out.write(FileUtils.readFileToString(new File(TMP_DIR + i + EXTENSION)) + "\r\n"); 
     } 
    } finally { 
     if (null != out) out.close(); 
     if (null != writer) writer.close(); 
    } 
    } 

    private int appendRow(int max, String row, int i) throws IOException { 

    for (String element : row.split(" ")) { 

     FileWriter writer = null; 
     BufferedWriter out = null; 
     try { 
     writer = new FileWriter(TMP_DIR + i + EXTENSION, true); 
     out = new BufferedWriter(writer); 
     out.write(columnPrefix(i) + element); 
     } finally { 
     if (null != out) out.close(); 
     if (null != writer) writer.close(); 
     } 
     max = Math.max(i++, max); 
    } 
    return max; 
    } 

    private String columnPrefix(int i) { 

    return (0 == i ? "" : " "); 
    } 

    public static void main(String[] args) throws IOException { 

    new MatrixTransposer("c:/temp/mt/original.txt", "c:/temp/mt/transposed.txt").transpose(); 
    } 
} 

Answer 2

總大小爲1.12GB（如果加倍），如果是浮點數則爲一半。這對於今天的機器來說足夠小，您可以在內存中執行它。不過，你可能想在原地進行轉位，而這是一項相當不重要的任務。 wikipedia article提供了進一步的鏈接。

Answer 3

我會建議，以評估可以在不消耗得多存儲器讀取列數。然後，通過讀取包含列數的塊的源文件幾次來編寫最終文件。假設你有10000列。首先讀取集合中源文件的列0到250，然後寫入最終文件。然後，再次爲第250列至第500列等等。

public class TransposeMatrixUtils { 

    private static final Logger logger = LoggerFactory.getLogger(TransposeMatrixUtils.class); 

    // Max number of bytes of the src file involved in each chunk 
    public static int MAX_BYTES_PER_CHUNK = 1024 * 50_000;// 50 MB 

    public static File transposeMatrix(File srcFile, String separator) throws IOException { 
     File output = File.createTempFile("output", ".txt"); 
     transposeMatrix(srcFile, output, separator); 
     return output; 
    } 

    public static void transposeMatrix(File srcFile, File destFile, String separator) throws IOException { 
     long bytesPerColumn = assessBytesPerColumn(srcFile, separator);// rough assessment of bytes par column 
     int nbColsPerChunk = (int) (MAX_BYTES_PER_CHUNK/bytesPerColumn);// number of columns per chunk according to the limit of bytes to be used per chunk 
     if (nbColsPerChunk == 0) nbColsPerChunk = 1;// in case a single column has more bytes than the limit ... 
     logger.debug("file length : {} bytes. max bytes per chunk : {}. nb columns per chunk : {}.", srcFile.length(), MAX_BYTES_PER_CHUNK, nbColsPerChunk); 
     try (FileWriter fw = new FileWriter(destFile); BufferedWriter bw = new BufferedWriter(fw)) { 
      boolean remainingColumns = true; 
      int offset = 0; 
      while (remainingColumns) { 
       remainingColumns = writeColumnsInRows(srcFile, bw, separator, offset, nbColsPerChunk); 
       offset += nbColsPerChunk; 
      } 
     } 
    } 

    private static boolean writeColumnsInRows(File srcFile, BufferedWriter bw, String separator, int offset, int nbColumns) throws IOException { 
     List<String>[] newRows; 
     boolean remainingColumns = true; 
     try (FileReader fr = new FileReader(srcFile); BufferedReader br = new BufferedReader(fr)) { 
      String[] split0 = br.readLine().split(separator); 
      if (split0.length <= offset + nbColumns) remainingColumns = false; 
      int lastColumnIndex = Math.min(split0.length, offset + nbColumns); 
      logger.debug("chunk for column {} to {} among {}", offset, lastColumnIndex, split0.length); 
      newRows = new List[lastColumnIndex - offset]; 
      for (int i = 0; i < newRows.length; i++) { 
       newRows[i] = new ArrayList<>(); 
       newRows[i].add(split0[i + offset]); 
      } 
      String line; 
      while ((line = br.readLine()) != null) { 
       String[] split = line.split(separator); 
       for (int i = 0; i < newRows.length; i++) { 
        newRows[i].add(split[i + offset]); 
       } 
      } 
     } 
     for (int i = 0; i < newRows.length; i++) { 
      bw.write(newRows[i].get(0)); 
      for (int j = 1; j < newRows[i].size(); j++) { 
       bw.write(separator); 
       bw.write(newRows[i].get(j)); 
      } 
      bw.newLine(); 
     } 
     return remainingColumns; 
    } 

    private static long assessBytesPerColumn(File file, String separator) throws IOException { 
     try (FileReader fr = new FileReader(file); BufferedReader br = new BufferedReader(fr)) { 
      int nbColumns = br.readLine().split(separator).length; 
      return file.length()/nbColumns; 
     } 
    } 

} 

它應該比創建大量臨時文件更有效，這些文件會生成大量的I/O。

對於10000 x 14000矩陣的示例，此代碼需要3分鐘來創建轉置文件。如果您設置MAX_BYTES_PER_CHUNK = 1024 * 100_000而不是1024 * 50_000，則需要2分鐘，但消耗更多的RAM當然。