在Java中的並行線程中寫入文件的最佳方式是什麼？

Question

我有一個程序執行大量的計算並頻繁地將它們報告給一個文件。我知道頻繁的寫操作會讓程序慢下來，所以爲了避免這種情況，我希望有第二個線程專門用於寫操作。

現在，我這個班，我寫這樣做（性急可以跳過這個問題的結束）：

public class ParallelWriter implements Runnable { 

    private File file; 
    private BlockingQueue<Item> q; 
    private int indentation; 

    public ParallelWriter(File f){ 
     file = f; 
     q = new LinkedBlockingQueue<Item>(); 
     indentation = 0; 
    } 

    public ParallelWriter append(CharSequence str){ 
     try { 
      CharSeqItem item = new CharSeqItem(); 
      item.content = str; 
      item.type = ItemType.CHARSEQ; 
      q.put(item); 
      return this; 
     } catch (InterruptedException ex) { 
      throw new RuntimeException(ex); 
     } 
    } 

    public ParallelWriter newLine(){ 
     try { 
      Item item = new Item(); 
      item.type = ItemType.NEWLINE; 
      q.put(item); 
      return this; 
     } catch (InterruptedException ex) { 
      throw new RuntimeException(ex); 
     } 
    } 

    public void setIndent(int indentation) { 
     try{ 
      IndentCommand item = new IndentCommand(); 
      item.type = ItemType.INDENT; 
      item.indent = indentation; 
      q.put(item); 
     } catch (InterruptedException ex) { 
      throw new RuntimeException(ex); 
     } 
    } 

    public void end(){ 
     try { 
      Item item = new Item(); 
      item.type = ItemType.POISON; 
      q.put(item); 
     } catch (InterruptedException ex) { 
      throw new RuntimeException(ex); 
     } 
    } 

    public void run() { 

     BufferedWriter out = null; 
     Item item = null; 

     try{ 
      out = new BufferedWriter(new FileWriter(file)); 
      while((item = q.take()).type != ItemType.POISON){ 
       switch(item.type){ 
        case NEWLINE: 
         out.newLine(); 
         for(int i = 0; i < indentation; i++) 
          out.append(" "); 
         break; 
        case INDENT: 
         indentation = ((IndentCommand)item).indent; 
         break; 
        case CHARSEQ: 
         out.append(((CharSeqItem)item).content); 
       } 
      } 
     } catch (InterruptedException ex){ 
      throw new RuntimeException(ex); 
     } catch (IOException ex) { 
      throw new RuntimeException(ex); 
     } finally { 
      if(out != null) try { 
       out.close(); 
      } catch (IOException ex) { 
       throw new RuntimeException(ex); 
      } 
     } 
    } 

    private enum ItemType { 
     CHARSEQ, NEWLINE, INDENT, POISON; 
    } 
    private static class Item { 
     ItemType type; 
    } 
    private static class CharSeqItem extends Item { 
     CharSequence content; 
    } 
    private static class IndentCommand extends Item { 
     int indent; 
    } 
} 

然後我做使用它：

ParallelWriter w = new ParallelWriter(myFile); 
new Thread(w).start(); 

/// Lots of 
w.append(" things ").newLine(); 
w.setIndent(2); 
w.newLine().append(" more things "); 

/// and finally 
w.end(); 

雖然這工作得很好，但我想知道： 有沒有更好的方法來實現這個目標？

Answer 1

你的基本方法看起來不錯。如下我將構建代碼：

import java.io.BufferedWriter; 
import java.io.File; 
import java.io.IOException; 
import java.io.Writer; 
import java.util.concurrent.BlockingQueue; 
import java.util.concurrent.LinkedBlockingQueue; 
import java.util.concurrent.TimeUnit; 

public interface FileWriter { 
    FileWriter append(CharSequence seq); 

    FileWriter indent(int indent); 

    void close(); 
} 

class AsyncFileWriter implements FileWriter, Runnable { 
    private final File file; 
    private final Writer out; 
    private final BlockingQueue<Item> queue = new LinkedBlockingQueue<Item>(); 
    private volatile boolean started = false; 
    private volatile boolean stopped = false; 

    public AsyncFileWriter(File file) throws IOException { 
     this.file = file; 
     this.out = new BufferedWriter(new java.io.FileWriter(file)); 
    } 

    public FileWriter append(CharSequence seq) { 
     if (!started) { 
      throw new IllegalStateException("open() call expected before append()"); 
     } 
     try { 
      queue.put(new CharSeqItem(seq)); 
     } catch (InterruptedException ignored) { 
     } 
     return this; 
    } 

    public FileWriter indent(int indent) { 
     if (!started) { 
      throw new IllegalStateException("open() call expected before append()"); 
     } 
     try { 
      queue.put(new IndentItem(indent)); 
     } catch (InterruptedException ignored) { 
     } 
     return this; 
    } 

    public void open() { 
     this.started = true; 
     new Thread(this).start(); 
    } 

    public void run() { 
     while (!stopped) { 
      try { 
       Item item = queue.poll(100, TimeUnit.MICROSECONDS); 
       if (item != null) { 
        try { 
         item.write(out); 
        } catch (IOException logme) { 
        } 
       } 
      } catch (InterruptedException e) { 
      } 
     } 
     try { 
      out.close(); 
     } catch (IOException ignore) { 
     } 
    } 

    public void close() { 
     this.stopped = true; 
    } 

    private static interface Item { 
     void write(Writer out) throws IOException; 
    } 

    private static class CharSeqItem implements Item { 
     private final CharSequence sequence; 

     public CharSeqItem(CharSequence sequence) { 
      this.sequence = sequence; 
     } 

     public void write(Writer out) throws IOException { 
      out.append(sequence); 
     } 
    } 

    private static class IndentItem implements Item { 
     private final int indent; 

     public IndentItem(int indent) { 
      this.indent = indent; 
     } 

     public void write(Writer out) throws IOException { 
      for (int i = 0; i < indent; i++) { 
       out.append(" "); 
      } 
     } 
    } 
} 

如果你不希望在一個單獨的線程（？也許在測試）寫的，你可以有FileWriter實現其在呼叫方的Writer呼籲append線。

Answer 2

使用LinkedBlockingQueue是一個不錯的主意。不知道我喜歡代碼的一些風格......但原理似乎很合理。

我可能會添加一個容量到LinkedBlockingQueue等於總內存的一定百分比..說10,000個項目..這種方式如果你的寫作速度太慢，你的工作線程將不會繼續添加更多的工作，直到堆被炸燬。

Answer 3

與單個消費者線程交換數據的一種好方法是使用交換器。

您可以使用StringBuilder或ByteBuffer作爲與後臺線程交換的緩衝區。發生的延遲可能在1微秒左右，不涉及創建任何對象，而使用BlockingQueue更低。

從我認爲值得在這裏重複的例子。

class FillAndEmpty { 
    Exchanger<DataBuffer> exchanger = new Exchanger<DataBuffer>(); 
    DataBuffer initialEmptyBuffer = ... a made-up type 
    DataBuffer initialFullBuffer = ... 

    class FillingLoop implements Runnable { 
    public void run() { 
     DataBuffer currentBuffer = initialEmptyBuffer; 
     try { 
     while (currentBuffer != null) { 
      addToBuffer(currentBuffer); 
      if (currentBuffer.isFull()) 
      currentBuffer = exchanger.exchange(currentBuffer); 
     } 
     } catch (InterruptedException ex) { ... handle ... } 
    } 
    } 

    class EmptyingLoop implements Runnable { 
    public void run() { 
     DataBuffer currentBuffer = initialFullBuffer; 
     try { 
     while (currentBuffer != null) { 
      takeFromBuffer(currentBuffer); 
      if (currentBuffer.isEmpty()) 
      currentBuffer = exchanger.exchange(currentBuffer); 
     } 
     } catch (InterruptedException ex) { ... handle ...} 
    } 
    } 

    void start() { 
    new Thread(new FillingLoop()).start(); 
    new Thread(new EmptyingLoop()).start(); 
    } 
} 

Answer 4

我知道，頻繁的寫操作 可以減緩程序了很多

也許並不像你想，只要你使用緩衝。