如何並行RNN功能Pytorch與數據並行

下面是一個RNN模式來運行基於字符的語言生成：如何並行RNN功能Pytorch與數據並行

class RNN(nn.Module): 
    def __init__(self, input_size, hidden_size, output_size, n_layers): 
     super(RNN, self).__init__() 
     self.input_size = input_size 
     self.hidden_size = hidden_size 
     self.output_size = output_size 
     self.n_layers = n_layers 

     self.encoder = nn.Embedding(input_size, hidden_size) 
     self.GRU = nn.GRU(hidden_size, hidden_size, n_layers, batch_first=True) 
     self.decoder = nn.Linear(hidden_size, output_size) 


    def forward(self, input, batch_size): 
     self.init_hidden(batch_size) 
     input = self.encoder(input) 
     output, self.hidden = self.GRU(input, self.hidden) 
     output = self.decoder(output.view(batch_size, self.hidden_size)) 
     return output 

    def init_hidden(self, batch_size): 
     self.hidden = Variable(torch.randn(self.n_layers, batch_size, self.hidden_size).cuda())

我實例使用數據並行，跨我的4個GPU批量投入拆分模式：

net = torch.nn.DataParallel(RNN(n_chars, hidden_size, n_chars, n_layers)).cuda()

這是full code。

不幸的是，數據並行需要輸入具有的batch_size的第一維度，但GRU函數希望隱藏張量具有的batch_size作爲第二維度：

output, self.hidden = self.GRU(input, self.hidden)

作爲是該代碼引發以下錯誤（注意打印輸出示出了編碼器正確地對4個GPU執行）：

... 
forward function: encoding input of shape: (16L, 1L) 
forward function: encoding input of shape: (16L, 1L) 
forward function: encoding input of shape: (16L, 
forward function: encoding input of shape: 

forward function: GRU processing input of shape: 
1L) 
((16L, 16L1L, 1L), 100L) 
forward function: GRU processing input of shape: 
(16L, 1L, 
forward function: GRU processing input of shape:100L) 
(16L 
forward function: GRU processing input of shape:, 1L, 100L) (
16L, 1L, 100L) 

Traceback (most recent call last): 
    File "gru2.py", line 166, in <module> 
    output = net(c, batch_size) 
    File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__ 
    result = self.forward(*input, **kwargs) 
    File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 61, in forward 
    outputs = self.parallel_apply(replicas, inputs, kwargs) 
    File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 71, in parallel_apply 
    return parallel_apply(replicas, inputs, kwargs) 
    File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py", line 45, in parallel_apply 
    raise output 
RuntimeError: Expected hidden size (2, 16L, 100), got (2L, 64L, 100L)

這裏，該模型具有2層，=的batch_size 64和hidden_size = 100。

如何在轉發功能中並行化GRU操作？

來源

2017-06-16 MichaelSB

您可以簡單地設置參數dim = 1，例如

net = torch.nn.DataParallel(RNN(n_chars, hidden_size, n_chars, n_layers), dim=1).cuda()

來源

2017-06-17 17:56:17 dalegebit

如何並行RNN功能Pytorch與數據並行

回答

相關問題