添加Xavier和bias_filler後，損失值開始變爲負值。爲什麼？

Question

在我將xavier初始化添加到每個卷積層後，損失開始變爲負數。有人可以提出任何建議/理由嗎？ 我添加以下行所有卷積層：

weight_filler { 
      type: "xavier" 
     } 
     bias_filler { 
      type: "constant" 
      value: 0.1 
     } 

---

I0305 14:31:53.356343 11179 solver.cpp:219] Iteration 0 (-4.02766e+28 iter/s, 0.528933s/100 iters), loss = 2.05371 
I0305 14:31:53.356374 11179 solver.cpp:238]  Train net output #0: accuracy = 0.11937 
I0305 14:31:53.356384 11179 solver.cpp:238]  Train net output #1: loss = 2.05371 (* 1 = 2.05371 loss) 
I0305 14:31:53.356395 11179 sgd_solver.cpp:105] Iteration 0, lr = 0.0001 
I0305 14:32:28.728870 11179 solver.cpp:219] Iteration 100 (2.82699 iter/s, 35.3733s/100 iters), loss = 0.0270034 
I0305 14:32:28.729014 11179 solver.cpp:238]  Train net output #0: accuracy = 1 
I0305 14:32:28.729028 11179 solver.cpp:238]  Train net output #1: loss = 0 (* 1 = 0 loss) 
I0305 14:32:28.729034 11179 sgd_solver.cpp:105] Iteration 100, lr = 0.0001 
I0305 14:33:03.729997 11179 solver.cpp:219] Iteration 200 (2.85701 iter/s, 35.0017s/100 iters), loss = -8.27284e-09 
I0305 14:33:03.730154 11179 solver.cpp:238]  Train net output #0: accuracy = 1 
I0305 14:33:03.730167 11179 solver.cpp:238]  Train net output #1: loss = 0 (* 1 = 0 loss) 
I0305 14:33:03.730172 11179 sgd_solver.cpp:105] Iteration 200, lr = 0.0001 
I0305 14:33:38.885211 11179 solver.cpp:219] Iteration 300 (2.84449 iter/s, 35.1557s/100 iters), loss = -8.27284e-09 
I0305 14:33:38.885368 11179 solver.cpp:238]  Train net output #0: accuracy = 1 
I0305 14:33:38.885383 11179 solver.cpp:238]  Train net output #1: loss = 0 (* 1 = 0 loss) 
I0305 14:33:38.885387 11179 sgd_solver.cpp:105] Iteration 300, lr = 0.0001 
I0305 14:34:14.174548 11179 solver.cpp:219] Iteration 400 (2.83368 iter/s, 35.2898s/100 iters), loss = -8.27284e-09 
I0305 14:34:14.174702 11179 solver.cpp:238]  Train net output #0: accuracy = 1 
I0305 14:34:14.174720 11179 solver.cpp:238]  Train net output #1: loss = 0 (* 1 = 0 loss) 
I0305 14:34:14.174724 11179 sgd_solver.cpp:105] Iteration 400, lr = 0.0001 
I0305 14:34:49.578112 11179 solver.cpp:219] Iteration 500 (2.82453 iter/s, 35.4041s/100 iters), loss = -8.27284e-09 
I0305 14:34:49.578254 11179 solver.cpp:238]  Train net output #0: accuracy = 1 
I0305 14:34:49.578269 11179 solver.cpp:238]  Train net output #1: loss = 0 (* 1 = 0 loss) 
I0305 14:34:49.578272 11179 sgd_solver.cpp:105] Iteration 500, lr = 0.0001 
I0305 14:35:25.042238 11179 solver.cpp:219] Iteration 600 (2.81971 iter/s, 35.4646s/100 iters), loss = -8.27284e-09 
I0305 14:35:25.042421 11179 solver.cpp:238]  Train net output #0: accuracy = 1 
I0305 14:35:25.042438 11179 solver.cpp:238]  Train net output #1: loss = 0 (* 1 = 0 loss) 
I0305 14:35:25.042443 11179 sgd_solver.cpp:105] Iteration 600, lr = 0.0001 
I0305 14:36:00.540053 11179 solver.cpp:219] Iteration 700 (2.81704 iter/s, 35.4983s/100 iters), loss = -8.27284e-09 
I0305 14:36:00.540194 11179 solver.cpp:238]  Train net output #0: accuracy = 1 
I0305 14:36:00.540207 11179 solver.cpp:238]  Train net output #1: loss = 

我的另一個問題是，在一些網絡中，Gaussian加入。像：

weight_filler { 
    type: "gaussian" 
    std: 0.005 
} 
bias_filler { 
    type: "constant" 
    value: 0.1 
} 

1. 我們爲什麼要添加這些參數來卷積層？是因爲我們從頭開始培訓網絡，所以是 ？

2. 如何將特定值分配給std和/或bias_filler 值？

我真的很感謝你的幫忙。

Answer 1

• 你的損失是-8.27284e-09這實際上來講是零，而不是否定（朱古力使用單精度浮點數，而不是雙精度）。
  你使用了什麼損失層？ "SoftmaxWithLoss"？

• bias_filler和wieght_filler參數添加，當我們想朱古力給隨機初始化層的權重，平時我們從頭開始培訓。如果您從現有模型開始訓練（即微調），則這些參數沒有意義。

• std根據扇入和扇出（即通道內和通道的數量）計算值，以保持Blob值的統計值大致爲零均值和單位方差。
  您可以在Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian SunDelving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (arXiv 2015)中找到這些參數的分析。