2017-01-24 184 views
0

我試圖解析大Apache日誌,但import_logs.py失敗,此錯誤:Import_logs.py失敗(大於900 MB)

2017年1月23日18: 30:39245:達到嘗試[INFO]最大數,服務器無法訪問_

Fatal error: HTTP Error 500 Internal Server Error, response:

{"status":"error","tracked":0,"invalid":0,"invalidindices":[]}_ You

can restart the import of

"/awdata/piwik/cosmote/geratgweb04/www.cosmote.grwith_ssl-access.log-20170123" from the point it failed by specifying --skip=78454 on the command line.

我使用的是虛擬機與8個CPU和8 GB MEM!

我的OS版本RHEL 7.3,

與Apache版本的Apache/2.4.6(紅帽企業Linux),

PHP是PHP 7.0.14和MariaDB的版本是52年5月5日 Piwik 3.0.1版

我使用的命令是:

/var/www/html/zak/piwik/misc/log-analytics/import_logs.py --url=http://middlinf.ote.gr:81/piwik/ --idsite=6 --recorders=8 --enable-http-errors --enable-http-redirects --enable-static /awdata/piwik/cosmote/geratgweb04/www.cosmote.gr_with_ssl-access.log-2017--retry-max-attempts=20 

Apache的錯誤日誌中顯示:

[Tue Jan 24 13:44:33.855134 2017] [:error] [pid 17456:tid 140544525190912] [client 172.18.20.26:16610] Piwik(tracker)中的錯誤:錯誤查詢:SQLSTATE [40001]:序列化失敗:1213嘗試鎖定時發現死鎖;嘗試重新啓動事務在查詢:

UPDATE piwik_log_visit SET idvisitor = ?, user_id = ?, 
    visit_last_action_time = ?, visit_exit_idaction_url = ?, 
    visit_total_actions = visit_total_actions + 1 , 
    visit_total_interactions = visit_total_interactions + 1 , 
    visit_total_time = ? 
    WHERE idsite = ? AND idvisit = ? 

參數:陣列(0 => '\ xa6 \ X96 \ XBC \ XEF \ xb9 \ XDE \ XF5',1 => ' 「」',2 =>「2017 -01-23 8時14分43' 秒,3 => 47298,4 => 0,5 => 4,6 => 11224,)

和:

[星期二年01月24 13點51:錯誤查詢:SQLSTATE [HY000]:常規錯誤:1205超出鎖定超時等待時間;錯誤代碼:SQLSTATE [HY000]:常規錯誤:嘗試重新啓動事務在查詢:

UPDATE piwik_log_visit SET idvisitor = ?, user_id = ?, 
    visit_last_action_time = ?, visit_exit_idaction_url = ?, 
    visit_total_actions = visit_total_actions + 1 , 
    visit_total_interactions = visit_total_interactions + 1 , 
    visit_total_time = ? 
    WHERE idsite = ? AND idvisit = ? 

參數:陣列(0 => '\ xa6 \ X96 \ XBC \ XEF \ xb9 \ XDE \ XF5',1 => ' 「」',2 =>「2017 -01-23 8時42分33' 秒,3 => 49791,4 => 242,5 => 4,6 => 11371)>

and mariadb log is showing: 

Time: 17:00:46 
[email protected]: root[root] @ localhost [127.0.0.1] 
Thread_id: 1691 Schema: piwik_db QC_hit: No 
Query_time: 3.858223 Lock_time: 0.000060 Rows_sent: 1 Rows_examined: 1 
 SET timestamp=1485187246; 
    SELECT visit_last_action_time, visit_first_action_time, idvisitor, 
idvisit, user_id, visit_exit_idaction_url, visit_exit_idaction_name, 
visitor_returning, visitor_days_since_first, visitor_days_since_order, 
visitor_count_visits, visit_goal_buyer, location_country, 
location_region, location_city, location_latitude, location_longitude, 
referer_name, referer_keyword, referer_type, idsite, 
visit_entry_idaction_url, visit_total_actions, 
visit_total_interactions, visit_total_searches, config_device_brand, 
config_device_model, config_device_type, visit_total_events, 
visit_total_time, location_ip, location_browser_lang, custom_var_k1, 
custom_var_v1, custom_var_k2, custom_var_v2, custom_var_k3, 
custom_var_v3, custom_var_k4, custom_var_v4, custom_var_k5, 
custom_var_v5 FROM piwik_log_visit 
WHERE visit_last_action_time >= '2017-01-22 05:14:21' 
    AND visit_last_action_time <= '2017-01-22 06:14:21' 
    AND idsite = '6' AND idvisitor = ' 
ORDER BY visit_last_action_time DESC 

我做了一些研究在論壇中的但沒有發現任何有趣的事情。 你有什麼建議嗎?

預先感謝您。

-Thanassis

回答

0

請提供SHOW CREATE TABLE piwik_log_visit。我懷疑你缺少複合

INDEX(idsite, idvisit) 

(各列可以按任意順序。)

對於

WHERE visit_last_action_time >= '2017-01-22 05:14:21' 
    AND visit_last_action_time <= '2017-01-22 06:14:21' 
    AND idsite = '6' AND idvisitor = ' 

你需要INDEX(idsite, idvisitor, visit_last_action_time),是一定有最後的範圍列。

您是否注意到該範圍涵蓋了3601秒?我推薦這種模式的原因有多種:

WHERE visit_last_action_time >= '2017-01-22 05:14:21' 
    AND visit_last_action_time < '2017-01-22 05:14:21' + INTERVAL 1 HOUR 
+0

非常感謝您的回覆。這是SHOW CREATE TABLE piwik_log_visit的輸出。 https://docs.google.com/document/d/1po0mmh9oxXXmCebns9392fxRpzPHWEYGd02eyYO6HjY/edit確實沒有這樣的索引。我將嘗試創建它並重新運行日誌導入。我會盡快更新你的。關於間隔模式,我不知道該怎麼做,因爲它是從piwik命令import_logs.py中手工完成的。這不是一個自定義查詢。 –

+0

抱怨piwik。 –