在大表中優化SELECT

我有一個包含遊戲服務器日誌的表。這些日誌以非常糟糕的方式保存（imo），但我需要設法收集它們。該表目前擁有4,000,000個行和增長（這是4或5個月的日誌）。在大表中優化SELECT

現在我正在尋找一種方法來過濾SELECT查詢而不會對數據庫造成太大的負擔。這是表結構：

CREATE TABLE `rp_logs` (
    `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, 
    `time` int(11) NOT NULL, 
    `type` enum('joinleave','kill','chat','spawn','ac','cmd','misc','hidden','cp','admin') NOT NULL, 
    `msg` text NOT NULL, 
    PRIMARY KEY (`id`) 
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

現在我可以過濾時間範圍和日誌類型。但我想要更多。大部分時間日誌消息都包含玩家ID。這將是很好，如果我可以添加一個msg LIKE '%playerid%'。但是，如果我只是用簡單的查詢來做到這一點，它會將搜索動作花費在+30秒。

目前，我只是做（這是不是100％的代碼只是我做的方法）：

SELECT * FROM rp_logs 
WHERE time > $start AND time < $end 
AND (type = 'kill' or type = 'cmd' or ...);

是否有更好的優化查詢，所以它不是那麼重的方式。

我所知道的可能是查詢時間，然後用msg過濾子查詢，但我不知道它是如何工作的。我似乎無法得到那個工作。

這裏是30行的採樣數據：

INSERT INTO `rp_logs` (`id`, `time`, `type`, `msg`) VALUES 
(4314121, 1366788849, 'chat', '[OOC] Danny Doss (STEAM_0:0:20730895): He logged a long time ago Itt'), 
(4314122, 1366788849, 'chat', 'Eric Crendor (STEAM_0:0:46944468): the *'), 
(4314123, 1366788851, 'spawn', 'Danny Doss (STEAM_0:0:20730895) created entity prop_physics with model models/props_building_details/storefront_template001a_bars.mdl'), 
(4314124, 1366788855, 'chat', '[OOC] Eric Crendor (STEAM_0:0:46944468): did he?'), 
(4314125, 1366788868, 'chat', '[OOC] Danny Doss (STEAM_0:0:20730895): 5 minutes ago'), 
(4314126, 1366788872, 'misc', 'Daedalus Richmond (STEAM_0:1:20442870) has been made a Arms Dealer'), 
(4314127, 1366788879, 'chat', '[OOC] Eric Crendor (STEAM_0:0:46944468): nontheless we killed him then started cracking, he still left whilst raid?'), 
(4314128, 1366788893, 'chat', '[OOC] Jasonafex Mittsies (STEAM_0:0:29686591): He left because he went to bed'), 
(4314129, 1366788899, 'chat', '[OOC] Danny Doss (STEAM_0:0:20730895): Can''t do anything about it'), 
(4314130, 1366788903, 'chat', 'Eric Crendor (STEAM_0:0:46944468): left go to next dealer'), 
(4314131, 1366788904, 'chat', '[OOC] Danny Doss (STEAM_0:0:20730895): It is not warnable'), 
(4314132, 1366788909, 'chat', 'Eric Crendor (STEAM_0:0:46944468): * my grAMMAR'), 
(4314133, 1366788912, 'chat', 'Eric Crendor (STEAM_0:0:46944468): damn keyboard'), 
(4314134, 1366788913, 'spawn', 'Danny Doss (STEAM_0:0:20730895) created entity prop_physics with model models/props_c17/furnituredrawer001a.mdl'), 
(4314135, 1366788919, 'spawn', 'Danny Doss (STEAM_0:0:20730895) created entity prop_physics with model models/props_combine/combine_window001.mdl'), 
(4314136, 1366788921, 'chat', 'Eric Crendor (STEAM_0:0:46944468): k'), 
(4314137, 1366788923, 'spawn', 'Danny Doss (STEAM_0:0:20730895) created entity prop_physics with model models/props_c17/furniturecouch002a.mdl'), 
(4314138, 1366788932, 'kill', 'Danny Doss (STEAM_0:0:20730895) was killed by Entity [0][worldspawn], weapon: N/A'), 
(4314139, 1366788947, 'spawn', 'Jasonafex Mittsies (STEAM_0:0:29686591) created entity prop_physics with model models/props_lab/blastdoor001c.mdl'), 
(4314140, 1366788960, 'misc', 'AceFTFW (STEAM_0:0:37117785) has been made a Citizen'), 
(4314141, 1366788960, 'joinleave', 'Ace Jones (STEAM_0:0:37117785) joined the game.'), 
(4314142, 1366788967, 'joinleave', 'Jasonafex Mittsies (STEAM_0:0:29686591) has left the game.'), 
(4314143, 1366788971, 'joinleave', 'Danny Doss (STEAM_0:0:20730895) has left the game.'), 
(4314144, 1366788981, 'misc', 'Ace Jones (STEAM_0:0:37117785) has been made a Arms Dealer'), 
(4314145, 1366788989, 'chat', 'Eric Crendor (STEAM_0:0:46944468): okay'), 
(4314146, 1366788996, 'chat', 'Eric Crendor (STEAM_0:0:46944468): Ace'), 
(4314147, 1366789011, 'chat', 'Eric Crendor (STEAM_0:0:46944468): wheres your sho'), 
(4314148, 1366789014, 'chat', 'Eric Crendor (STEAM_0:0:46944468): we wanna get some guns'), 
(4314149, 1366789018, 'chat', 'Eric Crendor (STEAM_0:0:46944468): ah k'), 
(4314150, 1366789035, 'chat', 'Eric Crendor (STEAM_0:0:46944468): no literally');

來源

2013-04-24 s.lenders

也許你可以在表格視圖中保存基於時間的查詢結果，而不是根據表格視圖上的玩家名稱提出新的查詢。看看SQL - Views – 2013-04-24 07:48:45

你可以創建索引嗎？表？ – 2013-04-24 08:01:17

您通常查詢的時間範圍 - 不到一天，超過一個月，...？ – 2013-04-24 08:01:47

A LIKE query can be replaced by a FULLTEXT search.

SELECT * 
FROM rp_logs 
WHERE MATCH (msg) AGAINST ('playerid');

你必須先聲明，你msg列作爲FULLTEXT

ALTER TABLE rp_logs ADD FULLTEXT(msg);

或者

CREATE TABLE `rp_logs` (
    `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, 
    `time` int(11) NOT NULL, 
    `type` enum('joinleave','kill','chat','spawn','ac','cmd','misc','hidden','cp','admin') NOT NULL, 
    `msg` text NOT NULL, 
    PRIMARY KEY (`id`), 
    FULLTEXT (msg) 
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

進一步閱讀：

來源

2013-04-24 07:51:28

工程就像一個魅力！搜索1個用戶的所有加入和離開的一個用4,5 mil行的表格花了我1秒 – 2013-04-24 12:38:39

添加新的字段，其包含的用戶名的DB。做一個「LIKE」，特別是使用通配符，總是佔用大量資源（從而佔用大量時間）。此外，如果您有一個用於用戶名的單獨字段，則可以在其上放置索引，並使其更快。

來源

2013-04-24 07:47:28 Borniet

我知道，我想但它不是一個選項 – 2013-04-24 07:51:32

你可以在數據庫中創建任何東西嗎？你需要這些數據是實時的嗎？否則，您可以基於（長時間運行）查詢來創建一個新表，該查詢會篩選出用戶名並將其置於單獨的字段中。 ETL的一小部分介於兩者之間。 – Borniet 2013-04-24 07:58:36

我無法改變服務器保存日誌的方式。我確實可以將數據解析成新表格。但是有很多不同類型的方式可以保存郵件，我寧可不要 – 2013-04-24 08:07:26

增加您的time領域的重複指數。只要您查詢的時間範圍相對較窄，現有查詢的性能提升就會顯着。

來源

2013-04-24 08:00:20

這個從1秒縮短的搜索實際上從.9秒減少到了.0005秒，ty – 2013-04-24 08:28:56

在大表中優化SELECT

回答

相關問題