MySQL - SQL代碼優化

Question

我對編寫SQL很感興趣，並剛剛構建了一些向MySQL數據庫添加數據的過程。問題是由於大量的查詢，它非常緩慢。我現在所做的是循環遍歷包含未分類的原始數據的表中的每條記錄，然後獲取該數據點並添加到數據庫中。由於我有一些我必須處理的FK，這變得很複雜。

你能幫我優化一下嗎？

作爲一個例子，添加指定表我做的：CALL add_table1(112,15);

過程添加數據
- 初級過程

CREATE PROCEDURE `add_table1`(
    IN c_id INT UNSIGNED; 
    IN t_id INT UNSIGNED; 
) 
BEGIN 
    -- Table variables 
    DECLARE r_id INT UNSIGNED; 
    DECLARE dh_name VARCHAR(50); 
    DECLARE d_value DECIMAL(20,10); 

    -- Loop variables 
    DECLARE done BOOLEAN; 

    -- Cursor for measurement table 
    DECLARE m_cur CURSOR FOR 
     SELECT Run_ID, DataHeader_Name, Data_Value 
     FROM `measurements`.`measurement_20131029_152902`; 

    -- Handlers for exceptions 
    DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE; 

    -- Set start time 
    UPDATE `measurements`.`queue` 
     SET Start_Time = NOW() 
     WHERE Experiment_ID = 112 AND Procedure_Name = 'add_table1'; 

    -- Loop through measurement table 
    OPEN m_cur; 
    m_loop: LOOP 
     FETCH m_cur INTO r_id, dh_name, d_value; 
     IF done THEN 
      CLOSE m_cur; 
      LEAVE m_loop; 
     END IF; 
     CALL add_measurement(dh_name, d_value, t_id, c_id, r_id); 
    END LOOP m_loop; 
END 

過程添加測量
- 二次程序，從add_table1調用

目前的解決方案
我偶然發現了this解決有關類似問題。我在TRANSACTION內附上了代碼的肉，並立即注意到速度的大幅提高。而不是查詢的預計完成時間大約爲36小時，我將實際完成時間縮短到大約5分鐘！我還對數據庫進行了輕微的設計更改，並刪除了不必要的FK。如果有人看到更多的方法來改善這個代碼，我仍然感興趣。我的應用程序的性能已達到可接受的範圍，但我總是對改善情況感興趣。

以顯示更改：

START TRANSACTION; 

    -- Loop through measurement table 
    OPEN m_cur; 
    m_loop: LOOP 
     FETCH m_cur INTO r_id, dh_name, d_value; 
     IF done THEN 
      CLOSE m_cur; 
      LEAVE m_loop; 
     END IF; 
     CALL add_measurement(dh_name, d_value, t_id, c_id, r_id); 
    END LOOP m_loop; 

    COMMIT; 

替代解決方案
基於以下關閉的答案，我可以更新我的新的解決方案，以下面的一個。從我的測試中看來，這個新解決方案正在按照需要運行。它也比以前的解決方案快兩倍以上。使用這個例程，我可以在大約2.5分鐘內添加一百萬個獨特的數據！

謝謝大家的幫助！

CREATE PROCEDURE `add_table`(
    IN config_id_var INT UNSIGNED 
) 
BEGIN 
    START TRANSACTION; 

    -- Add header 
    INSERT IGNORE INTO data_headers(DataHeader_Name) 
     SELECT DataHeader_Name 
     FROM `measurements`.`measurement_20131114_142402`; 

    -- Add measurement 
    INSERT IGNORE INTO tool_data(Data_Value) 
     SELECT Data_Value 
     FROM `measurements`.`measurement_20131114_142402`; 

    -- Link measurement to header and configuration 
     -- INSERT Non-Unique Values 
    INSERT IGNORE INTO tool_data_link(DataHeader_ID, ToolData_ID, Run_ID) 
     SELECT h.DataHeader_ID, d.ToolData_ID, m.Run_ID 
     FROM `measurements`.`measurement_20131114_142402` AS m 
     JOIN data_headers AS h ON h.DataHeader_Name = m.DataHeader_Name 
     JOIN tool_data AS d ON d.Data_Value = m.Data_Value; 
     -- INSERT Unique Values 
    INSERT IGNORE INTO tool_data_link(DataHeader_ID, ToolData_ID, Run_ID) 
     SELECT h.DataHeader_ID, d.ToolData_ID, m.Run_ID 
     FROM `measurements`.`measurement_20131114_142402` AS m 
     LEFT OUTER JOIN data_headers AS h ON h.DataHeader_Name = m.DataHeader_Name 
     LEFT OUTER JOIN tool_data AS d ON d.Data_Value = m.Data_Value 
     WHERE ((h.DataHeader_Name IS NULL) OR (d.Data_Value IS NULL)); 

    -- Link measurement to experiment configuration 
     -- INSERT Non-Unique Values 
    INSERT IGNORE INTO tool_link(ToolDataLink_ID, Config_ID) 
     SELECT tdl.ToolDataLink_ID, config_id_var 
     FROM tool_data_link AS tdl 
     JOIN data_headers AS h ON h.DataHeader_ID = tdl.DataHeader_ID 
     JOIN tool_data AS d ON d.ToolData_ID = tdl.ToolData_ID; 
     -- INSERT Unique Values 
    INSERT IGNORE INTO tool_link(ToolDataLink_ID, Config_ID) 
     SELECT tdl.ToolDataLink_ID, config_id_var 
     FROM tool_data_link AS tdl 
     LEFT OUTER JOIN data_headers AS h ON h.DataHeader_ID = tdl.DataHeader_ID 
     LEFT OUTER JOIN tool_data AS d ON d.ToolData_ID = tdl.ToolData_ID 
     WHERE ((h.DataHeader_ID IS NULL) OR (d.ToolData_ID IS NULL)); 

    COMMIT; 
END 

結論
我與未使用遊標解決方案的一些更多的測試。起初，這絕對是更快的;但是，當數據庫的大小增長時，執行時間急劇增加。

我在數據庫中添加了幾百萬個數據點。然後我嘗試添加大約幾百個數據點的小數據集。它比遊標解決方案花費了近400倍的時間。我相信這是因爲遊標只查看所需的數據點，因爲連接必須查看所有數據。

基於這些結果，似乎光標解決方案將更適合我的應用程序。

Answer 1

只是作爲一個例子，拿這個說法從你的問題：

-- Link measurement to header and configuration 
INSERT IGNORE INTO tool_data_link(DataHeader_ID, ToolData_ID, Run_ID) 
    SELECT t1.DataHeader_ID, t2.ToolData_ID, t3.Run_ID 
    FROM data_headers t1, tool_data t2, runs t3 
    WHERE ((t1.DataHeader_Name in (
     SELECT DataHeader_Name 
     FROM `measurements`.`measurement_20131114_110059`) 
    ) AND (t2.Data_Value in (
     SELECT Data_Value 
     FROM `measurements`.`measurement_20131114_110059`) 
    ) AND (t3.Run_ID in (
     SELECT Run_ID 
     FROM `measurements`.`measurement_20131114_110059`))); 

比較的只是SELECT表現有以下：

SELECT h.DataHeader_ID, d.ToolData_ID, m.Run_ID 
    FROM measurements.measurement_20131114_110059 AS m 
    JOIN data_headers AS h ON h.DataHeader_Name = m.DataHeader_Name 
    JOIN tool_data AS d ON d.Data_Value = m.Data_Value 
    ; 

注意：您絕對會需要DataHeader_Name和Data_Value在兩個表中都有索引。 （注意：加入臨時表幾乎總是很慢並且永遠無法正確索引）

這裏的底線是所有的數據都來自measurement_ *。如果您想要表演，請儘可能少地使用其他表格。

Answer 2

數據庫將使用基於集合的邏輯。 嘗試編寫沒有遊標的相同查詢。

What your code does: 
-1: Reads the records from measurement_xxxx.... table and for each one it executes 5 insert. 

How this can be writen with set logic: 
-1: make the first insert for all the records from measurement_XXXX.... 
-2: make the second insert for all the records from measurement_XXX.... 
....... 

This will change the code like the following: (I did not test the code, so it is a guideline) 

CREATE PROCEDURE `using_set_logic`(
    IN c_id INT UNSIGNED; 
    IN t_id INT UNSIGNED; 
) 
begin 
-- the first insert 
insert into data_headers(DataHeader_Name) 
select DataHeader_Name FROM `measurements`.`measurement_20131029_152902 
-- the second insert if the dataheadername is uniqueu 
insert into tool_header_link(DataHeader_ID, Tool_ID) 
select DataHeader_ID, t_id from data_headers where DataHeader_Name in (
    select DataHeader_Name FROM `measurements`.`measurement_20131029_152902 
) 
--the second insert if the dataheadername isnot unique. 
--take the last id for the dataheaders before the first insert 
--take the last id for the dataheaders after the insert 
--use those values to bound the data header id. 
insert into tool_header_link(DataHeader_ID, Tool_ID) 
select DataHeader_ID, @t_id from data_headers 
where DataHeader_id between @beforeFirstInsert and @afterFirstInsert 

end