查找共有5個字段中的3個的行 - 如何加快查詢速度？

Question

下面的查詢很好，但速度很慢。在約7500行的表中，大約需要30秒才能執行。我怎麼能加快速度？

目標是在同一張表中找到「幾乎重複」的行。當匹配的5個領域中有3個我們有一個命中。

SELECT  
originalTable.id, 
originalTable.lastname, 
originalTable.firstname, 
originalTable.address, 
originalTable.city, 
originalTable.email 

FROM 
address as originalTable, 
address as compareTable 

WHERE 

# do not find the same record 
originalTable.id != compareTable.id and 

# at least 3 out of those 5 should match 
(originalTable.firstname = compareTable.firstname) + 
(originalTable.lastname = compareTable.lastname)  + 
(originalTable.address = compareTable.address and originalTable.address != '')  + 
(originalTable.city = compareTable.city and originalTable.city != '')  + 
(originalTable.email = compareTable.email and originalTable.email != '') 
>= 3 


GROUP BY 
originalTable.id 

ORDER BY 
originalTable.lastname asc, 
originalTable.firstname asc, 
originalTable.city asc 

感謝您的任何優化提示。

Answer 1

笛卡爾產品在這裏是必需的，這是真的。我想出了以下解決方案：

CREATE TABLE address_dups(INDEX (is_duplicate)) ENGINE=MEMORY 
SELECT 
    originalTable.id, 
    compareTable.id, 
(
    (originalTable.firstname = compareTable.firstname) + 
    (originalTable.lastname = compareTable.lastname) + 
    (originalTable.address = compareTable.address and originalTable.address != '') + 
    (originalTable.city = compareTable.city and originalTable.city != '') + 
    (originalTable.email = compareTable.email and originalTable.email != '') 
    >= 3 
) AS is_duplicate 
FROM 
address as originalTable, 
address as compareTable 
WHERE originalTable.id != compareTable.id; 

SELECT * FROM address_dups WHERE is_duplicate = 1; 

這會給你的每一行標識的模糊重複的行標識您的要求爲好。

Answer 2

您已經注意到的比較將需要笛卡爾...但只有一個部分。既然你在第一個和最後一個姓名字段中都需要一個值，那麼我至少需要一個索引名，即姓。然後，僅在部分姓氏上添加WHERE子句以適應您的條件...說出前2-3個字符。這樣，它只會對笛卡兒與其他名稱的前綴進行比較。將「Bill Jones」與「Tonya Smith」進行比較沒有意義。但是，您可能會對常見地址，城市和/或電子郵件中的「Bill Jones」與「William Jones」感興趣。考慮以下用於笛卡爾比較的名稱部分。

(names fictitious for sample) 
ID Last  First 
1 Adams Brian 
2 Adams Marsha 
3 Andrews Jeff 
4 Brown Steve 
5 Johns Dave 
6 Johnson Bill 
7 Johnson William 

如果您只在您的where子句中限定每個姓氏的左邊3，那麼將比較這兩個「Adams」。 「安德魯斯」和「布朗」沒有比較匹配任何人。然後，從「Joh」開始的3將被笛卡爾測試...

現在，在where子句中添加一個MORE ...因爲您有一個ID列，請確保它也是where子句的一部分。第二張桌子的ID總是大於你所在的桌子上的ID。例如：比較「亞當斯」的名字。您將已經知道ID1是否與ID2作爲重複進行比較（在本例中不是），爲什麼後退並重新比較ID2和ID1。

所以，7條本示例將導致

1-2 
2-no more to compare against 
3-no more to compare against 
4-no more to compare against 
5-6 
5-7 
6-7 
7-no more to compare against 

所以比較的決賽，會是這樣的（包括那是一場勢均力敵的比賽爲基礎，回顧一下。你可以在ID即使得到的所有列作爲 「MatchFirstName，MatchLastName，MatchCity等」 只是用於預覽的目的......）

SELECT 
     originalTable.id, 
     originalTable.lastname, 
     originalTable.firstname, 
     originalTable.address, 
     originalTable.city, 
     originalTable.email, 
     compareTable.ID as MatchID 
    FROM 
     address as originalTable, 
     address as compareTable 
    WHERE 
      originalTable.ID < CompareTable.ID 
     AND left(originalTable.LastName, 3) = left(CompareTable.LastName, 3) 
     AND (originalTable.firstname = compareTable.firstname) 
     + (originalTable.lastname = compareTable.lastname) 
     + (originalTable.address = compareTable.address and originalTable.address != '') 
     + (originalTable.city = compareTable.city and originalTable.city != '') 
     + (originalTable.email = compareTable.email and originalTable.email != '') >= 3