刪除重複行但保持多對多關係

Question

我將我的Tensorflow圖像分類器的結果保存在SQL數據庫中。我有3張桌子。圖像，類別和一個表格將兩個與權重變量連接起來。有些圖片沒有關係，有些圖片有很多。

問題是我在圖像表中有需要刪除的重複行。但是如果重複的圖像有一個或多個，我需要保留多對多的關係。

下面是一個例子：

表名：my_images

+----+------------+-----------------+ 
| ID | image_path | image_filename | 
+----+------------+-----------------+ 
| 1 | Film 1  | Film 1 001.jpg | 
| 2 | Film 1  | Film 1 001.jpg | 
| 3 | Film 1  | Film 1 002.jpg | 
| 4 | Film 1  | Film 1 002.jpg | 
| 5 | Film 1  | Film 1 003.jpg | 
| 6 | Film 1  | Film 1 003.jpg | 
+----+------------+-----------------+ 

表名：my_terms

+---------+------------+ 
| term_id | term_name | 
+---------+------------+ 
|  1 | cat  | 
|  2 | dog  | 
|  3 | automobile | 
+---------+------------+ 

表名：my_term_relationships

+----------+---------+---------+ 
| image_id | term_id | weight | 
+----------+---------+---------+ 
|  2 |  1 | 0.58516 | 
|  2 |  3 | 0.16721 | 
|  3 |  2 | 0.21475 | 
+----------+---------+---------+ 

所以在這個例子中，理想的結果是從my_images刪除第1,4行和第5或6行。

Answer 1

您需要查詢兩組圖像ID，並使用它們進行過濾。假設image_path和image_filename是UNIQUE一起：

1. 所有my_images ID，即不通過my_term_relationships引用，但相應的image_path + image_filename可能被引用。
2. 唯一ID，屬於image_path + image_filename對，在my_term_relationships中根本沒有被引用。

在此查詢請看：

DELETE FROM my_images 
WHERE 
    ID NOT IN (SELECT DISTINCT image_id FROM my_term_relationships) -- 1 
    AND 
    ID NOT IN (SELECT id FROM (
    SELECT MIN(ID) as id 
    FROM my_images 
    LEFT JOIN my_term_relationships ON ID = image_id 
    GROUP BY image_path,image_filename 
    HAVING COUNT(image_id) = 0 
    ) as u_ids -- 2 
); 

注意，你必須包裹my_images表中DELETE的其中一個子查詢子句。閱讀此線程解釋：Can't specify target table for update in FROM clause

舉例：從my_term_relationships去除重複行sqlfiddle

---

示例更新查詢：

UPDATE my_term_relationships 
SET image_id = (
    select min(my_images.ID) 
    from my_images 
    join my_images as ref_image on (my_images.image_path = ref_image.image_path and my_images.image_filename = ref_image.image_filename) 
    where ref_image.ID = image_id 
); 

Answer 2

方法一步一步來。

首先，找到重複的條目：

SELECT 
image_path, image_filename 
FROM my_images 
GROUP BY image_path, image_filename 
HAVING COUNT(*) > 1 

其次，碰到一些重複的所有行：

SELECT mi.* 
FROM my_images mi 
JOIN (
    SELECT 
    image_path, image_filename 
    FROM my_images 
    GROUP BY image_path, image_filename 
    HAVING COUNT(*) > 1 
) dups ON mi.image_path = dups.image_path AND mi.image_filename = dups.image_filename 

最後，得到的ID不刪除。

SELECT MIN(ID) 
FROM my_images mi 
JOIN (
    SELECT 
    image_path, image_filename 
    FROM my_images 
    GROUP BY image_path, image_filename 
    HAVING COUNT(*) > 1 
) dups ON mi.image_path = dups.image_path AND mi.image_filename = dups.image_filename 
LEFT JOIN my_term_relationships mtr ON mi.ID = mtr.image_id 
WHERE mtr.image_id IS NULL 
GROUP BY mi.image_path, mi.image_filename 
HAVING COUNT(*) > 0 

• 看到它在sqlfiddle

檢查帶電作業，如果一切是正確的。如果是，請將其轉換爲刪除語句。

DELETE my_images.* FROM my_images 
JOIN (
SELECT MIN(ID) AS ID 
FROM my_images mi 
JOIN (
    SELECT 
    image_path, image_filename 
    FROM my_images 
    GROUP BY image_path, image_filename 
    HAVING COUNT(*) > 1 
) dups ON mi.image_path = dups.image_path AND mi.image_filename = dups.image_filename 
LEFT JOIN my_term_relationships mtr ON mi.ID = mtr.image_id 
WHERE mtr.image_id IS NULL 
GROUP BY mi.image_path, mi.image_filename 
HAVING COUNT(*) > 0 
) sq USING(ID); 

• sqlfiddle用delete語句：http://sqlfiddle.com/#!9/ed63de/1/0

編輯：還修復洛加爾提到的問題，DELETE語句前使用此UPDATE語句。

UPDATE my_term_relationships mtr 
JOIN (
    SELECT mi.ID, minID 
    FROM my_images mi 
    JOIN (
     SELECT 
     image_path, image_filename, MIN(ID) AS minID 
     FROM my_images 
     GROUP BY image_path, image_filename 
     HAVING COUNT(*) > 1 
    ) dups ON mi.image_path = dups.image_path AND mi.image_filename = dups.image_filename 
) sq ON mtr.image_id = sq.ID 
SET mtr.image_id = sq.minID; 

• 看到這一切在這個sqlfiddle一起工作：http://sqlfiddle.com/#!9/268539/1/0