如何優化由SQLAlchemy生成的查詢？

我有一個由SQLAlchemy ORM生成的查詢。它應該檢索特定課程的stream_items及其所有部分 - 資源，內容文本塊等，以及發佈它們的用戶。但是，此查詢似乎非常緩慢，我們的生產數據庫需要數分鐘，數據庫中有20,000個左右的用戶，25個左右的stream_items用於該課程，並且每個stream_item有幾個內容文本塊。請注意，除了數據庫中的用戶之外，其他任何記錄的數量都很少，因爲我們導入了一堆用戶，但內容很少。如何優化由SQLAlchemy生成的查詢？

編輯：請注意，每個對象ID都是franklin_object表中的外鍵。

我試圖尋找查詢，並已經確定了幾個令人不安位（看解釋輸出）

其中一個查找的是「使用臨時的;使用filesort'。
用戶表中沒有索引
內容文本塊表，沒有指數創下兩次

但是打了兩拳，我真的不知道該怎麼做這些，尤其是後兩個的問題。

下面是該查詢：

SELECT stream_item.id        AS stream_item_id, 
     franklin_object.id       AS franklin_object_id, 
     franklin_object.type       AS franklin_object_type, 
     franklin_object.uuid       AS franklin_object_uuid, 
     stream_item.parent_id      AS stream_item_parent_id, 
     stream_item.shown_at       AS stream_item_shown_at, 
     stream_item.author_id      AS stream_item_author_id, 
     stream_item.stream_sort_at     AS stream_item_stream_sort_at, 
     stream_item.PUBLIC       AS stream_item_public, 
     stream_item.created_at      AS stream_item_created_at, 
     stream_item.updated_at      AS stream_item_updated_at, 
     anon_1.content_text_block_text    AS anon_1_content_text_block_text, 
     anon_2.resource_id       AS anon_2_resource_id, 
     anon_2.franklin_object_id     AS anon_2_franklin_object_id, 
     anon_2.franklin_object_type     AS anon_2_franklin_object_type, 
     anon_2.franklin_object_uuid     AS anon_2_franklin_object_uuid, 
     anon_2.resource_top_parent_resource   AS anon_2_resource_top_parent_resource, 
     anon_2.resource_top_parent_id    AS anon_2_resource_top_parent_id, 
     anon_2.resource_title      AS anon_2_resource_title, 
     anon_2.resource_url       AS anon_2_resource_url, 
     anon_2.resource_image      AS anon_2_resource_image, 
     anon_2.resource_created_at     AS anon_2_resource_created_at, 
     anon_2.resource_updated_at     AS anon_2_resource_updated_at, 
     franklin_object_1.id       AS franklin_object_1_id, 
     franklin_object_1.type      AS franklin_object_1_type, 
     franklin_object_1.uuid      AS franklin_object_1_uuid, 
     anon_1.content_text_block_id     AS anon_1_content_text_block_id, 
     anon_1.franklin_object_id     AS anon_1_franklin_object_id, 
     anon_1.franklin_object_type     AS anon_1_franklin_object_type, 
     anon_1.franklin_object_uuid     AS anon_1_franklin_object_uuid, 
     anon_1.content_text_block_position   AS anon_1_content_text_block_position, 
     anon_1.content_text_block_franklin_object_id AS anon_1_content_text_block_franklin_object_id, 
     anon_1.content_text_block_created_at   AS anon_1_content_text_block_created_at, 
     anon_1.content_text_block_updated_at   AS anon_1_content_text_block_updated_at, 
     anon_3.user_password       AS anon_3_user_password, 
     anon_3.user_auth_token      AS anon_3_user_auth_token, 
     anon_3.user_id        AS anon_3_user_id, 
     anon_3.franklin_object_id     AS anon_3_franklin_object_id, 
     anon_3.franklin_object_type     AS anon_3_franklin_object_type, 
     anon_3.franklin_object_uuid     AS anon_3_franklin_object_uuid, 
     anon_3.user_email       AS anon_3_user_email, 
     anon_3.user_auth_token_expiration   AS anon_3_user_auth_token_expiration, 
     anon_3.user_active       AS anon_3_user_active, 
     anon_3.user_activation_token     AS anon_3_user_activation_token, 
     anon_3.user_first_name      AS anon_3_user_first_name, 
     anon_3.user_last_name      AS anon_3_user_last_name, 
     anon_3.user_image       AS anon_3_user_image, 
     anon_3.user_bio        AS anon_3_user_bio, 
     anon_3.user_aspirations      AS anon_3_user_aspirations, 
     anon_3.user_website       AS anon_3_user_website, 
     anon_3.user_resume       AS anon_3_user_resume, 
     anon_3.user_resume_name      AS anon_3_user_resume_name, 
     anon_3.user_primary_role      AS anon_3_user_primary_role, 
     anon_3.user_institution_id     AS anon_3_user_institution_id, 
     anon_3.user_birth_date      AS anon_3_user_birth_date, 
     anon_3.user_gender       AS anon_3_user_gender, 
     anon_3.user_graduation_year     AS anon_3_user_graduation_year, 
     anon_3.user_complete       AS anon_3_user_complete, 
     anon_3.user_masthead_y_position    AS anon_3_user_masthead_y_position, 
     anon_3.user_masthead       AS anon_3_user_masthead, 
     anon_3.user_fb_access_token     AS anon_3_user_fb_access_token, 
     anon_3.user_fb_user_id      AS anon_3_user_fb_user_id, 
     anon_3.user_location       AS anon_3_user_location, 
     anon_3.user_created_at      AS anon_3_user_created_at, 
     anon_3.user_updated_at      AS anon_3_user_updated_at, 
     anon_4.content_text_block_text    AS anon_4_content_text_block_text, 
     anon_4.content_text_block_id     AS anon_4_content_text_block_id, 
     anon_4.franklin_object_id     AS anon_4_franklin_object_id, 
     anon_4.franklin_object_type     AS anon_4_franklin_object_type, 
     anon_4.franklin_object_uuid     AS anon_4_franklin_object_uuid, 
     anon_4.content_text_block_position   AS anon_4_content_text_block_position, 
     anon_4.content_text_block_franklin_object_id AS anon_4_content_text_block_franklin_object_id, 
     anon_4.content_text_block_created_at   AS anon_4_content_text_block_created_at, 
     anon_4.content_text_block_updated_at   AS anon_4_content_text_block_updated_at, 
     anon_5.user_password       AS anon_5_user_password, 
     anon_5.user_auth_token      AS anon_5_user_auth_token, 
     anon_5.user_id        AS anon_5_user_id, 
     anon_5.franklin_object_id     AS anon_5_franklin_object_id, 
     anon_5.franklin_object_type     AS anon_5_franklin_object_type, 
     anon_5.franklin_object_uuid     AS anon_5_franklin_object_uuid, 
     anon_5.user_email       AS anon_5_user_email, 
     anon_5.user_auth_token_expiration   AS anon_5_user_auth_token_expiration, 
     anon_5.user_active       AS anon_5_user_active, 
     anon_5.user_activation_token     AS anon_5_user_activation_token, 
     anon_5.user_first_name      AS anon_5_user_first_name, 
     anon_5.user_last_name      AS anon_5_user_last_name, 
     anon_5.user_image       AS anon_5_user_image, 
     anon_5.user_bio        AS anon_5_user_bio, 
     anon_5.user_aspirations      AS anon_5_user_aspirations, 
     anon_5.user_website       AS anon_5_user_website, 
     anon_5.user_resume       AS anon_5_user_resume, 
     anon_5.user_resume_name      AS anon_5_user_resume_name, 
     anon_5.user_primary_role      AS anon_5_user_primary_role, 
     anon_5.user_institution_id     AS anon_5_user_institution_id, 
     anon_5.user_birth_date      AS anon_5_user_birth_date, 
     anon_5.user_gender       AS anon_5_user_gender, 
     anon_5.user_graduation_year     AS anon_5_user_graduation_year, 
     anon_5.user_complete       AS anon_5_user_complete, 
     anon_5.user_masthead_y_position    AS anon_5_user_masthead_y_position, 
     anon_5.user_masthead       AS anon_5_user_masthead, 
     anon_5.user_fb_access_token     AS anon_5_user_fb_access_token, 
     anon_5.user_fb_user_id      AS anon_5_user_fb_user_id, 
     anon_5.user_location       AS anon_5_user_location, 
     anon_5.user_created_at      AS anon_5_user_created_at, 
     anon_5.user_updated_at      AS anon_5_user_updated_at, 
     anon_6.stream_item_id      AS anon_6_stream_item_id, 
     anon_6.franklin_object_id     AS anon_6_franklin_object_id, 
     anon_6.franklin_object_type     AS anon_6_franklin_object_type, 
     anon_6.franklin_object_uuid     AS anon_6_franklin_object_uuid, 
     anon_6.stream_item_parent_id     AS anon_6_stream_item_parent_id, 
     anon_6.stream_item_shown_at     AS anon_6_stream_item_shown_at, 
     anon_6.stream_item_author_id     AS anon_6_stream_item_author_id, 
     anon_6.stream_item_stream_sort_at   AS anon_6_stream_item_stream_sort_at, 
     anon_6.stream_item_public     AS anon_6_stream_item_public, 
     anon_6.stream_item_created_at    AS anon_6_stream_item_created_at, 
     anon_6.stream_item_updated_at    AS anon_6_stream_item_updated_at 
FROM franklin_object 
     INNER JOIN stream_item 
       ON franklin_object.id = stream_item.id 
     INNER JOIN (SELECT franklin_object.id     AS franklin_object_id, 
          franklin_object.type     AS franklin_object_type, 
          franklin_object.uuid     AS franklin_object_uuid, 
          content_text_block.id     AS content_text_block_id, 
          content_text_block.text    AS content_text_block_text, 
          content_text_block.position   AS content_text_block_position, 
          content_text_block.franklin_object_id AS content_text_block_franklin_object_id, 
          content_text_block.created_at   AS content_text_block_created_at, 
          content_text_block.updated_at   AS content_text_block_updated_at 
        FROM franklin_object 
          INNER JOIN content_text_block 
            ON franklin_object.id = content_text_block.id) AS anon_1 
       ON stream_item.id = anon_1.content_text_block_franklin_object_id 
     LEFT OUTER JOIN contents_resources AS contents_resources_1 
        ON anon_1.content_text_block_id = contents_resources_1.content_id 
     LEFT OUTER JOIN (SELECT franklin_object.id   AS franklin_object_id, 
           franklin_object.type   AS franklin_object_type, 
           franklin_object.uuid   AS franklin_object_uuid, 
           resource.id     AS resource_id, 
           resource.top_parent_resource AS resource_top_parent_resource, 
           resource.top_parent_id  AS resource_top_parent_id, 
           resource.title    AS resource_title, 
           resource.url     AS resource_url, 
           resource.image    AS resource_image, 
           resource.created_at   AS resource_created_at, 
           resource.updated_at   AS resource_updated_at 
         FROM franklin_object 
           INNER JOIN resource 
             ON franklin_object.id = resource.id) AS anon_2 
        ON anon_2.resource_id = contents_resources_1.resource_id 
     LEFT OUTER JOIN contents_franklin_objects AS contents_franklin_objects_1 
        ON anon_1.content_text_block_id = contents_franklin_objects_1.content_id 
     LEFT OUTER JOIN franklin_object AS franklin_object_1 
        ON franklin_object_1.id = contents_franklin_objects_1.franklin_object_id 
     LEFT OUTER JOIN likers AS likers_1 
        ON stream_item.id = likers_1.post_id 
     LEFT OUTER JOIN (SELECT franklin_object.id   AS franklin_object_id, 
           franklin_object.type  AS franklin_object_type, 
           franklin_object.uuid  AS franklin_object_uuid, 
           USER.id     AS user_id, 
           USER.email     AS user_email, 
           USER.password    AS user_password, 
           USER.auth_token   AS user_auth_token, 
           USER.auth_token_expiration AS user_auth_token_expiration, 
           USER.active    AS user_active, 
           USER.activation_token  AS user_activation_token, 
           USER.first_name   AS user_first_name, 
           USER.last_name    AS user_last_name, 
           USER.image     AS user_image, 
           USER.bio     AS user_bio, 
           USER.aspirations   AS user_aspirations, 
           USER.website    AS user_website, 
           USER.resume    AS user_resume, 
           USER.resume_name   AS user_resume_name, 
           USER.primary_role   AS user_primary_role, 
           USER.institution_id  AS user_institution_id, 
           USER.birth_date   AS user_birth_date, 
           USER.gender    AS user_gender, 
           USER.graduation_year  AS user_graduation_year, 
           USER.complete    AS user_complete, 
           USER.masthead_y_position AS user_masthead_y_position, 
           USER.masthead    AS user_masthead, 
           USER.fb_access_token  AS user_fb_access_token, 
           USER.fb_user_id   AS user_fb_user_id, 
           USER.location    AS user_location, 
           USER.created_at   AS user_created_at, 
           USER.updated_at   AS user_updated_at 
         FROM franklin_object 
           INNER JOIN USER 
             ON franklin_object.id = USER.id) AS anon_3 
        ON anon_3.user_id = likers_1.user_id 
     LEFT OUTER JOIN contents_franklin_objects AS contents_franklin_objects_2 
        ON franklin_object.id = contents_franklin_objects_2.franklin_object_id 
     LEFT OUTER JOIN (SELECT franklin_object.id     AS franklin_object_id, 
           franklin_object.type     AS franklin_object_type, 
           franklin_object.uuid     AS franklin_object_uuid, 
           content_text_block.id     AS content_text_block_id, 
           content_text_block.text    AS content_text_block_text, 
           content_text_block.position   AS content_text_block_position, 
           content_text_block.franklin_object_id AS content_text_block_franklin_object_id, 
           content_text_block.created_at   AS content_text_block_created_at, 
           content_text_block.updated_at   AS content_text_block_updated_at 
         FROM franklin_object 
           INNER JOIN content_text_block 
             ON franklin_object.id = content_text_block.id) AS anon_4 
        ON anon_4.content_text_block_id = contents_franklin_objects_2.content_id 
     LEFT OUTER JOIN (SELECT franklin_object.id   AS franklin_object_id, 
           franklin_object.type  AS franklin_object_type, 
           franklin_object.uuid  AS franklin_object_uuid, 
           stream_item.id    AS stream_item_id, 
           stream_item.parent_id  AS stream_item_parent_id, 
           stream_item.shown_at  AS stream_item_shown_at, 
           stream_item.author_id  AS stream_item_author_id, 
           stream_item.stream_sort_at AS stream_item_stream_sort_at, 
           stream_item.PUBLIC   AS stream_item_public, 
           stream_item.created_at  AS stream_item_created_at, 
           stream_item.updated_at  AS stream_item_updated_at 
         FROM franklin_object 
           INNER JOIN stream_item 
             ON franklin_object.id = stream_item.id) AS anon_6 
        ON anon_6.stream_item_parent_id = franklin_object.id 
     LEFT OUTER JOIN likers AS likers_2 
        ON anon_6.stream_item_id = likers_2.post_id 
     LEFT OUTER JOIN (SELECT franklin_object.id   AS franklin_object_id, 
           franklin_object.type  AS franklin_object_type, 
           franklin_object.uuid  AS franklin_object_uuid, 
           USER.id     AS user_id, 
           USER.email     AS user_email, 
           USER.password    AS user_password, 
           USER.auth_token   AS user_auth_token, 
           USER.auth_token_expiration AS user_auth_token_expiration, 
           USER.active    AS user_active, 
           USER.activation_token  AS user_activation_token, 
           USER.first_name   AS user_first_name, 
           USER.last_name    AS user_last_name, 
           USER.image     AS user_image, 
           USER.bio     AS user_bio, 
           USER.aspirations   AS user_aspirations, 
           USER.website    AS user_website, 
           USER.resume    AS user_resume, 
           USER.resume_name   AS user_resume_name, 
           USER.primary_role   AS user_primary_role, 
           USER.institution_id  AS user_institution_id, 
           USER.birth_date   AS user_birth_date, 
           USER.gender    AS user_gender, 
           USER.graduation_year  AS user_graduation_year, 
           USER.complete    AS user_complete, 
           USER.masthead_y_position AS user_masthead_y_position, 
           USER.masthead    AS user_masthead, 
           USER.fb_access_token  AS user_fb_access_token, 
           USER.fb_user_id   AS user_fb_user_id, 
           USER.location    AS user_location, 
           USER.created_at   AS user_created_at, 
           USER.updated_at   AS user_updated_at 
         FROM franklin_object 
           INNER JOIN USER 
             ON franklin_object.id = USER.id) AS anon_5 
        ON anon_5.user_id = likers_2.user_id 
WHERE stream_item.parent_id = 11 
ORDER BY stream_item.stream_sort_at DESC, 
      anon_1.content_text_block_position, 
      anon_6.stream_item_stream_sort_at DESC

並解釋輸出：

ID SELECT_TYPE TABLE POSSIBLY_KEYS KEY KEY_LEN REF ROWS EXTRA 
1 PRIMARY <derived2> ALL NULL NULL NULL NULL 599 Using  temporary; Using filesort 
1 PRIMARY stream_item eq_ref PRIMARY,parent_id PRIMARY 4 anon_1.content_text_block_franklin_object_id 1 Using where 
1 PRIMARY contents_resources_1 ref content_id content_id 5 anon_1.content_text_block_id 2 
1 PRIMARY <derived3> ALL NULL NULL NULL NULL 7 
1 PRIMARY contents_franklin_objects_1 ref content_id content_id 5 anon_1.content_text_block_id 1 
1 PRIMARY franklin_object eq_ref PRIMARY PRIMARY 4 franklin.stream_item.id 1 Using where 
1 PRIMARY franklin_object_1 eq_ref PRIMARY PRIMARY 4 franklin.contents_franklin_objects_1.franklin_object_id 1 
1 PRIMARY likers_1 ref post_id post_id 5 franklin.stream_item.id 1 
1 PRIMARY <derived4> ALL NULL NULL NULL NULL 136 
1 PRIMARY contents_franklin_objects_2 ref franklin_object_id franklin_object_id 5 franklin.stream_item.id 1 
1 PRIMARY <derived5> ALL NULL NULL NULL NULL 599 
1 PRIMARY <derived6> ALL NULL NULL NULL NULL 608 
1 PRIMARY likers_2 ref post_id post_id 5 anon_6.stream_item_id 1 
1 PRIMARY <derived7> ALL NULL NULL NULL NULL 136 
7 DERIVED user ALL PRIMARY NULL NULL NULL 133 
7 DERIVED franklin_object eq_ref PRIMARY PRIMARY 4 franklin.user.id 1 
6 DERIVED stream_item ALL PRIMARY NULL NULL NULL 709 
6 DERIVED franklin_object eq_ref PRIMARY PRIMARY 4 franklin.stream_item.id 1 
5 DERIVED content_text_block ALL PRIMARY NULL NULL NULL 666 
5 DERIVED franklin_object eq_ref PRIMARY PRIMARY 4 franklin.content_text_block.id  1 
4 DERIVED user ALL PRIMARY NULL NULL NULL 133 
4 DERIVED franklin_object eq_ref PRIMARY PRIMARY 4 franklin.user.id 1 
3 DERIVED resource ALL PRIMARY NULL NULL NULL 7 
3 DERIVED franklin_object eq_ref PRIMARY PRIMARY 4 franklin.resource.id 1 
2 DERIVED content_text_block ALL PRIMARY NULL NULL NULL 666 
2 DERIVED franklin_object eq_ref PRIMARY PRIMARY 4 franklin.content_text_block.id 1

如何降低所有查詢的東西更快？有什麼其他方法可以加快速度？

franklin_objects是否被設置爲反模式？它的工作方式是franklin_object表有兩列：id和type。然後每個類型都是一個表，主鍵是franklin_object中的外鍵。

生成的SQL代碼是沿着線的東西：

stream_item_query = StreamItem.query.options(db.joinedload('stream_items'),db.joinedload('contents_included_in'),db.joinedload('contents.resources'),db.joinedload('contents.objects'),db.subqueryload('likers'))

stream_items = stream_item_query.filter(StreamItem.parent_id == community_id).order_by(db.desc(StreamItem.stream_sort_at)).all()

來源

2012-07-16 Nicholas Meyer

添加上面的orm代碼。 – 2012-07-16 21:09:12

您的類是映射到表還是跨多個表選擇statemnts？連接有點奇怪l連接（從連接b選擇*）r'不是我期望的，'l連接b連接'' – SingleNegationElimination 2012-07-17 10:16:50

每個類都從franklin對象繼承（連接繼承）。 – 2012-07-17 15:06:10

哇，這個傷害我的大腦有點。試圖找出查詢在做什麼，所有表格是什麼，以及關係是單調乏味的。如果你有類似的經歷，那麼讓這是你可能試圖在這個單一查詢中做太多的第一個暗示。

我的建議是重新考慮你的整個方法。 SQLAlchemy是一個非常不錯的工具，我不打算（或者你選擇mysql），但是和大多數ORM工具一樣，你需要考慮它們的使用成本。一個例子是這個franklin_object表業務。這是一種反模式嗎？是的，沒有。從純粹的面向對象的角度來看，它是有意義的。您可以通過在此表中查找id來確定要查詢哪些表。從關係查詢的角度來看，它的用途很小。我可以從您的查詢中刪除franklin_object的每個實例，並且不會丟失任何內容，但是...從franklin_object的列中不會丟失任何內容。如果這是一個可行的選擇，我會馬上這樣做。

讓我們來進一步檢查這個與franklin_object的鏈接。縱觀子查詢，它們都具有相同的形式：

SELECT franklin_object.id   AS franklin_object_id, 
     franklin_object.type   AS franklin_object_type, 
     franklin_object.uuid   AS franklin_object_uuid, 
     linked_table.id    AS linked_table_id, 
     linked_table.col2   AS col2 --and more 
    FROM franklin_object 
    INNER JOIN linked_table 
     ON franklin_object.id = linked_table.id) AS anon_n

沒有多少信息數據庫，至於如何優化查詢的這部分下去，不管統計數據。也許如果franklin_object受限於在where子句中指定type，則查詢會更好。也許。

這對USER表尤其有問題，因爲這個表有很多記錄（所以你說）。由於您正在查詢大多數列，並且優化程序無法準確計算出將檢索多少行，因此執行全表掃描是有意義的。在你的情況，兩次。

另一方面是所涉及的連接的數量。如果我們取出所有franklin_object引用，則仍然有11個連接。如果您的數據模型更具關聯性，那並不糟糕，但事實並非如此。生成的查詢對數據庫沒有太大幫助，無法找出執行查詢的最佳方式，因此它不會很好。也許你可以通過提示等等來緩解這種情況，但我敢打賭，這會長期困擾你。

您使用的是ORM工具，所以真的使用它。通過一次性完成這麼大的查詢，您不會獲得任何收益。性能可能會有所分化。執行延遲檢索以避免龐大而複雜的查詢。我會說，試試看看它是怎麼回事，懶洋洋地去做一切事情。性能可能會好，我會說更好。不是很好，可能不是可以接受的，但是比數據庫攪動時能夠喝咖啡更好。

然後，開始拼湊成更精簡的塊。將邏輯上合理的對象連接在一起，如resource和contents_resources。另一個例子，stream_item,likers和user之間的連接是重複的。做一個查詢，讓SQLAlchemy做它的事情。

作爲最後的手段，可以實施某種緩存機制。也許在某處將表格非規範化。在緩慢變化的讀取系統中，您可以將這些表格輸入另一個結構，查詢可以直接快速進行。也就是說，要預先處理並將其存儲在一個表中。

好運

來源

2012-07-20 18:56:31

如何優化由SQLAlchemy生成的查詢？

回答

相關問題