2011-05-05 83 views
3

我有一個查詢可以快或慢,具體取決於我要提取多少條記錄。下面是顯示我的LIMIT條款,它需要執行查詢和獲取結果的相應時間的數量的表格:查詢的結果集太大

LIMIT | Seconds (Duration/Fetch) 
------+------------------------- 
    10 | 0.030/ 0.0 
    100 | 0.062/ 0.0 
1000 | 1.700/ 0.8 
10000 | 25.000/100.0 

正如你可以看到,它的罰款高達至少1000,但10,000真的很慢,這主要是由於獲取時間太長。我不明白爲什麼獲取時間的增長不是線性的,但是我從70多張表中抓取了200列,所以結果集需要很長時間才能獲取並不意外。

順便提一下,我提取的是某個銀行所有賬戶的數據。我正在處理的銀行擁有大約160,000個賬戶,因此我最終需要從數據庫中提取160,000行。

嘗試一次獲取160,000行顯然是不可行的(至少除非我能以某種方式顯着優化我的查詢)。在我看來,我可以合理地抓住的最大塊是1,000行,所以我寫了一個腳本,可以反覆運行查詢,限制和偏移量爲SELECT INTO OUTFILE。然後,最後,我將所有我傾倒的CSV文件和cat它們放在一起。它有效,但速度很慢。它需要幾小時。我的腳本現在正在運行,大約一個小時只能運行43,000行。

我應該在查詢優化級別攻擊這個問題還是長時間提示我應該把注意力集中在其他地方?你會推薦我做什麼?

如果你想查看查詢,你可以看到它here

+0

請顯示解釋選擇... – 2011-05-05 16:08:03

+0

老兄,你真的想要從SQL查詢中的字段中刪除銀行的名稱。 – smci 2011-11-23 00:00:29

+0

你有三個很好的答案值得選擇。你從來沒有回答Karelzarath的關鍵問題**'必須查詢真的返回215個字段中的每一個字段,並使用29個連接嗎?'**爲什麼不拿出一個簡單的30個字段的查詢,然後至少你可以測量每增加一個連接,性能都會下降。此外,您在STRAIGHT_JOIN DRapp上的表現數字是什麼給了你的? – smci 2011-11-23 00:10:04

回答

0

提取時間的非線性增加可能是密鑰緩衝區填滿的結果,也可能是其他內存相關問題。您應該使用EXPLAIN優化查詢以最大限度地利用索引,並調整您的MySQL服務器設置。

+0

好的,也許我應該配置MySQL使用盡可能多的內存,因爲我可以備用,所以我可以在合理的時間內獲取大量的行數? – 2011-05-05 15:55:17

+0

調整MySQL是一個棘手的問題。我會盡量調整各種相關設置並記錄下你的發現,直到你達到看起來令人滿意的改善程度。 – 2011-05-05 16:20:27

1

答案很大程度上取決於您對數據的處理方式。通過29個連接查詢215列對於非平凡的記錄大小決不會很快。

如果您嘗試向用戶顯示160,000條記錄,則應該對結果進行分頁,並且一次只取一頁。這將保持結果集足夠小,即使相對低效的查詢也會很快返回。在這種情況下,您還需要檢查用戶需要多少數據才能選擇或操作數據。有機會很好,你可以把它削減到一些領域和一些聚合(計數,總和等),讓用戶做出明智的決定,他們想要使用哪些記錄。使用帶偏移量的LIMIT可以拉出任意大小的單頁。

如果您需要導出數據用於報告目的,請確保您只需提取報告所需的確切數據。儘可能消除連接,並在需要聚合子數據的地方使用子查詢。您需要調整/添加常用連接和條件的索引。在提供查詢的情況下,ib.id以及您加入的衆多外鍵。您可以不使用布爾列,因爲沒有足夠的不同值來形成有意義的索引。

無論您嘗試完成什麼,刪除某些連接和列都會固有地加快您的處理速度。 MySQL需要做的事情來填補這個問題是你主要的絆腳石。

1

我重組了您的查詢,希望能夠提供顯着的性能改進時間。通過使用STRAIGHT_JOIN告訴MySQL按照你所說的順序(或者我在這裏調整過)來完成。最內層的第一個查詢「PreQuery」別名按照您的導入包和通用導入條件啓動,導入到帳戶的帳戶中...通過在那裏預先應用WHERE子句(以及您將要測試的內容,添加您的限制條款在這裏)你正在預先加入這些表格,並且在浪費任何時間試圖獲得客戶,地址等信息之前,將它們排除在外。在查詢中,我調整了連接/左連接以更好地顯示底層鏈接表的關係(主要針對其他人閱讀)。

正如另一個人指出的,我在PREQUERY中所做的工作可能是用於瀏覽和提供頁面的主查詢列表中的「Account.ID」記錄的基礎。我會很好奇你的現有表現,特別是在10,000限制範圍內。

PREQUERY獲得獨特的元素(包括下游,銀行,月份,年份和類別使用的帳戶ID),因此這些表不必在加入過程的其餘部分重新加入。

SELECT STRAIGHT_JOIN 
     PreQuery.*, 
     customer.customer_number, 
     customer.name, 
     customer.has_bad_address, 
     address.line1, 
     address.line2, 
     address.city, 
     state.name, 
     address.zip, 
     po_box.line1, 
     po_box.line2, 
     po_box.city, 
     po_state.name, 
     po_box.zip, 
     customer.date_of_birth, 
     northway_account.cffna, 
     northway_account.cfinsc, 
     customer.deceased, 
     customer.social_security_number, 
     customer.has_internet_banking, 
     customer.safe_deposit_box, 
     account.has_bill_pay, 
     account.has_e_statement, 
     branch.number, 
     northway_product.code, 
     macatawa_product.code, 
     account.account_number, 
     account.available_line, 
     view_macatawa_atm_card.number, 
     view_macatawa_debit_card.number, 
     uc.code use_class, 
     account.open_date, 
     account.balance, 
     account.affinion, 
     northway_account.ytdsc, 
     northway_account.ytdodf, 
     northway_account.ytdnsf, 
     northway_account.rtckcy, 
     northway_account.rtckwy, 
     northway_account.odwvey, 
     northway_account.ytdscw, 
     northway_account.feeytd, 
     customer.do_not_mail, 
     northway_account.aledq1, 
     northway_account.aledq2, 
     northway_account.aledq3, 
     northway_account.aledq4, 
     northway_account.acolq1, 
     northway_account.acolq2, 
     northway_account.acolq3, 
     northway_account.acolq4, 
     o.officer_number, 
     northway_account.avg_bal_1, 
     northway_account.avg_bal_2, 
     northway_account.avg_bal_3, 
     account.maturity_date, 
     account.interest_rate, 
     northway_account.asslc, 
     northway_account.paidlc, 
     northway_account.lnuchg, 
     northway_account.ytdlc, 
     northway_account.extfee, 
     northway_account.penamt, 
     northway_account.cdytdwaive, 
     northway_account.cdterm, 
     northway_account.cdtcod, 
     account.date_of_last_statement, 
     northway_account.statement_cycle, 
     northway_account.cfna1, 
     northway_account.cfna2, 
     northway_account.cfna3, 
     northway_account.cfna4, 
     northway_account.cfcity, 
     northway_account.cfstate, 
     northway_account.cfzip, 
     northway_account.actype, 
     northway_account.sccode, 
     macatawa_account.account_type_code, 
     macatawa_account.account_type_code_description, 
     macatawa_account.advance_code, 
     macatawa_account.amount_last_advance, 
     macatawa_account.amount_last_payment, 
     macatawa_account.available_credit, 
     macatawa_account.balance_last_statement, 
     macatawa_account.billing_day, 
     macatawa_account.birthday_3, 
     macatawa_account.birthday_name_2, 
     macatawa_account.ceiling_rate, 
     macatawa_account.class_code, 
     macatawa_account.classified_doubtful, 
     macatawa_account.classified_loss, 
     macatawa_account.classified_special, 
     macatawa_account.classified_substandard, 
     macatawa_account.closed_account_flag, 
     macatawa_account.closing_balance, 
     macatawa_account.compounding_code, 
     macatawa_account.cost_center_full, 
     macatawa_account.cytd_aggregate_balance, 
     macatawa_account.cytd_amount_of_advances, 
     macatawa_account.cytd_amount_of_payments, 
     macatawa_account.cytd_average_balance, 
     macatawa_account.cytd_average_principal_balance, 
     macatawa_account.cytd_interest_paid, 
     macatawa_account.cytd_number_items_nsf, 
     macatawa_account.cytd_number_of_advanes, 
     macatawa_account.cytd_number_of_payments, 
     macatawa_account.cytd_number_times_od, 
     macatawa_account.cytd_other_charges, 
     macatawa_account.cytd_other_charges_waived, 
     macatawa_account.cytd_reporting_points, 
     macatawa_account.cytd_service_charge, 
     macatawa_account.cytd_service_charge_waived, 
     macatawa_account.date_closed, 
     macatawa_account.date_last_activity, 
     macatawa_account.date_last_advance, 
     macatawa_account.date_last_payment, 
     macatawa_account.date_paid_off, 
     macatawa_account.ddl_code, 
     macatawa_account.deposit_rate_index, 
     macatawa_account.employee_officer_director_full_desc, 
     macatawa_account.floor_rate, 
     macatawa_account.handling_code, 
     macatawa_account.how_paid_code, 
     macatawa_account.interest_frequency, 
     macatawa_account.ira_plan, 
     macatawa_account.load_rate_code, 
     macatawa_account.loan_rate_code, 
     macatawa_account.loan_rating_code, 
     macatawa_account.loan_rating_code_1_full_desc, 
     macatawa_account.loan_rating_code_2_full_desc, 
     macatawa_account.loan_rating_code_3_full_desc, 
     macatawa_account.loan_to_value_ratio, 
     macatawa_account.maximum_credit, 
     macatawa_account.miscellaneous_code_full_desc, 
     macatawa_account.months_to_maturity, 
     macatawa_account.msa_code, 
     macatawa_account.mtd_agg_available_balance, 
     macatawa_account.naics_code, 
     macatawa_account.name_2, 
     macatawa_account.name_3, 
     macatawa_account.name_line, 
     macatawa_account.name_line_2, 
     macatawa_account.name_line_3, 
     macatawa_account.name_line_1, 
     macatawa_account.net_payoff, 
     macatawa_account.opened_by_responsibility_code_full, 
     macatawa_account.original_issue_date, 
     macatawa_account.original_maturity_date, 
     macatawa_account.original_note_amount, 
     macatawa_account.original_note_date, 
     macatawa_account.original_prepaid_fees, 
     macatawa_account.participation_placed_code, 
     macatawa_account.participation_priority_code, 
     macatawa_account.pay_to_account, 
     macatawa_account.payment_code, 
     macatawa_account.payoff_principal_balance, 
     macatawa_account.percent_participated_code, 
     macatawa_account.pmtd_number_deposit_type_1, 
     macatawa_account.pmtd_number_deposit_type_2, 
     macatawa_account.pmtd_number_deposit_type_3, 
     macatawa_account.pmtd_number_type_1, 
     macatawa_account.pmtd_number_type_2, 
     macatawa_account.pmtd_number_type_6, 
     macatawa_account.pmtd_number_type_8, 
     macatawa_account.pmtd_number_type_9, 
     macatawa_account.principal, 
     macatawa_account.purpose_code, 
     macatawa_account.purpose_code_full_desc, 
     macatawa_account.pytd_number_of_items_nsf, 
     macatawa_account.pytd_number_of_times_od, 
     macatawa_account.rate_adjuster, 
     macatawa_account.rate_over_split, 
     macatawa_account.rate_under_split, 
     macatawa_account.renewal_code, 
     macatawa_account.renewal_date, 
     macatawa_account.responsibility_code_full, 
     macatawa_account.secured_unsecured_code, 
     macatawa_account.short_first_name_1, 
     macatawa_account.short_first_name_2, 
     macatawa_account.short_first_name_3, 
     macatawa_account.short_last_name_1, 
     macatawa_account.short_last_name_2, 
     macatawa_account.short_last_name_3, 
     macatawa_account.statement_cycle, 
     macatawa_account.statement_rate, 
     macatawa_account.status_code, 
     macatawa_account.tax_id_number_name_2, 
     macatawa_account.tax_id_number_name_3, 
     macatawa_account.teller_alert_1, 
     macatawa_account.teller_alert_2, 
     macatawa_account.teller_alert_3, 
     macatawa_account.term, 
     macatawa_account.term_code, 
     macatawa_account.times_past_due_01_29, 
     macatawa_account.times_past_due_01_to_29_days, 
     macatawa_account.times_past_due_30_59, 
     macatawa_account.times_past_due_30_to_59_days, 
     macatawa_account.times_past_due_60_89, 
     macatawa_account.times_past_due_60_to_89_days, 
     macatawa_account.times_past_due_over_90, 
     macatawa_account.times_past_due_over_90_days, 
     macatawa_account.tin_code_name_1, 
     macatawa_account.tin_code_name, 
     macatawa_account.tin_code_name_2, 
     macatawa_account.tin_code_name_3, 
     macatawa_account.total_amount_past_due, 
     macatawa_account.waiver_od_charge, 
     macatawa_account.waiver_od_charge_description, 
     macatawa_account.waiver_service_charge_code, 
     macatawa_account.waiver_transfer_advance_fee, 
     macatawa_account.short_first_name, 
     macatawa_account.short_last_name   
FROM 
    (SELECT STRAIGHT_JOIN DISTINCT 
     b.name bank, 
     ib.YEAR, 
     ib.MONTH, 
     ip.category, 
     Account.ID 
     FROM import_bundle ib 
      JOIN generic_import gi ON ib.id = gi.import_bundle_id 
       JOIN account_import AI ON gi.id = ai.generic_import_id 
        JOIN Account ON AI.ID = account.account_import_id 
       JOIN import_profile ip ON gi.import_profile_id = ip.id 
      JOIN bank b ib.Bank_ID = b.id 
     WHERE 
      IB.ID = 95 
     AND IB.Active = 1 
     AND GI.Active = 1 
     LIMIT 1000) PreQuery 
    JOIN Account on PreQuery.ID = Account.ID 
     JOIN Customer on Account.Customer_ID = Customer.ID 
     JOIN Officer on Account.Officer_ID = Officer.ID 
     LEFT JOIN branch ON Account.branch_id = branch.id 
     LEFT JOIN cd_type ON account.cd_type_id = cd_type.id 
     LEFT JOIN use_class uc ON account.use_class_id = uc.id 
     LEFT JOIN account_type at ON account.account_type_id = at.id 
     LEFT JOIN northway_account ON account.id = northway_account.account_id 
     LEFT JOIN macatawa_account ON account.id = macatawa_account.account_id 
     LEFT JOIN view_macatawa_debit_card ON account.id = view_macatawa_debit_card.account_id 
     LEFT JOIN view_macatawa_atm_card ON account.id = view_macatawa_atm_card.account_id 
     LEFT JOIN original_address OA ON Account.ID = OA.account_id 

     JOIN Account_Address AA ON Account.ID = AA.account_id 
     JOIN address ON AA.address_id = address.id 
      JOIN state ON address.state_id = state.id 

     LEFT JOIN Account_po_box APB ON Account.ID = APB.account_id 
     LEFT JOIN address po_box ON APB.address_id = po_box.id 
      LEFT JOIN state po_state ON po_box.state_id = po_state.id 

     LEFT JOIN Account_macatawa_product amp ON account.id = amp.account_id 
     LEFT JOIN macatawa_product ON amp.macatawa_product_id = macatawa_product.id 
      LEFT JOIN product_type pt ON macatawa_product.product_type_id = pt.id 
      LEFT JOIN harte_hanks_service_category hhsc ON macatawa_product.harte_hanks_service_category_id = hhsc.id 
      LEFT JOIN core_file_type cft ON macatawa_product.core_file_type_id = cft.id 

     LEFT JOIN Account_northway_product anp ON account.id = anp.account_id 
     LEFT JOIN northway_product ON anp.northway_product_id = northway_product.id