我有2個innoDB表,其中一個有15166897個記錄,另一個有700000個記錄,有時候服務器上有負載時,基於這些表上的電子郵件地址的簡單聯接查詢花費了很多時間來執行一段時間很快就會執行。但是,這兩個表格都已正確編制索引。加入大表越來越慢
我們有大約800張桌子加入我們的數據庫,數據庫大小爲40GB。我們在AWS中運行t2.medium RDS實例,其中有4GB RAM,其中3GB分配給INNO DB BUFFER POOL。
我有以下問題,請對其建議。
- 將大表分解成小表是一個好主意,這樣一張表上的記錄就會變少,這是否會提高查詢性能?
- 整個表的大表是否會降低查詢的性能?
- 這是RDS配置足夠的數據庫,或應該需要改善配置?
編輯: 下面是我的表架構和索引的定義:
表1:總記錄數:7,00,000
CREATE TABLE IF NOT EXISTS `tbl_contact_master` (
`id` int(11) NOT NULL,
`first_name` varchar(60) NOT NULL,
`last_name` varchar(60) NOT NULL,
`email_address` varchar(250) NOT NULL,
`agency_name` varchar(150) NOT NULL,
`state` varchar(30) NOT NULL,
`zip_code` varchar(20) NOT NULL,
`srch_zip` varchar(20) NOT NULL,
`title` varchar(100) NOT NULL,
`street` varchar(255) NOT NULL,
`street2` varchar(255) NOT NULL,
`city` varchar(30) NOT NULL,
`country` varchar(30) NOT NULL,
`phone` varchar(20) NOT NULL,
`ext` varchar(20) NOT NULL,
`fax` varchar(20) NOT NULL,
`years_of_experience` varchar(50) DEFAULT NULL,
`owner_manager` varchar(100) NOT NULL,
`agency_type` varchar(100) NOT NULL,
`agency_sales` varchar(100) NOT NULL,
`agency_website` varchar(100) NOT NULL,
`agency_host` varchar(100) NOT NULL,
`agency_host_name` varchar(100) NOT NULL,
`agency_affiliation` varchar(100) NOT NULL,
`agent_sales` varchar(100) NOT NULL,
`id_number_type` varchar(16) NOT NULL,
`id_number` varchar(15) NOT NULL,
`destination1` varchar(100) NOT NULL,
`destination2` varchar(100) NOT NULL,
`destination3` varchar(100) NOT NULL,
`travel_type1` varchar(100) NOT NULL,
`travel_type2` varchar(100) NOT NULL,
`travel_type3` varchar(100) NOT NULL,
`travel_type4` varchar(100) NOT NULL,
`update_first_name` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_last_name` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_email_address` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_agency_name` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_state` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_zip_code` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_title` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_street` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_street2` datetime NOT NULL,
`update_city` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_country` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_phone` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_ext` datetime NOT NULL,
`update_fax` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_years_of_experience` datetime NOT NULL,
`update_owner_manager` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_agency_type` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_agency_sales` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_agency_website` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_agency_host` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_agency_host_name` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_agency_affiliation` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_agent_sales` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_id_number_type` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_id_number` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_destination1` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_destination2` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_destination3` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_travel_type1` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_travel_type2` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_travel_type3` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_travel_type4` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`create_status` tinyint(1) NOT NULL COMMENT '1 for website or 0 for add by csv',
`created_date` datetime NOT NULL,
`modified_date` datetime NOT NULL,
`submited_date` datetime NOT NULL,
`FirstPageDate` datetime NOT NULL,
`is_retired` tinyint(1) NOT NULL COMMENT '0 = No, 1 = Yes',
`update_is_retired` datetime NOT NULL,
`ip_address` varchar(30) NOT NULL,
`is_bademail` tinyint(1) NOT NULL,
`update_is_bademail` datetime NOT NULL,
`is_mice` tinyint(4) NOT NULL DEFAULT '0',
`update_is_mice` datetime NOT NULL,
`mice_company_disc` mediumint(5) unsigned NOT NULL,
`update_mice_company_disc` datetime NOT NULL,
`mice_meeting_region_plan` text NOT NULL,
`update_mice_meeting_region_plan` datetime NOT NULL,
`mice_ida` text NOT NULL,
`update_mice_ida` datetime NOT NULL,
`mice_associations` text NOT NULL,
`update_mice_associations` datetime NOT NULL,
`mice_meeting_regions` text NOT NULL,
`update_mice_meeting_regions` datetime NOT NULL,
`mice_average_attendance` mediumint(5) unsigned NOT NULL,
`update_mice_average_attendance` datetime NOT NULL,
`mice_annual_budget` mediumint(5) unsigned NOT NULL,
`update_mice_annual_budget` datetime NOT NULL,
`mice_facilities` text NOT NULL,
`update_mice_facilities` datetime NOT NULL,
`mice_annual_peak_rooms` mediumint(5) unsigned NOT NULL,
`update_mice_annual_peak_rooms` datetime NOT NULL,
`mice_nof_meetings` mediumint(5) unsigned NOT NULL,
`update_mice_nof_meetings` datetime NOT NULL,
`mice_job_resp_level` mediumint(5) unsigned NOT NULL,
`update_mice_job_resp_level` datetime NOT NULL,
`mice_event_resp_level` mediumint(5) unsigned NOT NULL,
`update_mice_event_resp_level` datetime NOT NULL,
`mice_is_planning` mediumint(5) unsigned NOT NULL,
`update_mice_is_planning` datetime NOT NULL,
`mice_experience` mediumint(5) unsigned NOT NULL,
`update_mice_experience` datetime NOT NULL,
`mice_primary_job` mediumint(5) unsigned NOT NULL,
`update_mice_primary_job` datetime NOT NULL,
`mice_company_size` mediumint(5) unsigned NOT NULL,
`update_mice_company_size` datetime NOT NULL,
`mice_primary_business` mediumint(5) unsigned NOT NULL,
`update_mice_primary_business` datetime NOT NULL,
`mice_primary_business_other` text NOT NULL,
`mice_event_specialty` text NOT NULL,
`update_mice_event_specialty` datetime NOT NULL,
`mice_created_date` datetime NOT NULL,
`mice_modified_date` datetime NOT NULL,
`mice_submitted_date` datetime NOT NULL,
`is_ta` tinyint(4) NOT NULL DEFAULT '0',
`update_is_ta` datetime NOT NULL,
`ta_created_date` datetime NOT NULL,
`ta_modified_date` datetime NOT NULL,
`ta_submitted_date` datetime NOT NULL,
`is_mice_retired` tinyint(4) NOT NULL DEFAULT '0',
`update_is_mice_retired` datetime NOT NULL,
`not_ta_by` tinyint(2) NOT NULL COMMENT '0 for default, 1 for superadmin, 2 for contact',
`update_not_ta_by` datetime NOT NULL,
`not_mice_by` tinyint(2) NOT NULL COMMENT '0 for default, 1 for superadmin, 2 for contact',
`update_not_mice_by` datetime NOT NULL
) ENGINE=InnoDB AUTO_INCREMENT=1096438 DEFAULT CHARSET=latin1;
--
-- Indexes for dumped tables
--
--
-- Indexes for table `tbl_contact_master`
--
ALTER TABLE `tbl_contact_master`
ADD PRIMARY KEY (`id`),
ADD KEY `email_address` (`email_address`),
ADD KEY `is_ta` (`is_ta`),
ADD KEY `mice_created_date` (`mice_created_date`),
ADD KEY `mice_modified_date` (`mice_modified_date`),
ADD KEY `mice_submitted_date` (`mice_submitted_date`),
ADD KEY `ta_created_date` (`ta_created_date`),
ADD KEY `ta_modified_date` (`ta_modified_date`),
ADD KEY `ta_submitted_date` (`ta_submitted_date`),
ADD KEY `is_mice_retired` (`is_mice_retired`),
ADD KEY `is_mice` (`is_mice`),
ADD KEY `is_bademail` (`is_bademail`),
ADD KEY `is_retired` (`is_retired`),
ADD KEY `created_date` (`created_date`),
ADD KEY `modified_date` (`modified_date`),
ADD KEY `submited_date` (`submited_date`),
ADD KEY `srch_zip` (`srch_zip`);
表2:
CREATE TABLE IF NOT EXISTS `tbl_master_partition` (
`contact_id` int(10) unsigned NOT NULL,
`inactive_group` varchar(200) DEFAULT NULL COMMENT '1=15,2=30,3=45,4=60',
`inactive_from` datetime DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
--
-- Indexes for dumped tables
--
--
-- Indexes for table `tbl_master_partition`
--
ALTER TABLE `tbl_master_partition`
ADD KEY `contact_id` (`contact_id`), ADD KEY `inactive_group` (`inactive_group`);
表3:總記錄數:14400000
CREATE TABLE IF NOT EXISTS `tbl_mandrill_email_mapping` (
`id` bigint(20) NOT NULL,
`log_id` int(11) NOT NULL,
`batch_id` varchar(30) COLLATE utf8_unicode_ci NOT NULL,
`m_id` varchar(50) COLLATE utf8_unicode_ci NOT NULL COMMENT 'unique mandrill message id',
`s_id` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`email` varchar(250) COLLATE utf8_unicode_ci NOT NULL,
`open` int(11) NOT NULL DEFAULT '0',
`click` int(11) NOT NULL DEFAULT '0',
`send` int(11) NOT NULL DEFAULT '0',
`hard_bounce` int(11) NOT NULL DEFAULT '0',
`soft_bounce` int(11) NOT NULL DEFAULT '0',
`reject` int(11) NOT NULL DEFAULT '0',
`spam` int(11) NOT NULL DEFAULT '0',
`unsub` int(11) NOT NULL DEFAULT '0'
) ENGINE=InnoDB AUTO_INCREMENT=15131814 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
--
-- Indexes for dumped tables
--
--
-- Indexes for table `tbl_mandrill_email_mapping`
--
ALTER TABLE `tbl_mandrill_email_mapping`
ADD PRIMARY KEY (`id`),
ADD KEY `m_id` (`m_id`),
ADD KEY `email` (`email`),
ADD KEY `hard_bounce` (`hard_bounce`),
ADD KEY `reject` (`reject`),
ADD KEY `open` (`open`),
ADD KEY `click` (`click`),
ADD KEY `send` (`send`),
ADD KEY `log_id` (`log_id`),
ADD KEY `batch_id` (`batch_id`),
ADD KEY `s_id` (`s_id`);
缺少的表添加
CREATE TABLE IF NOT EXISTS `tbl_contact_173_230` (
`id` int(11) NOT NULL,
`contact_id` int(11) NOT NULL,
`salutation` varchar(10) NOT NULL,
`first_name` varchar(60) NOT NULL,
`last_name` varchar(60) NOT NULL,
`email_address` varchar(250) NOT NULL,
`secondary_email_address` varchar(250) NOT NULL,
`agency_name` varchar(150) NOT NULL,
`state` varchar(30) NOT NULL,
`zip_code` varchar(20) NOT NULL,
`srch_zip` varchar(20) NOT NULL,
`title` varchar(255) NOT NULL,
`street` varchar(255) NOT NULL,
`street2` varchar(255) NOT NULL,
`city` varchar(30) NOT NULL,
`country` varchar(30) NOT NULL,
`phone` varchar(20) NOT NULL,
`ext` varchar(20) NOT NULL,
`fax` varchar(20) NOT NULL,
`owner_manager` varchar(140) DEFAULT NULL,
`years_of_experience` varchar(40) DEFAULT NULL,
`agency_type` varchar(140) NOT NULL,
`agency_sales` varchar(140) NOT NULL,
`agency_website` varchar(140) NOT NULL,
`agency_host` varchar(140) DEFAULT NULL,
`agency_host_name` varchar(140) NOT NULL,
`agency_affiliation` varchar(140) NOT NULL,
`agent_sales` varchar(140) NOT NULL,
`id_number_type` varchar(16) NOT NULL,
`id_number` varchar(15) NOT NULL,
`destination1` varchar(140) NOT NULL,
`destination2` varchar(140) NOT NULL,
`destination3` varchar(140) NOT NULL,
`travel_type1` varchar(140) NOT NULL,
`travel_type2` varchar(140) NOT NULL,
`travel_type3` varchar(140) NOT NULL,
`travel_type4` varchar(140) NOT NULL,
`update_first_name` datetime NOT NULL,
`update_last_name` datetime NOT NULL,
`update_email_address` datetime NOT NULL,
`update_agency_name` datetime NOT NULL,
`update_state` datetime NOT NULL,
`update_zip_code` datetime NOT NULL,
`update_title` datetime NOT NULL,
`update_street` datetime NOT NULL,
`update_street2` datetime NOT NULL,
`update_city` datetime NOT NULL,
`update_country` datetime NOT NULL,
`update_phone` datetime NOT NULL,
`update_ext` datetime NOT NULL,
`update_fax` datetime NOT NULL,
`update_owner_manager` datetime NOT NULL,
`update_years_of_experience` datetime NOT NULL,
`update_agency_type` datetime NOT NULL,
`update_agency_sales` datetime NOT NULL,
`update_agency_website` datetime NOT NULL,
`update_agency_host` datetime NOT NULL,
`update_agency_host_name` datetime NOT NULL,
`update_agency_affiliation` datetime NOT NULL,
`update_agent_sales` datetime NOT NULL,
`update_id_number_type` datetime NOT NULL,
`update_id_number` datetime NOT NULL,
`update_destination1` datetime NOT NULL,
`update_destination2` datetime NOT NULL,
`update_destination3` datetime NOT NULL,
`update_travel_type1` datetime NOT NULL,
`update_travel_type2` datetime NOT NULL,
`update_travel_type3` datetime NOT NULL,
`update_travel_type4` datetime NOT NULL,
`opt_in_marketing` varchar(10) DEFAULT NULL,
`travel_pro_user` varchar(10) DEFAULT NULL,
`create_status` tinyint(1) NOT NULL,
`crm_created_date` datetime NOT NULL,
`crm_modified_date` datetime NOT NULL,
`crm_submited_date` datetime NOT NULL,
`crm_sync_date` datetime NOT NULL,
`created_date` datetime NOT NULL,
`modified_date` datetime NOT NULL,
`submited_date` datetime NOT NULL,
`offer_entry` datetime NOT NULL,
`sync_date` datetime NOT NULL,
`crm_status` tinyint(1) NOT NULL,
`created_by` int(11) NOT NULL,
`is_retired` tinyint(1) NOT NULL,
`update_is_retired` datetime NOT NULL,
`is_bademail` tinyint(1) NOT NULL,
`update_is_bademail` datetime NOT NULL,
`is_ta` tinyint(4) NOT NULL DEFAULT '0',
`update_is_ta` datetime NOT NULL,
`ta_created_date` datetime NOT NULL,
`ta_modified_date` datetime NOT NULL,
`ta_submitted_date` datetime NOT NULL,
`mice_company_disc` text NOT NULL,
`update_mice_company_disc` datetime NOT NULL,
`mice_meeting_region_plan` text NOT NULL,
`update_mice_meeting_region_plan` datetime NOT NULL,
`mice_ida` text NOT NULL,
`update_mice_ida` datetime NOT NULL,
`mice_associations` text NOT NULL,
`update_mice_associations` datetime NOT NULL,
`mice_meeting_regions` text NOT NULL,
`update_mice_meeting_regions` datetime NOT NULL,
`mice_average_attendance` mediumint(5) unsigned NOT NULL,
`update_mice_average_attendance` datetime NOT NULL,
`mice_annual_budget` mediumint(5) unsigned NOT NULL,
`update_mice_annual_budget` datetime NOT NULL,
`mice_facilities` text NOT NULL,
`update_mice_facilities` datetime NOT NULL,
`mice_annual_peak_rooms` mediumint(5) unsigned NOT NULL,
`update_mice_annual_peak_rooms` datetime NOT NULL,
`mice_nof_meetings` mediumint(5) unsigned NOT NULL,
`update_mice_nof_meetings` datetime NOT NULL,
`mice_job_resp_level` mediumint(5) unsigned NOT NULL,
`update_mice_job_resp_level` datetime NOT NULL,
`mice_event_resp_level` mediumint(5) unsigned NOT NULL,
`update_mice_event_resp_level` datetime NOT NULL,
`mice_is_planning` mediumint(5) unsigned NOT NULL,
`update_mice_is_planning` datetime NOT NULL,
`mice_experience` mediumint(5) unsigned NOT NULL,
`update_mice_experience` datetime NOT NULL,
`mice_primary_job` mediumint(5) unsigned NOT NULL,
`update_mice_primary_job` datetime NOT NULL,
`mice_company_size` mediumint(5) unsigned NOT NULL,
`update_mice_company_size` datetime NOT NULL,
`mice_primary_business` mediumint(5) unsigned NOT NULL,
`mice_primary_business_other` text NOT NULL,
`update_mice_primary_business` datetime NOT NULL,
`mice_event_specialty` text NOT NULL,
`update_mice_event_specialty` datetime NOT NULL,
`is_mice` tinyint(4) NOT NULL DEFAULT '0',
`update_is_mice` datetime NOT NULL,
`mice_created_date` datetime NOT NULL,
`mice_modified_date` datetime NOT NULL,
`mice_submitted_date` datetime NOT NULL,
`is_mice_retired` tinyint(4) NOT NULL DEFAULT '0',
`update_is_mice_retired` datetime NOT NULL,
`recurring_opt_out` tinyint(1) unsigned NOT NULL DEFAULT '0',
`custom_648` varchar(255) NOT NULL,
`custom_649` varchar(255) NOT NULL,
`custom_650` varchar(255) NOT NULL,
`custom_717` varchar(255) NOT NULL,
`custom_718` varchar(255) NOT NULL,
`custom_719` varchar(255) NOT NULL,
`custom_1369` varchar(255) NOT NULL,
`custom_1454` varchar(255) NOT NULL,
`custom_1455` varchar(255) NOT NULL,
`custom_1469` varchar(255) NOT NULL
) ENGINE=InnoDB AUTO_INCREMENT=103603 DEFAULT CHARSET=latin1;
--
-- Indexes for table `tbl_contact_173_230`
--
ALTER TABLE `tbl_contact_173_230`
ADD PRIMARY KEY (`id`),
ADD KEY `email_address` (`email_address`),
ADD KEY `contact_id` (`contact_id`),
ADD KEY `zip_code` (`zip_code`),
ADD KEY `srch_zip` (`srch_zip`),
ADD KEY `ta_submitted_date` (`ta_submitted_date`);
下面是其越來越慢或有時只是卡住了一個小時的查詢。
1.
SELECT cm.id AS contact_id, mem.email, mem.open
FROM tbl_mandrill_email_mapping AS mem
JOIN tbl_contact_master AS cm ON cm.email_address = mem.email
WHERE mem.log_id = XXXXX
2.
SELECT COUNT(crm.id) as total_record, SUM(UPPER(opt_in_marketing)='NO') as unsub_email_count,
SUM(is_bademail=1) as invalid_email_count, COUNT(DISTINCT mp.`contact_id`) as inactive_email_count,
SUM(is_retired=1
OR is_mice_retired=1
) as retired_count
FROM (`tbl_contact_173_230` AS crm)
INNER JOIN `tbl_mandrill_email_mapping` as mem
ON `mem`.`email` = `crm`.`email_address`
LEFT JOIN`tbl_master_partition` AS mp
ON `mp`.`contact_id` = `crm`.`contact_id`
AND crm.`ta_submitted_date` < '2016-10-19'
AND FIND_IN_SET(5, mp.`inactive_group`) > 0
WHERE `crm`.`email_address` != '' AND`mem`.`log_id` = 'xxxx'
AND `mem`.`open` = 0
AND `mem`.`hard_bounce` = 0
AND `mem`.`reject` = 0
AND `mem`.`spam` = 0
ORDER BY `crm`.`id` ASC
編輯注: 我有通知說,它主要是創建一個問題,當有非常頻繁的更新/上tbl_mandrill_email_mapping
表寫入。這是一個處理電子郵件事件的日誌表(發送,打開,點擊,反彈等)。在我的情況下,一旦發送大量電子郵件(大約100,000封電子郵件),並且它開始處理所有以上事件,此表會變得非常忙碌,有時這種事件非常頻繁。在這種情況下,引用tbl_mandrill_email_mapping
的所有查詢都卡住了。
任何建議來處理這種情況?我正在考慮只讀副本,以便所有更新/插入將由不同的服務器處理,並通過不同的服務器讀取句柄,它會起作用嗎?或者是否有任何擊球手解決方案來處理這種情況?
我打開任何其他建議,請建議。
我剛剛編輯了問題並添加了表格定義和查詢,請仔細研究一下。我已經將RAM增加到了8GB,並且爲'innodb_buffer_pool_size'分配了6GB,但在加載時仍然具有相同的性能。 – Irk
@Ricky James,謝謝!非常有用的信息。我會尋找額外的索引,並刪除如果不利用。我有缺少的表格,你看看它。 – Irk
@瑞奇詹姆斯,我已經在底部添加了額外的註釋,請儘管去吧。 – Irk