如何將大的CSV文件導入到MySQL中

我的web應用程序接收逗號分隔的電話列表，該列表必須導入到「聯繫人」表中。該列表可以包含100,000到1,000,000個項目。我實現了存儲過程。但它對我來說仍然工作得太慢。你能幫我改進嗎？如何將大的CSV文件導入到MySQL中

DELIMITER $$ 

CREATE DEFINER=`root`@`localhost` PROCEDURE `import_phones` 
(IN ContactGroupId int, IN strPhones mediumtext, OUT total int, OUT inserted int) 
main: BEGIN 
    declare delim varchar(1) default(','); 
    declare delimPtr int unsigned default(0); -- pointer to current spliter position 
    declare startPtr int unsigned default(1); 
    declare phone1 nvarchar(20); 

    set total = 0; -- counter of total rows 
    set inserted = 0; -- counter of inserted rows 

    if strPhones is null or length(strPhones) < 1 then 
     leave main; 
    end if; 

    drop table if exists insertphones; 
    create temporary table insertphones(phone nvarchar(20)) 
     engine = memory; 
/*** 
-- split strPhones by delimiter 
*/ 
    SET delimPtr = locate(delim, strPhones,startPtr); 
    loop_label: while(delimPtr > 0) do 
     insert into insertphones (phone) values (substring(strPhones,startPtr,delimPtr-startPtr)); 
     -- select delimPtr,startPtr, substring(strPhones,startPtr,delimPtr-startPtr); 
     set startPtr = delimPtr+1; 
     set delimPtr = locate(delim, strPhones,startPtr); 
    end while; 
    if delimPtr = 0 then 
     insert into insertphones (phone) values (substring(strPhones,startPtr,delimPtr-startPtr)); 
    end if; 

    -- select delimPtr,startPtr; 
    select count(*) from insertphones into total; 
/*** 
-- insert phones 
*/ 

    insert into contacts (Phone, ContactGroupId) 
     select distinct(phone), ContactGroupId from insertphones where 
     phone not in (select Phone from contacts where ContactGroupId = ContactGroupId); 
    SELECT ROW_COUNT() into inserted ; 
    -- select total, inserted; 
END

來源

2013-03-09 Roman Podlinov

這屬於在[代碼審查（http://codereview.stackexchange.com/）... – Clive 2013-03-09 12:57:47

克萊夫，其實我並不需要代碼審查，但它可以顯著提高性能方向 – 2013-03-09 18:37:25

這是也是CodeReview.SE所做的。 – Ryan 2013-03-09 20:27:09

確定。該解決方案比初始版本更快地工作（大約10倍-20倍）。

$tempFile = tempnam(sys_get_temp_dir(), 'phones'); 
$tempTable = 'tmp'.$contactGroupId.time(); 
file_put_contents($tempFile, $phones); 
try { 
    chmod($tempFile, 0777); //required for MySQL import 
    $sql = sprintf(
"set @total = 0; 
set @inserted = 0; 
drop table if exists %s; 
create temporary table %s(phone nvarchar(20)) engine = memory; 
LOAD DATA INFILE '%s' INTO TABLE %s FIELDS TERMINATED BY '' LINES TERMINATED BY ','; 
SELECT count(*) FROM %s INTO @total; 
DELETE FROM %s WHERE length(phone) < 10; 
UPDATE %s SET phone = CONCAT('7',phone) WHERE length(phone) = 10; 
UPDATE %s SET phone = INSERT(phone,1,1,'7') WHERE phone like '8%%' and length(phone) = 11; 
insert into contacts (Phone, ContactGroupId) select distinct(phone), %d from %s where phone not in (select Phone from contacts where ContactGroupId = %d); 
SELECT ROW_COUNT() into @inserted;" 
    , $tempTable, $tempTable, str_replace('\\','\\\\',$tempFile), $tempTable, $tempTable, $tempTable, $tempTable, $tempTable, $contactGroupId, $tempTable, $contactGroupId); 

$command = Yii::app()->db->createCommand($sql); 
$command->execute(); 
$command = false; 
$result = Yii::app()->db->createCommand("select @total as total, @inserted as inserted")->queryRow(); 
$contactCounter['all'] += $result['total']; 
$contactCounter['created'] += $result['inserted'];

來源

2013-03-10 08:53:22

思考load data infile，像

load data infile 'filename' into tablename lines terminated by ','

來源

2013-03-09 13:22:51 maniek

謝謝你的回答。小記：以'，'結尾的行要求以''結尾的字段 – 2013-03-10 08:47:45

如何將大的CSV文件導入到MySQL中

回答

相關問題