使用PHP重複檢查將大型CSV文件導入到依賴MySQL表中

Question

我正在閱讀約50,000+個.csv行並創建插入查詢。插入查詢值中有一個動態值。它來自SELECT，如果存在或INSERT，如果不存在並且獲取插入的ID。這只是測試目的，只是忽略了安全缺乏。

$multiquery = ""; 
$lines = file($furl); 

foreach ($lines as $line_num => $line) { 
    if (($line_num + 1) % 1000 == 0) { 
     include('connection.php'); 
    } 

    $cols = split(';', $line); 

    $originid = 1; 
    $dest = $cols[3]; 
    $cost = (int)$cols[5]; 

    // === start: consume a lot of connections === 
    $query = "SELECT id FROM dests WHERE name = '$dest'"; 
    if (!$dests = mysqli_query($link, $query)) { 
     die(json_encode(array("errmsg" => "Selecting existing shipdest. Error: ".mysqli_error($link)))); 
    } 
    if (mysqli_num_rows($dests) > 0) { 
     $dest = mysqli_fetch_assoc($dests); 
    } 
    else { 
     $query = "INSERT INTO dests (name) VALUES ('$dest')"; 
     if (!mysqli_query($link, $query)) { 
      die(json_encode(array("errmsg" => "Inserting new dest."))); 
     } 

     $dest['id'] = mysqli_insert_id($link); 
    } 
    // === end: consume a lot of connections === 

    $multiquery .= "INSERT INTO packages (id_origin, id_dest, cost) VALUES ($originid, ".$dest['id'].", ".$cost."); "; 

    if (($line_num + 1) % 1000 == 0 && !mysqli_multi_query($link, $multiquery)) { 
     die(json_encode(array("errmsg" => "Failed at line ".$line_num))); 
    } 
} 

如何將該PHP塊合併爲$ multiquery中的id_dest值？

Answer 1

我結束了這段代碼：

ini_set('max_execution_time', 900); // 15 minutes 

$lines = file($furl); 

$multiquery = ""; 
foreach ($lines as $line_num => $line) { 
    $cols = split(';', $line); 
    $dest = $cols[3]; 

    $multiquery .= "INSERT IGNORE INTO dests (name) VALUES ('$dest'); "; 

    if (($line_num + 1) % 1000 == 0) { 
     include('connection.php'); 
     if (!mysqli_multi_query($link, $multiquery)) { 
      die(json_encode(array("errmsg" => "Failed at line ".$line_num." Error: ".mysqli_error($link)))); 
     } 
     $multiquery = ""; 
    } 
} 
include('connection.php'); 
if (!mysqli_multi_query($link, $multiquery)) { 
    die(json_encode(array("errmsg" => "Failed at line ".$line_num." Error: ".mysqli_error($link)))); 
} 

$multiquery = ""; 
foreach ($lines as $line_num => $line) { 
    $cols = split(';', $line); 

    $originid = 1; 
    $dest = $cols[3]; 
    $cost = (int)$cols[5]; 

    $multiquery .= "INSERT INTO packages (id_origin, id_dest, cost) VALUES ($originid, (SELECT id FROM dests WHERE name = '$dest'), ".$cost."); "; 

    include('connection.php'); 
    if (!mysqli_multi_query($link, $multiquery)) { 
     die(json_encode(array("errmsg" => "Failed at line ".$line_num." Error: ".mysqli_error($link)))); 
    } 
    $multiquery = ""; 
} 
include('connection.php'); 
if (!mysqli_multi_query($link, $multiquery)) { 
    die(json_encode(array("errmsg" => "Failed at line ".$line_num." Error: ".mysqli_error($link)))); 
}