如何處理/優化數千個不同的執行SELECT查詢？

Question

我需要爲數千行同步兩個數據庫（一個mysql，另一個是遠程託管的SQL Server數據庫）之間的特定信息。當我執行這個PHP文件時，它會在幾分鐘後卡住/超時，所以我想知道如何解決這個問題，也許還優化了「同步」它的方式。

哪些代碼需要做的：

基本上我想在我的數據庫中的每一行（=一個賬戶），這被更新 - 從另一個SQL Server的兩個信息的某些片段（= 2個SELECT查詢）數據庫。因此，我使用foreach循環爲每行創建2個SQL查詢，然後將這些信息更新爲該行的2列。我們討論需要運行這個foreach循環的〜10k行。

我的想法可能有幫助嗎？

我聽說過PDO Transactions應該收集所有這些查詢，然後在所有SELECT查詢的包中發送它們，但我不知道我是否正確使用它們，或者他們是否甚至在這種情況下提供幫助。

這是我當前的代碼，這是幾分鐘後超時：

UPDATE Gifting 
SET 
    refsInserted=(SELECT COUNT(USERNAME) 
        FROM accounts 
        WHERE referral=Gifting.refId), 
    refsCompleted=(SELECT COUNT(USERNAME) 
        FROM accounts 
        WHERE referral=Gifting.refId 
         AND finished=1) 

一個相關：

// DBH => MSSQL DB | DB => MySQL DB 
$dbh->beginTransaction(); 
// Get all referral IDs which needs to be updated: 
$listAccounts = "SELECT * FROM Gifting WHERE refsCompleted <= 100 ORDER BY idGifting ASC"; 
$ps_listAccounts = $db->prepare($listAccounts); 
$ps_listAccounts->execute(); 

foreach($ps_listAccounts as $row) { 
    $refid=$row['refId']; 
    // Refsinserted 
    $refsInserted = "SELECT count(username) as done FROM accounts WHERE referral='$refid'"; 
    $ps_refsInserted = $dbh->prepare($refsInserted); 
    $ps_refsInserted->execute(); 
    $row = $ps_refsInserted->fetch(); 
    $refsInserted = $row['done']; 

    // Refscompleted 
    $refsCompleted = "SELECT count(username) as done FROM accounts WHERE referral='$refid' AND finished=1"; 
    $ps_refsCompleted = $dbh->prepare($refsCompleted); 
    $ps_refsCompleted->execute(); 
    $row2 = $ps_refsCompleted->fetch(); 
    $refsCompleted = $row2['done']; 

    // Update fields for local order db 
    $updateGifting = "UPDATE Gifting SET refsInserted = :refsInserted, refsCompleted = :refsCompleted WHERE refId = :refId"; 
    $ps_updateGifting = $db->prepare($updateGifting); 

    $ps_updateGifting->bindParam(':refsInserted', $refsInserted); 
    $ps_updateGifting->bindParam(':refsCompleted', $refsCompleted); 
    $ps_updateGifting->bindParam(':refId', $refid); 
    $ps_updateGifting->execute(); 
    echo "$refid: $refsInserted Refs inserted/$refsCompleted Refs completed<br>"; 
} 

$dbh->commit(); 

Answer 1

您可以用相關子查詢做到這一切在一個查詢子查詢基本上使用引用父查詢的子查詢（查詢內的查詢）。因此請注意，在每個子查詢中，我引用每個子查詢的where子句中的Gifting.refId列。雖然這不是最好的性能，因爲每個子查詢仍然需要獨立於其他查詢運行，所以它會比您在那裏執行得更好（可能和您將要獲得的一樣好）。

編輯：

僅供參考。我不知道交易是否會對此有所幫助。通常，當您有多個相互依賴的查詢時，會使用它們，並在出現故障時爲您提供回滾方式。例如，銀行交易。您不希望餘額扣除一些金額，直到購買已經插入。如果購買由於某種原因無法插入，您希望將更改回滾到餘額。因此，在插入購買時，您開始一個交易，運行更新餘額查詢和插入購買查詢，並且只有在兩者都正確進入並且已經驗證您承諾保存的情況下。

EDIT2：

如果我這樣做，沒有做一個出口/進口，這是我會做什麼。儘管如此，這還是有一些假設。首先是您使用的是mssql 2008或更新版本，其次是推薦ID始終是一個數字。我還使用了一個將數字插入的臨時表，因爲您可以使用單個查詢輕鬆插入多行，然後運行單個更新查詢來更新贈送表。該臨時表遵循結構CREATE TABLE tempTable (refId int, done int, total int)。

//get list of referral accounts 
//if you are using one column, only query for one column 
$listAccounts = "SELECT DISTINCT refId FROM Gifting WHERE refsCompleted <= 100 ORDER BY idGifting ASC"; 
$ps_listAccounts = $db->prepare($listAccounts); 
$ps_listAccounts->execute(); 

//loop over and get list of refIds from above. 
$refIds = array(); 
foreach($ps_listAccounts as $row){ 
    $refIds[] = $row['refId']; 
} 


if(count($refIds) > 0){ 
    //implode into string for use in query below 
    $refIds = implode(',',$refIds); 

    //select out total count 
    $totalCount = "SELECT referral, COUNT(username) AS cnt FROM accounts WHERE referral IN ($refIds) GROUP BY referral"; 
    $ps_totalCounts = $dbh->prepare($totalCount); 
    $ps_totalCounts->execute(); 

    //add to array of counts 
    $counts = array(); 

    //loop over total counts 
    foreach($ps_totalCounts as $row){ 
     //if referral id not found, add it 
     if(!isset($counts[$row['referral']])){ 
      $counts[$row['referral']] = array('total'=>0,'done'=>0); 
     } 
     //add to count 
     $counts[$row['referral']]['total'] += $row['cnt']; 
    } 

    $doneCount = "SELECT referral, COUNT(username) AS cnt FROM accounts WHERE finished=1 AND referral IN ($refIds) GROUP BY referral"; 
    $ps_doneCounts = $dbh->prepare($doneCount); 
    $ps_doneCounts->execute(); 

    //loop over total counts 
    foreach($ps_totalCounts as $row){ 
     //if referral id not found, add it 
     if(!isset($counts[$row['referral']])){ 
      $counts[$row['referral']] = array('total'=>0,'done'=>0); 
     } 
     //add to count 
     $counts[$row['referral']]['done'] += $row['cnt']; 
    } 

    //now loop over counts and generate insert queries to a temp table. 
    //I suggest using a temp table because you can insert multiple rows 
    //in one query and then the update is one query. 
    $sqlInsertList = array(); 
    foreach($count as $refId=>$count){ 
     $sqlInsertList[] = "({$refId}, {$count['done']}, {$count['total']})"; 
    } 

    //clear out the temp table first so we are only inserting new rows 
    $truncSql = "TRUNCATE TABLE tempTable"; 
    $ps_trunc = $db->prepare($truncSql); 
    $ps_trunc->execute(); 

    //make insert sql with multiple insert rows 
    $insertSql = "INSERT INTO tempTable (refId, done, total) VALUES ".implode(',',$sqlInsertList); 
    //prepare sql for insert into mssql 
    $ps_insert = $db->prepare($insertSql); 
    $ps_insert->execute(); 

    //sql to update existing rows 
    $updateSql = "UPDATE Gifting 
        SET refsInserted=(SELECT total FROM tempTable WHERE refId=Gifting.refId), 
         refsCompleted=(SELECT done FROM tempTable WHERE refId=Gifting.refId) 
        WHERE refId IN (SELECT refId FROM tempTable) 
         AND refsCompleted <= 100"; 
    $ps_update = $db->prepare($updateSql); 
    $ps_update->execute(); 
} else { 
    echo "There were no reference ids found from \$dbh"; 
}