技巧優化這個.NET履帶式算法

Question

我寫的東西就像一個網絡爬蟲，它的引擎遵循下列步驟操作：

1. 閱讀RSS鏈接（參數）
2. 定義（中）RSS項目列表
3. 由一個單獨的查詢

4. 如果鏈接是新的話，會由一個單獨的查詢插入字段DB檢查數據庫（SQL Server）的各環節存在

   Public Sub MyTickHandler() 
       Dim NewItems As New List(Of Structures.RSSItem) 
       Dim founded As Boolean = False 
   
       NewItems = RssReader.ParseRssFile(RssURL) 
   
       Dim connString = Configs.NewsDBConnection 
       Dim myConnection As SqlConnection = New SqlConnection("Server=localhost;Database=db;Integrated Security=SSPI;;Connection Timeout=45;Max Pool Size= 300") 
       myConnection.Open() 
   
       For Each item In NewItems 
        Dim cmdString As String = "SELECT id FROM posts with (nolock) WHERE link LIKE '" & item.link.Trim.ToLower & "'" 
        Dim TheCommand As SqlCommand = New SqlCommand(cmdString, myConnection) 
        Dim result = TheCommand.ExecuteScalar() 
        If result Is Nothing Then 
         TheCommand = New SqlCommand("INSERT INTO posts (link) VALUES ('" & item.link.ToLower.Trim & "')") 
         TheCommand.Connection = myConnection 
         TheCommand.ExecuteNonQuery() 
   
         TheCommand = New SqlCommand("INSERT INTO queue (link,descrip,site,title,category) VALUES ('" & item.link.ToLower.Trim & "','" & StringToBase64(item.description) & "','" & RssSite & "','" & StringToBase64(item.title) & "','" & RssCategory & "')") 
         TheCommand.Connection = myConnection 
         TheCommand.ExecuteNonQuery() 
        End If 
        TheCommand.Dispose() 
       Next 
   
       myConnection.Close() 
       myConnection.Dispose() 
       SqlConnection.ClearPool(myConnection) 
   
   End Sub 
   

這適用於單一呼叫。
但我有一些關於150 Rss鏈接我應該通過線程每隔2分鐘檢查它們中的每一個，所以通過增加SQL查詢的裝載，這個過程以及sql server不會響應和應用程序崩潰！ ！

我嘗試了一些提示，如增加sql服務器響應超時，但它根本沒有幫助。

這個過程有什麼更好的方法或提示？
感謝

Answer 1

• 只做一單取，外面的for-each循環：

SELECT id, link FROM posts with (nolock) WHERE link in (@listOfLowerCaseLinks)

Dim myListOfLinks As New List(Of String) 
... 
TheCommand.Parameters.AddWithValue("@listOfLowerCaseLinks", myListOfLinks) 

• 裹插入的整體的動作（全換每個循環）到一個sql事務中。這樣，數據庫將不必在中間提交。

Answer 2

我建議你傳遞一個表值參數給這個任務的存儲過程。這將允許整個列表被插入到一個調用中。以下是您可以調整實際列長度的示例。在帖子表的鏈接列上有一個索引是非常重要的。我假設鏈接在這個例子中是唯一的。

T-SQL創建表類型和PROC：

CREATE TYPE dbo.linkInfo AS TABLE(
    link varchar(255) NOT NULL PRIMARY KEY 
    ,descrip varchar(255) 
    ,title varchar(255) 
    ); 
GO 

ALTER PROC dbo.usp_InsertRssItems 
    @site varchar(255) 
    ,@category varchar(255) 
    ,@linkInfo dbo.linkInfo READONLY 
AS 

SET NOCOUNT ON; 

DECLARE @InsertedPosts TABLE(link varchar(255)); 

INSERT INTO dbo.posts(link) 
OUTPUT inserted.link INTO @InsertedPosts 
SELECT link 
FROM @linkInfo AS li 
WHERE NOT EXISTS(
    SELECT * 
    FROM dbo.posts AS p 
    WHERE p.link = li.link 
    ); 

INSERT INTO dbo.queue(link,descrip,site,title,category) 
SELECT li.link, li.descrip, @site,li. title, @category 
FROM @linkInfo AS li 
WHERE EXISTS(
    SELECT * 
    FROM @InsertedPosts AS ip 
    WHERE ip.link = li.link 
    ); 
GO 

樣品VB.NET代碼：

Sub MyTickHandler() 

    Dim NewItems As New List(Of Structures.RssItem) 
    Dim founded As Boolean = False 

    NewItems = RssReader.ParseRssFile(RssURL) 

    Dim dt = getNewRssItemDataTable(NewItems) 

    Dim connString = Configs.NewsDBConnection 
    Dim myConnection As SqlConnection = New SqlConnection("Server=localhost;Database=db;Integrated Security=SSPI;;Connection Timeout=45;Max Pool Size= 300") 
    Dim TheCommand As SqlCommand = New SqlCommand("dbo.usp_InsertRssItems", myConnection) 
    TheCommand.Parameters.Add(New SqlParameter("@site", SqlDbType.VarChar, 255)).Value = "z" 
    TheCommand.Parameters.Add(New SqlParameter("@category", SqlDbType.VarChar, 255)).Value = "z" 
    TheCommand.Parameters.Add(New SqlParameter("@linkInfo", SqlDbType.Structured)).Value = dt 
    TheCommand.CommandType = CommandType.StoredProcedure 

    myConnection.Open() 
    TheCommand.ExecuteNonQuery() 

    myConnection.Close() 
    myConnection.Dispose() 

End Sub 

Private Function getNewRssItemDataTable(NewRssItems As List(Of Structures.RssItem)) As DataTable 

    Dim dt As New DataTable 
    dt.Columns.Add("link", GetType(String)).MaxLength = 255 
    dt.Columns.Add("descrip", GetType(String)).MaxLength = 255 
    dt.Columns.Add("title", GetType(String)).MaxLength = 255 

    For Each NewRssItem In NewRssItems 
     Dim row = dt.NewRow 
     dt.Rows.Add(row) 
     row(0) = NewRssItem.link 
     row(1) = NewRssItem.description 
     row(2) = NewRssItem.title 

    Next NewRssItem 

    Return dt 

End Function 

編輯：

我看到你提到你想一個SqlBulkCopy示例。如果插入是無條件的，您可以使用此技術：

Sub executeBulkInsert(connectionString As String, site As String, category As String, NewRssItems As List(Of Structures.RssItem)) 

    Dim dt As New DataTable 

    dt.Columns.Add("link", GetType(String)).MaxLength = 255 
    dt.Columns.Add("descrip", GetType(String)).MaxLength = 255 
    dt.Columns.Add("site", GetType(String)).MaxLength = 255 
    dt.Columns.Add("title", GetType(String)).MaxLength = 255 
    dt.Columns.Add("category", GetType(String)).MaxLength = 255 

    For Each NewRssItem In NewRssItems 
     Dim row = dt.NewRow 
     dt.Rows.Add(row) 
     row(0) = site 
     row(1) = category 
     row(2) = NewRssItem.link 
     row(3) = NewRssItem.description 
     row(4) = NewRssItem.title 

    Next NewRssItem 

    Dim bcp = New SqlBulkCopy(connectionString) 
    bcp.DestinationTableName = "dbo.queue" 

    bcp.WriteToServer(dt) 

End Sub