爲什麼這個SQLite查詢速度慢並且變慢？

Question

我有一個長期的數據記錄服務，每個服務產生一天包含一天數據的文件。我正在將文件加載到Windows窗體應用程序中的SQLite數據庫中。將文件中的數據插入到數據庫中的過程包括兩個查詢，其結果將在後續插入中使用。

Using SQLconnect As New SQLite.SQLiteConnection("Data Source=" & fn & ";") 
SQLconnect.Open() 
Using SQLcommand As SQLite.SQLiteCommand = SQLconnect.CreateCommand 
    Dim SqlTrans As System.Data.SQLite.SQLiteTransaction = SQLconnect.BeginTransaction 
    For Each Path As String In paths 
     fs = System.IO.File.Open(Path, IO.FileMode.Open, IO.FileAccess.Read, IO.FileShare.Read) 'Open file 
     Do While ReadFromStoreFile(fs, dt, Sent) = True 'Read a Timestamp/sentence pair 
      'Create a positions table for this MMSI if one doesn't already exist 
      SQLcommand.CommandText = "CREATE TABLE IF NOT EXISTS MMSI" & msg.MMSI & " (PosID INTEGER PRIMARY KEY AUTOINCREMENT, Date NUMERIC, Lat REAL, Lon REAL, Status INTEGER, SOG REAL, COG INTEGER, HDG INTEGER, VoyageID INTEGER);" 
      SQLcommand.ExecuteNonQuery() 

      Select Case msg.Type 'Dynamic position report 

       Case AIS.MsgType.PosRptClsA 

        '###THIS QUERY TAKES 20 secs per file (day) and increases 3 seconds per day! 
        SQLcommand.CommandText = "SELECT * FROM Voyages WHERE MMSI = " & msg.MMSI & " ORDER BY VoyageID DESC LIMIT 1" 'still the same 
        SQLreader = SQLcommand.ExecuteReader() 
        SQLreader.Read() 
        VID = SQLreader.Item(0) 
        SQLreader.Close() 

        SQLcommand.CommandText = "INSERT INTO MMSI" & msg.MMSI & " (Date, Lat, Lon, Status, SOG, COG, HDG, VoyageID) VALUES (" & ts & ", " & msg.Latitude & ", " & msg.Longitude & ", " & msg.NavStatus & ", " & SOG & ", " & COG & ", " & HDG & ", " & VID & ")" 
        SQLcommand.ExecuteNonQuery() 
        SQLreader.Close() 

       Case AIS.MsgType.StatAndVge 

        'Find the latest entry for this MMSI in the Voyages table 
        '###THIS QUERY takes 3 secs for same number of queries and does NOT increase 
        SQLcommand.CommandText = "SELECT * FROM Voyages WHERE MMSI = " & msg.MMSI & " ORDER BY VoyageID DESC LIMIT 1" 
        SQLreader = SQLcommand.ExecuteReader() 
        SQLreader.Read() 

        Dim NoVoyage As Boolean = Not (SQLreader.HasRows) 
        If Not NoVoyage Then 
         'If the data has changed, add a new entry 
         If Not (SQLreader.Item(2) = msg.Length) Then Changed = True 
         If Not (SQLreader.Item(3) = msg.Breadth) Then Changed = True 
         If Not (SQLreader.Item(4) = msg.Draught) Then Changed = True 
         If Not (SQLreader.Item(5) = msg.Destination) Then Changed = True 
         If Not (SQLreader.Item(6) = msg.ETA.Ticks) Then Changed = True 
         VoyageID = SQLreader.Item(0) 
        End If 
        SQLreader.Close() 

        If Changed Or NoVoyage Then 
         Changed = False 'reset flag 
         SQLcommand.CommandText = "INSERT INTO Voyages (Date, Length, Breadth, Draught, Destination, ETA, MMSI) VALUES (" & ts & ", " & msg.Length & ", " & msg.Breadth & ", " & msg.Draught & ", '" & msg.Destination.Replace("'", "''") & "', " & msg.ETA.Ticks & ", " & msg.MMSI_int & ")" 
         SQLcommand.ExecuteNonQuery() 

         SQLcommand.CommandText = "SELECT last_insert_rowid() FROM Voyages" 
         SQLreader = SQLcommand.ExecuteReader() 
         SQLreader.Read() 
         VoyageID = SQLreader.Item(0) 
         SQLreader.Close() 

        End If 
      End Select 'message type 
     Loop 'Read next entry from file 
     fs.Close() 'Close the file 

     'Write this file into the files table, so we know it has been written to the DB 
     fileinf = New System.IO.FileInfo(Path) 
     SQLcommand.CommandText = "INSERT OR REPLACE INTO Files (Name, Size, Date) VALUES ('" & fileinf.Name & "', '" & fileinf.Length & "', '" & fileinf.LastWriteTimeUtc.Ticks & "')" 
     SQLcommand.ExecuteNonQuery() 

    Next 'The next path in the list of paths to decode 

    SqlTrans.Commit() 'End of all files reached, commit all the changes to the DB 

End Using 'SQLcommand 
End Using 'SQLconnect 

如在碼所指示，第一查詢花費很長的時間和（更重要的），其數據被加載到DB中的持續時間增加。當在數據庫中添加21天的數據時，該查詢每天的累積時間大約爲20秒，並且每增加一天就會增加大約3秒。真奇怪的是，第二個查詢（對我來說看起來是一樣的）很快（對於相同數量的查詢大約需要3秒），並且不會隨着更多數據的添加而增加。

下面是創建空數據庫的功能：

Public Function CreateDB(fn As String, Force As Boolean) As Boolean 

    If System.IO.File.Exists(fn) Then 
     If Force Then 
      System.IO.File.Delete(fn) 'Delete the old DB and create a new one 
     Else 
      Return True 'DB alrewady exists so just return true 
     End If 
    End If 


    Using SQLconnect As New SQLite.SQLiteConnection 
     SQLconnect.ConnectionString = "Data Source=" & fn & ";" 
     SQLconnect.Open() 

     'Create Tables 
     Using SQLcommand As SQLite.SQLiteCommand = SQLconnect.CreateCommand 

      'Set page size 
      SQLcommand.CommandText = "PRAGMA Page_size = 4096;" 
      SQLcommand.ExecuteNonQuery() 

      'Set journalling mode to off 
      SQLcommand.CommandText = "PRAGMA journal_mode = OFF;" 
      SQLcommand.ExecuteNonQuery() 

      'Set auto indexing off 
      SQLcommand.CommandText = "PRAGMA automatic_index = false;" 
      SQLcommand.ExecuteNonQuery() 

      'Create Vessels Table 
      SQLcommand.CommandText = "CREATE TABLE Vessels(MMSI TEXT PRIMARY KEY, Name TEXT, Type INTEGER, IMO TEXT, CallSign TEXT, MothershipMMSI INTEGER, LastVoyageID INTEGER);" 
      SQLcommand.ExecuteNonQuery() 

      'Create Voyages Table 
      SQLcommand.CommandText = "CREATE TABLE Voyages(VoyageID INTEGER PRIMARY KEY AUTOINCREMENT, Date NUMERIC, Length INTEGER, Breadth INTEGER, Draught INTEGER, Destination TEXT, ETA NUMERIC, MMSI INTEGER);" 
      SQLcommand.ExecuteNonQuery() 

      'Create Meta Table 
      SQLcommand.CommandText = "CREATE TABLE Files(Name TEXT PRIMARY KEY, Size NUMERIC, Date NUMERIC);" 
      SQLcommand.ExecuteNonQuery() 

     End Using 'SQLcommand 

    End Using ' SQLconnect 

    Return True 

End Function 

這可能是造成第一查詢這麼慢，相比於第二查詢，需要更長的時間隨着更多的數據添加到數據庫？

SQlite和System.Data.Sqlite是最新版本。

Answer 1

假設msg由ReadFromStoreFile改變，則該查詢是否有用於給定MMSI更航行

"SELECT * FROM Voyages WHERE MMSI = " & msg.MMSI & " ORDER BY VoyageID DESC LIMIT 1" 

速度會變慢。所以我假設具有AIS.MsgType.PosRptClsA的MMSI值比其他MMSI更頻繁地插入。

它似乎查詢獲取MMSI的最大航程ID。你可以直接用這個做更多

"SELECT max(VoyageID) FROM Voyages WHERE MMSI = " & msg.MMSI 

我不知道這樣會跑得更快。 或者，您可以保留一個MMSI和最大航程id的字典，並在您插入航行以消除查詢時進行更新。