VB.Net導入CSV文件到SQL數據庫

Question

所以，我需要幫助搞清楚如何使這項工作，並希望開始與其他VB專家的對話。

我有一個CSV文件每小時轉儲到一個導入文件夾。我編寫了一個掃描該文件夾的VB應用程序，獲取CSV文件並將其導入到SQL數據庫中。那裏沒有問題。

嘗試查詢SQL中的數據時出現問題。所有我需要的是一個領域的重要數據，這裏的字段內容修改的例子：

Successful Write by DOMAIN\USERLOGIN on /SOMENETWORKSHARE/FOLDER/MyFolder/USERLOGIN/DesktopBackUp/log.txt 

所以基本上我需要一種方法來分割上述行到多個領域，它需要發生在SQL導入的時間。

Successful Write「是事件」通過「不改變以往可作爲一個分隔符」

DOMAIN/USERLOGIN字段1

「DOMAIN不改變但用戶登陸將」字段2

在「不改變以往任何時候都可以用作分隔符」

/SOMENETWORKSHARE/FOLDER/MyFolder/USERLOGIN/DesktopBackUp/「這是文件路徑」欄3

log.txt「已修改的文件名」字段4

有時候沒有文件名，只是一個文件路徑。

我很樂於提供建議，但是，CSV文件不會更改，我無法更改文件傳遞給我的方式。

在此先感謝。

這裏是數據庫結構

SET ANSI_NULLS ON 
GO 
SET QUOTED_IDENTIFIER ON 
GO 
SET ANSI_PADDING ON 
GO 
CREATE TABLE [dbo].[SACL](
    [Unique ID] [varchar](max) NULL, 
    [Event ID] [varchar](max) NULL, 
    [Event Time] [datetime] NULL, 
    [Severity] [varchar](max) NULL, 
    [Workspace] [varchar](max) NULL, 
    [Headline] [varchar](max) NULL, 
    [ID] [int] IDENTITY(1,1) NOT NULL, 
    [Event] [varchar](max) NULL, 
    [Username] [varchar](max) NULL, 
    [Path] [varchar](max) NULL, 
    [Filename] [varchar](max) NULL, 
PRIMARY KEY CLUSTERED 
(
    [ID] ASC 
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] 
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY] 
GO 
SET ANSI_PADDING OFF 
GO 

Answer 1

我希望這可以幫助你。在調用此函數後，可以使用結果將數據庫中其他表中的每個字段包含在其中。

Public Function divideThis(CompleteString As String) 

    Dim n As Integer = InStr(CompleteString, " by ") 
    Dim Field1 As String = CompleteString.Substring(0, n - 1) 

    Dim m As Integer = InStr(CompleteString, " on ") 
    Dim Field2 As String = CompleteString.Substring(n + 3, m - n - 4) 

    Dim o As Integer = InStrRev(CompleteString, "/") 
    Dim Field3 As String = CompleteString.Substring(m + 3, o - m - 3) 

    Dim Field4 As String = CompleteString.Substring(o, CompleteString.Length - o) 

    Dim fragmented As String() = {Field1, Field2, Field3, Field4} 

    Return (fragmented) 

End Function 

Answer 2

確定試試這個...它會導入CSV文件，拆分所需的字段併合並它。

我修改了代碼以在LINQ語句中包含RegEx來拆分該字段，如果沒有文件名，它將工作，但是在那裏必須有一個正斜槓並且這必須是唯一字段（在你的CSV文件中），其中包含正斜槓...原因是，如果正斜槓丟失，並且沒有文件名，你將丟失一個字段，或者如果csv文件中的其他字段包含斜槓.....

Sub Main() 
    Dim regex As Text.RegularExpressions.Regex = New Text.RegularExpressions.Regex("\d+") 
    Dim readText() As String = File.ReadAllLines("<Path to your CSV file>") 

    Using connection As SqlConnection = New SqlConnection("<DB connection string>") 

     Dim command As SqlCommand = connection.CreateCommand() 
     connection.Open() 

     command.CommandType = System.Data.CommandType.StoredProcedure 
     command.CommandText = "SP_InsertCSVData" 

     For i As Integer = 0 To readText.Count 
      Dim readRecord() As String = readText.ToArray 

      Dim CSVData As XElement = New XElement("Root", 
          From str In readText 
          Let fields = Text.RegularExpressions.Regex.Replace(str, "(by|on|\/(?=[^\/]*$))", ",").Split(",") 
          Select New XElement("CSVDataRecord", 
           New XAttribute("Field_1", fields(0)), 
           New XAttribute("Field_2", fields(1)), 
           New XAttribute("Field_3", fields(2)), 
           New XAttribute("Field_4", fields(3)), 
           New XAttribute("Field_5", fields(4)), 
           New XAttribute("Field_6", fields(5)), 
           New XAttribute("Field_7", fields(6)), 
           New XAttribute("Field_8", fields(7)) 
          ) 
         ) 

      command.Parameters.Clear() 
      command.Parameters.Add(New SqlParameter With 
      { 
       .ParameterName = "@InputXML", 
       .DbType = DbType.Xml, 
       .Value = CSVData.CreateReader 
      }) 

      command.ExecuteNonQuery() 

     Next 
    End Using 

End Sub 

這裏是新表...

/****** Object: Table [dbo].[CSV_Import_Table] Script Date: 06/02/2016 01:55:32 ******/ 
SET ANSI_NULLS ON 
GO 

SET QUOTED_IDENTIFIER ON 
GO 

CREATE TABLE [dbo].[CSV_Import_Table]( 
[Field_1] [nvarchar](max) NULL, 
[Field_2] [nvarchar](max) NULL, 
[Field_3] [nvarchar](max) NULL, 
[Field_4] [nvarchar](max) NULL, 
[Field_5] [nvarchar](max) NULL, 
[Field_6] [nvarchar](max) NULL, 
[Field_7] [nvarchar](max) NULL, 
[Field_8] [nvarchar](max) NULL 
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY] 

GO 

這裏是SP ...

/****** Object: StoredProcedure [dbo].[SP_InsertCSVData] Script Date: 06/02/2016 01:56:20 ******/ 
SET ANSI_NULLS ON 
GO 
SET QUOTED_IDENTIFIER ON 
GO 
CREATE PROCEDURE [dbo].[SP_InsertCSVData] (@InputXML xml) 
as 
begin 
MERGE CSV_Import_Table AS main 
USING (select Row.id.value('@Field_1','[nvarchar](MAX)') as Field_1, Row.id.value('@Field_2','[nvarchar](MAX)') as Field_2, Row.id.value('@Field_3','[nvarchar](MAX)') as Field_3, Row.id.value('@Field_4','[nvarchar](MAX)') as Field_4, Row.id.value('@Field_5','[nvarchar](MAX)') as Field_5, Row.id.value('@Field_6','[nvarchar](MAX)') as Field_6, Row.id.value('@Field_7','[nvarchar](MAX)') as Field_7, Row.id.value('@Field_8','[nvarchar](MAX)') as Field_8 
      from @InputXML.nodes('/Root/CSVDataRecord') as Row(id)) as stage 
ON main.Field_1=stage.Field_1 
WHEN MATCHED THEN 
    UPDATE SET main.Field_1=stage.Field_1, main.Field_2=stage.Field_2, main.Field_3=stage.Field_3, main.Field_4=stage.Field_4 , main.Field_5=stage.Field_5 , main.Field_6=stage.Field_6, main.Field_7=stage.Field_7, main.Field_8=stage.Field_8  
WHEN NOT MATCHED THEN 
    INSERT (Field_1, Field_2, Field_3, Field_4, Field_5, Field_6, Field_7, Field_8) VALUES (stage.Field_1, stage.Field_2, stage.Field_3, stage.Field_4, stage.Field_5, stage.Field_6, stage.Field_7, stage.Field_8); 
end 

而且我猜你CSV文件看起來是這樣的....

記錄1場1，通過DOMAIN \ USERLOGIN成功的寫上/ SOMENETWORKSHARE /文件夾/ MyFolder中/ USERLOGIN/DesktopBackUp /Log.txt，字段6，字段7，字段8

Answer 3

下面是一個選擇，你應該可以在你的表上運行來測試。

更換FROM (....) as tbl位與真表

你可以看到，在選擇的表情是相當醜陋，笨拙等都有想想你是否幸福維持這種

有一個WHERE在阻止無效數據進入的最終目標。如果缺少分隔符的數據進入，那麼索引將最終爲負數，並且會出現各種錯誤。

有了這些東西，數據質量始終是一個問題。因此，我們不希望通過假設該域永遠具有相同的名稱，或者假設您只會從一個域導入數據。

所以你需要對你的桌子運行這個，看它是否有效。如果是的話，下一步是：

• 把文件名從路徑
• 改變成一個更新

這應該是簡單的

SELECT 
SRC, 
LEFT(SRC,CHARINDEX(' by ',SRC)) AS Operation, 
SUBSTRING(
    SRC, 
    CHARINDEX(' by ',SRC)+4, 
    CHARINDEX(' on ',SRC) - CHARINDEX(' by ',SRC)-4 
    ) AS LoginName, 

RIGHT(
    SRC, 
    LEN(SRC) - CHARINDEX(' on ',SRC)-4 
) AS FullPath, 

REVERSE(LEFT(
    REVERSE(SRC), 
    CHARINDEX('/',REVERSE(SRC))-1) 
    ) AS PathAndFileName 
FROM 
(
SELECT 'Successful Write by DOMAIN\USERLOGIN on /SOMENETWORKSHARE/FOLDER/MyFolder/USERLOGIN/DesktopBackUp/log.txt' AS SRC 
UNION ALL 
SELECT 'some invalid stuff' AS SRC 
) as tbl 
WHERE SRC LIKE '% by %\% on %/%' 

的UPDATE語句適用這將是：

UPDATE [dbo].[SACL] 
SET 
[Event]= 
LEFT(Headline,CHARINDEX(' by ',Headline)), 
[Username] = 
SUBSTRING(
    Headline, 
    CHARINDEX(' by ',Headline)+4, 
    CHARINDEX(' on ',Headline) - CHARINDEX(' by ',Headline)-4 
    ), 
[Path] = 
SUBSTRING(
    Headline, 
    CHARINDEX(' by ',Headline)+4, 
    CHARINDEX(' on ',Headline) - CHARINDEX(' by ',Headline)-4 
    ), 

[FileName]= 
REVERSE(LEFT(
    REVERSE(Headline), 
    CHARINDEX('/',REVERSE(Headline))-1) 
    ) 
WHERE 
-- Only update rows that fit the pattern 
Headline LIKE '% by %\% on %/%' 
-- Only update rows that haven't been processed 
AND ([Event] IS NULL OR [Event] = '') 

請注意：

• 我不知道哪個字段包含完整的字符串。我假設Headline
• 我建議你標準化Event字段的內容。 NULL =未處理，空白意味着別的東西（已處理但無法解決）
• 我還建議您添加ImportedDate和ProcessedDate字段，它是datetime類型。 ImportedDate的default爲GETDATE()，這意味着這是記錄導入的日期時間。 ProcessedDate應該由UPDATE語句更新爲GETDATE()，這會告訴您記錄何時更新。