odbc reader 用于 vb.net 中的 csv

Question

我正在编写一个程序，应该在短时间内处理 csv 中的数百万个数据集，出于性能原因，我的想法是使用 odbc，因此我使用 odbc 读取所有数据并将其保存在内存中，此后我添加参数并将其插入 sql db，这是我目前的代码：

Using connection As New OdbcConnection("jdbc:odbc:Driver={Microsoft Text Driver (*.txt; *.csv)};" & filePath & "Extensions=csv;Persist Security Info=False;")
                Dim reader As OdbcDataReader

                Dim i As Integer
                Dim r As SeekZeilen

                Dim TextFileTable As DataTable = Nothing


                Dim line As String = reader.Read()
                Me.ParseString(line)

                Dim memStream As New MemoryStream(Encoding.Default.GetBytes(line))

                Using TextFileReader As New Microsoft.VisualBasic.FileIO.TextFieldParser(memStream)
                    TextFileReader.TextFieldType = FileIO.FieldType.Delimited
                    TextFileReader.SetDelimiters(";") 
                    r.erste_Zeile = TextFileReader.ReadFields()

                    If TextFileTable Is Nothing Then
                        TextFileTable = New DataTable("TextFileTable")

                        For i = 0 To r.erste_Zeile.Length - 1
                            Dim Column As New DataColumn(r.erste_Zeile(i))

                            Column.ReadOnly = True
                            TextFileTable.Columns.Add(Column)
                        Next
                    End If
                    DataGridView1.DataSource = TextFileTable
                End Using

                While reader.HasRows
                    line = reader.Read()
                    Me.ParseString(line)
                    memStream = New MemoryStream(Encoding.Default.GetBytes(line))

                    Using TextFileReader As New Microsoft.VisualBasic.FileIO.TextFieldParser(memStream)
                        TextFileReader.TextFieldType = FileIO.FieldType.Delimited
                        TextFileReader.SetDelimiters(";") 

                        DataGridView1.DataSource = TextFileTable
                        Try
                            r._Rest = TextFileReader.ReadFields()


                            ReplaceChars(r._Rest)

                            If Not r._Rest Is Nothing Then
                                Dim oSQL As New DBUmgebung.cdb.SQL()
                                oSQL.init()
                                AddParameters(oSQL, r)
                                oSQL.ausfuehrenSQL(DBUmgebung.cdb.KSQLCommand.INSERT, _table, "")
                                Dim dtRow As DataRow = TextFileTable.NewRow

                                For i = 0 To r._Rest.Length - 1

                                    dtRow(i) = r._Rest(i).ToString()
                                Next

                                TextFileTable.Rows.Add(dtRow)
                                DataGridView1.Refresh()
                                Application.DoEvents()
                            End If
                        Catch ex As Microsoft.VisualBasic.FileIO.MalformedLineException
                            MsgBox("Error! " & ex.Message & _
                            "")

                        Catch sqlEx As SqlException
                            MessageBox.Show(sqlEx.Message)

                            rtbSql.Focus()
                            Exit For
                        Catch ex As Exception
                            MessageBox.Show(ex.Message)
                            rtbSql.Focus()
                            Exit For

                        End Try
                    End Using
                End While
                reader.Close()
            End Using

问题是由于未知原因我得到了空指针异常，有人知道我做错了什么吗？可能是因为我的 odbc reader 没有正确初始化？

Answer 1

试试这个。这会将 csv 文件作为所有文本读取到数据表中。一旦进入数据表，您就可以将记录插入 SQL。您可以随时调整它以处理多个 csv 文件。

Friend Shared Function GetExcelFile(ByVal strFileName As String, ByVal strPath As String) As DataTable

    Try

        Dim dt As New DataTable

            Dim ConStr As String = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" & strPath & ";Extended Properties=""Text;HDR=Yes;FMT=Delimited\"""
            Dim conn As New OleDb.OleDbConnection(ConStr)
            Dim da As New OleDb.OleDbDataAdapter("Select * from " & strFileName, conn)
            da.Fill(dt)

        Return dt

    Catch ex As Exception
        Return Nothing
    End Try

End Function

odbc reader 用于 vb.net 中的 csv

odbc reader for csv in vb.net

vb.net

csv

odbc