从 CSV 写入 SQL 数据库

Question

我手头有这些 csv 文件，我必须将它们上传到远程数据库，并且我在 python 中使用 pyodbc 和 csv 库来做 it.I 不知道为什么但是它非常慢（100 行大约需要 30 秒），而且我必须上传的这些 csv 文件中有一些超过 30k 行。我也尝试过使用 pandas 但速度没有变化。这或多或少是我的code.Unnecessary部分被省略了。

if len(sys.argv) == 1:
print("This program needs needs an input state")
exit()

state_code = str(sys.argv[1])

f = open(state_code+".csv", "r")

reader = csv.reader(f, delimiter=',')


 cnxn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)

insert_query = '''INSERT INTO table (Zipcode, Pers_Property_Coverage, Deductible, 
Liability,Average_Rate,
            Highest_Rate, Lowest_Rate, CREATE_DATE, Active_Flag)
            VALUES(?,?,?,?,?,?,?,?,?)'''
for row in reader:
    zipcode = row[0]

    if len(zipcode) == 4:
        zipcode = "0" + zipcode

    ppc=row[1][1:]
    ppc=ppc.replace(',', '')

    deductible = row[2][1:]
    deductible = deductible.replace(',', '')

    liability = row[3][1:]
    liability = liability.replace(',', '')

    average_rate = row[4][1:]
    average_rate = average_rate.replace(',', '')

    highest_rate = row[5][1:]
    highest_rate = highest_rate.replace(',', '')

    lowest_rate=row[6][1:]
    lowest_rate = lowest_rate.replace(',', '')

    ctr=ctr+1

    if ctr % 100 == 0:
        print("Time Elapsed = ", round(time.time() - start_time)," seconds")

    values = (zipcode, ppc, deductible, liability, average_rate, highest_rate, lowest_rate, date, "Y")

    print("Inserting "+zipcode ,ppc , deductible, liability, average_rate, highest_rate, lowest_rate,date, "Y")

    cursor.execute(insert_query, values)
cnxn.commit()

Answer 1

更新您的代码以使用带选项 fast_executemany=True 的 pyodbc executemany 可能是一种节省时间的简单方法：

https://github.com/mkleehammer/pyodbc/wiki/Cursor#executemanysql-params-with-fast_executemanytrue

从您的文件中探索批量插入可能是另一种选择，尽管它很可能不会使用 pyodbc 或 python:

https://docs.microsoft.com/en-us/sql/t-sql/statements/bulk-insert-transact-sql?view=sql-server-ver15

从 CSV 写入 SQL 数据库

Writing to SQL Database from CSV

database

csv

performance

pyodbc