使用python批量更新MySql

Question

我必須將數百萬行更新到MySQL。我目前使用for循環來執行查詢。爲了更快地實現更新，我想使用Python MySQL Connector的executemany()，這樣我可以使用每個批次的單個查詢進行批量更新。

Answer 1

我不認爲MySQLdb的具有一次處理多個UPDATE查詢的方式。

但是，您可以在最後使用具有ON DUPLICATE KEY UPDATE條件的INSERT查詢。

爲了易用性和可讀性，我編寫了以下示例。

import MySQLdb 

def update_many(data_list=None, mysql_table=None): 
    """ 
    Updates a mysql table with the data provided. If the key is not unique, the 
    data will be inserted into the table. 

    The dictionaries must have all the same keys due to how the query is built. 

    Param: 
     data_list (List): 
      A list of dictionaries where the keys are the mysql table 
      column names, and the values are the update values 
     mysql_table (String): 
      The mysql table to be updated. 
    """ 

    # Connection and Cursor 
    conn = MySQLdb.connect('localhost', 'jeff', 'atwood', 'stackoverflow') 
    cur = conn.cursor() 

    query = "" 
    values = [] 

    for data_dict in data_list: 

     if not query: 
      columns = ', '.join('`{0}`'.format(k) for k in data_dict) 
      duplicates = ', '.join('{0}=VALUES({0})'.format(k) for k in data_dict) 
      place_holders = ', '.join('%s'.format(k) for k in data_dict) 
      query = "INSERT INTO {0} ({1}) VALUES ({2})".format(mysql_table, columns, place_holders) 
      query = "{0} ON DUPLICATE KEY UPDATE {1}".format(query, duplicates) 

     v = data_dict.values() 
     values.append(v) 

    try: 
     cur.executemany(query, values) 
    except MySQLdb.Error, e: 
     try: 
      print"MySQL Error [%d]: %s" % (e.args[0], e.args[1]) 
     except IndexError: 
      print "MySQL Error: %s" % str(e) 

     conn.rollback() 
     return False 

    conn.commit() 
    cur.close() 
    conn.close() 

一個襯墊

columns = ', '.join('`{}`'.format(k) for k in data_dict) 

的說明是一樣的

column_list = [] 
for k in data_dict: 
    column_list.append(k) 
columns = ", ".join(columns) 

這裏的用法的一例

test_data_list = [] 
test_data_list.append({'id' : 1, 'name' : 'Marco', 'articles' : 1 }) 
test_data_list.append({'id' : 2, 'name' : 'Keshaw', 'articles' : 8 }) 
test_data_list.append({'id' : 3, 'name' : 'Wes', 'articles' : 0 }) 

update_many(data_list=test_data_list, mysql_table='writers') 

查詢輸出

個

INSERT INTO writers (`articles`, `id`, `name`) VALUES (%s, %s, %s) ON DUPLICATE KEY UPDATE articles=VALUES(articles), id=VALUES(id), name=VALUES(name) 

價值觀輸出

[[1, 1, 'Marco'], [8, 2, 'Keshaw'], [0, 3, 'Wes']] 

Answer 2

也許這可以幫助 How to update multiple rows with single MySQL query in python?

cur.executemany("UPDATE Writers SET Name = %s WHERE Id = %s ", 
    [("new_value" , "3"),("new_value" , "6")]) 
conn.commit()