將數據複製到Vertica的使用python

Question

我用python和vertica-python庫COPY數據Vertica DB

connection = vertica_python.connect(**conn_info) 
vsql_cur = connection.cursor() 

with open("/tmp/vertica-test-insert", "rb") as fs: 
    vsql_cur.copy("COPY table FROM STDIN DELIMITER ',' ", fs, buffer_size=65536) 
    connection.commit() 

它插入數據，但只有5行，雖然文件中包含多個。這可能與數據庫設置有關，或者這是一些客戶端問題？

Answer 1

很可能您的行遭到拒絕。假設你正在使用7.x中，您可以添加：

[ REJECTED DATA {'path' [ ON nodename ] [, ...] | AS TABLE 'reject_table'} ]

您還可以查詢該副本執行後看到的結果的摘要：

SELECTGET_NUM_ACCEPTED_ROWS(),GET_NUM_REJECTED_ROWS();

Answer 2

此代碼適用於我：

對於JSON

# for json file 
with open("D:/SampleCSVFile_2kb/tweets.json", "rb") as fs: 
    my_file = fs.read().decode('utf-8') 
    cur.copy("COPY STG.unstruc_data FROM STDIN parser fjsonparser()", my_file) 
    connection.commit() 

對於CSV

# for csv file 
with open("D:/SampleCSVFile_2kb/SampleCSVFile_2kb.csv", "rb") as fs: 
    my_file = fs.read().decode('utf-8','ignore') 
    cur.copy("COPY STG.unstruc_data FROM STDIN PARSER FDELIMITEDPARSER (delimiter=',', header='false') ", my_file) # buffer_size=65536 
    connection.commit()