我試圖導出250MB的數據(75 chararray列)從hdfs到sqlserver。它沒有與下面的錯誤,sqoop導出到sql server失敗
引起:產生java.io.IOException:com.microsoft.sqlserver.jdbc.SQLServerException:傳入的表格格式的數據流(TDS)遠程過程調用(RPC)協議流不正確。此RPC請求中提供的參數太多。最大值是2100.
然後我通過"-D sqoop.export.records.per.statement=10"
這個聲明以及sqoop導出它的工作,但它非常緩慢。花了15分鐘來加載250MB的數據。
無論如何,我們可以提高性能。
下面是實際的sqoop CMD:
sqoop出口-D sqoop.export.records.per.statement = 10 --connect「JDBC:SQLSERVER://199.198.165.191:1433;用戶名= ;密碼=;數據庫=數據庫」 --table Facttable --columns DimDateID,DimQHourID,ETLMergedFileQHourlyNortelID,DimSWVersionID,DimFreqCellRelationID,OSSC_RC,SubNetwork1,SubNetwork2,MeContext,ENodeBFunction,EUtranCellFDD,EUtranFreqRelation,EUtranCellRelation,時間,GmtOffset,FFV ,SN,ST,VN,CBT,TS,NeuN的,nedn,NESW,MTS,GP,SF,pmHoExeAttLteInterF,pmHoExeAttLteIntraF,pmHoExeSuccLteInterF,pmHoExeSuccLteIntraF,pmHoPrepAttLteInterF,pmHoPrepAttLteIntr AF,pmHoPrepSuccLteInterF,pmHoPrepSuccLteIntraF,Count_Null,Count_Negative,Count_Threshold,pmHoExeAttLteInterFLb,pmHoExeSuccLteInterFLb,pmHoOscInterF,pmHoOscIntraF,pmHoPrepAttLteInterFLb,pmHoPrepSuccLteInterFLb,pmHoPrepTNotAllowedLteInterF,pmHoPrepTNotAllowedLteIntraF,pmHoTooEarlyHoInterF,pmHoTooEarlyHoIntraF,pmHoTooLateHoInterF,pmHoTooLateHoIntraF,pmHoWrongCellInterF,pmHoWrongCellIntraF,pmHoWrongCellReestInterF,pmHoWrongCellReestIntraF,pmLbQualifiedUe,pmZtemporary36,pmHoExeAttLteIntraFTuneOut, pmHoExeSuccLteIntraFTuneOut --export-dir/Fact_Peg --direct -m 8 --input-fields-terminated-by「,」--input-lines-terminated-by「\ n」;
「