2016-11-04 29 views
0

我需要超過65個文件與RCurl上傳到我的主機的FTP服務器,但我收到後幾個文件已被上傳錯誤:如何在R中上傳多個文件而不會收到421太多的連接錯誤?

< 421 Too many connections (8) from this IP 
* We got a 421 - timeout! 
* Closing connection 291 

從我在日誌中看到它說Connection #283 to host ftp.myserver.com left intact使直到15分鐘不活動後連接過期,才需要進一步授權< 220 You will be disconnected after 15 minutes of inactivity.

至少這是我的理解,因爲,出於某種原因,後立即重新發送用戶名和密碼的第二個文件:

> USER [email protected] 
< 331 User [email protected] OK. Password required 
> PASS mypassword 
< 230 OK. Current restricted directory is/

我知道有倍數的授權請求(每個文件一個),這樣我淹沒服務器。

是否有可能首先打開到FTP服務器的連接,然後上傳所有文件?像在FileZilla中?

請找我使用R代碼:

for(file in list.files(localPath)) { 

    ftpUpload(paste0(localPath,file), userpwd = userpwd, to=paste0("ftp://", ftpPath, file), verbose=TRUE) 

} 

另一個SO成員在這裏有同樣的問題,但它看起來我已經使用了類似於提供給他的迴應的東西,忽略了最低的工作。 Using R to upload many files

謝謝


編輯(添加日誌):

> for(file in list.files(localPath)) { 
+ 
+ print(paste0("ftp://", ftpPath, file)) 
+ ftpUpload(paste0(localPath,file), userpwd = userpwd, to=paste0("ftp://", ftpPath, file), verbose=TRUE) 
+ 
+ } 
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation-1.csv" 
* Hostname was NOT found in DNS cache 
* Trying 11.22.33.44... 
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#310) 
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ---------- 
< 220-You are user number 2 of 1000 allowed. 
< 220-Local time is now 08:50. Server port: 21. 
< 220-This is a private system - No anonymous login 
< 220-IPv6 connections are also welcome on this server. 
< 220 You will be disconnected after 15 minutes of inactivity. 
> USER [email protected] 
< 331 User [email protected] OK. Password required 
> PASS mypassword 
< 230 OK. Current restricted directory is/
> PWD 
< 257 "/" is your current location 
* Entry path is '/' 
> CWD portfolios 
* ftp_perform ends with SECONDARY: 0 
< 250 OK. Current directory is /portfolios 
> CWD csv 
< 250 OK. Current directory is /data/csv 
> EPSV 
* Connect data stream passively 
< 229 Extended Passive mode OK (|||44516|) 
* Hostname was NOT found in DNS cache 
* Trying 11.22.33.44... 
* Connecting to 11.22.33.44 (11.22.33.44) port 44516 
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#310) 
> TYPE I 
< 200 TYPE is now 8-bit binary 
> STOR data_private_allocation-1.csv 
< 150 Accepted data connection 
* Remembering we are in dir "data/csv/" 
< 226-File successfully transferred 
< 226 0.180 seconds (measured here), 1.15 Kbytes per second 
* Connection #310 to host ftp.mywebsite.com left intact 
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation-2.csv" 
* Hostname was NOT found in DNS cache 
* Trying 11.22.33.44... 
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#311) 
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ---------- 
< 220-You are user number 3 of 1000 allowed. 
< 220-Local time is now 08:50. Server port: 21. 
< 220-This is a private system - No anonymous login 
< 220-IPv6 connections are also welcome on this server. 
< 220 You will be disconnected after 15 minutes of inactivity. 
> USER [email protected] 
< 331 User [email protected] OK. Password required 
> PASS mypassword 
< 230 OK. Current restricted directory is/
> PWD 
< 257 "/" is your current location 
* Entry path is '/' 
> CWD portfolios 
* ftp_perform ends with SECONDARY: 0 
< 250 OK. Current directory is /portfolios 
> CWD csv 
< 250 OK. Current directory is /data/csv 
> EPSV 
* Connect data stream passively 
< 229 Extended Passive mode OK (|||45077|) 
* Hostname was NOT found in DNS cache 
* Trying 11.22.33.44... 
* Connecting to 11.22.33.44 (11.22.33.44) port 45077 
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#311) 
> TYPE I 
< 200 TYPE is now 8-bit binary 
> STOR data_private_allocation-2.csv 
< 150 Accepted data connection 
* Remembering we are in dir "data/csv/" 
< 226-File successfully transferred 
< 226 0.207 seconds (measured here), 1.00 Kbytes per second 
* Connection #311 to host ftp.mywebsite.com left intact 
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation-3.csv" 
* Hostname was NOT found in DNS cache 
* Trying 11.22.33.44... 
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#312) 
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ---------- 
< 220-You are user number 4 of 1000 allowed. 
< 220-Local time is now 08:50. Server port: 21. 
< 220-This is a private system - No anonymous login 
< 220-IPv6 connections are also welcome on this server. 
< 220 You will be disconnected after 15 minutes of inactivity. 
> USER [email protected] 
< 331 User [email protected] OK. Password required 
> PASS mypassword 
< 230 OK. Current restricted directory is/
> PWD 
< 257 "/" is your current location 
* Entry path is '/' 
> CWD portfolios 
* ftp_perform ends with SECONDARY: 0 
< 250 OK. Current directory is /portfolios 
> CWD csv 
< 250 OK. Current directory is /data/csv 
> EPSV 
* Connect data stream passively 
< 229 Extended Passive mode OK (|||40390|) 
* Hostname was NOT found in DNS cache 
* Trying 11.22.33.44... 
* Connecting to 11.22.33.44 (11.22.33.44) port 40390 
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#312) 
> TYPE I 
< 200 TYPE is now 8-bit binary 
> STOR data_private_allocation-3.csv 
< 150 Accepted data connection 
* Remembering we are in dir "data/csv/" 
< 226-File successfully transferred 
< 226 0.189 seconds (measured here), 1.12 Kbytes per second 
* Connection #312 to host ftp.mywebsite.com left intact 
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation-4.csv" 
* Hostname was NOT found in DNS cache 
* Trying 11.22.33.44... 
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#313) 
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ---------- 
< 220-You are user number 5 of 1000 allowed. 
< 220-Local time is now 08:50. Server port: 21. 
< 220-This is a private system - No anonymous login 
< 220-IPv6 connections are also welcome on this server. 
< 220 You will be disconnected after 15 minutes of inactivity. 
> USER [email protected] 
< 331 User [email protected] OK. Password required 
> PASS mypassword 
< 230 OK. Current restricted directory is/
> PWD 
< 257 "/" is your current location 
* Entry path is '/' 
> CWD portfolios 
* ftp_perform ends with SECONDARY: 0 
< 250 OK. Current directory is /portfolios 
> CWD csv 
< 250 OK. Current directory is /data/csv 
> EPSV 
* Connect data stream passively 
< 229 Extended Passive mode OK (|||46329|) 
* Hostname was NOT found in DNS cache 
* Trying 11.22.33.44... 
* Connecting to 11.22.33.44 (11.22.33.44) port 46329 
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#313) 
> TYPE I 
< 200 TYPE is now 8-bit binary 
> STOR data_private_allocation-4.csv 
< 150 Accepted data connection 
* Remembering we are in dir "data/csv/" 
< 226-File successfully transferred 
< 226 0.187 seconds (measured here), 1.11 Kbytes per second 
* Connection #313 to host ftp.mywebsite.com left intact 
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation_protected-1.csv" 
* Hostname was NOT found in DNS cache 
* Trying 11.22.33.44... 
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#314) 
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ---------- 
< 220-You are user number 6 of 1000 allowed. 
< 220-Local time is now 08:50. Server port: 21. 
< 220-This is a private system - No anonymous login 
< 220-IPv6 connections are also welcome on this server. 
< 220 You will be disconnected after 15 minutes of inactivity. 
> USER [email protected] 
< 331 User [email protected] OK. Password required 
> PASS mypassword 
< 230 OK. Current restricted directory is/
> PWD 
< 257 "/" is your current location 
* Entry path is '/' 
> CWD portfolios 
* ftp_perform ends with SECONDARY: 0 
< 250 OK. Current directory is /portfolios 
> CWD csv 
< 250 OK. Current directory is /data/csv 
> EPSV 
* Connect data stream passively 
< 229 Extended Passive mode OK (|||46556|) 
* Hostname was NOT found in DNS cache 
* Trying 11.22.33.44... 
* Connecting to 11.22.33.44 (11.22.33.44) port 46556 
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#314) 
> TYPE I 
< 200 TYPE is now 8-bit binary 
> STOR data_private_allocation_protected-1.csv 
< 150 Accepted data connection 
* Remembering we are in dir "data/csv/" 
< 226-File successfully transferred 
< 226 0.186 seconds (measured here), 2.75 Kbytes per second 
* Connection #314 to host ftp.mywebsite.com left intact 
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation_protected-2.csv" 
* Hostname was NOT found in DNS cache 
* Trying 11.22.33.44... 
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#315) 
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ---------- 
< 220-You are user number 7 of 1000 allowed. 
< 220-Local time is now 08:50. Server port: 21. 
< 220-This is a private system - No anonymous login 
< 220-IPv6 connections are also welcome on this server. 
< 220 You will be disconnected after 15 minutes of inactivity. 
> USER [email protected] 
< 331 User [email protected] OK. Password required 
> PASS mypassword 
< 230 OK. Current restricted directory is/
> PWD 
< 257 "/" is your current location 
* Entry path is '/' 
> CWD portfolios 
* ftp_perform ends with SECONDARY: 0 
< 250 OK. Current directory is /portfolios 
> CWD csv 
< 250 OK. Current directory is /data/csv 
> EPSV 
* Connect data stream passively 
< 229 Extended Passive mode OK (|||43730|) 
* Hostname was NOT found in DNS cache 
* Trying 11.22.33.44... 
* Connecting to 11.22.33.44 (11.22.33.44) port 43730 
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#315) 
> TYPE I 
< 200 TYPE is now 8-bit binary 
> STOR data_private_allocation_protected-2.csv 
< 150 Accepted data connection 
* Remembering we are in dir "data/csv/" 
< 226-File successfully transferred 
< 226 0.190 seconds (measured here), 2.69 Kbytes per second 
* Connection #315 to host ftp.mywebsite.com left intact 
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation_protected-3.csv" 
* Hostname was NOT found in DNS cache 
* Trying 11.22.33.44... 
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#316) 
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ---------- 
< 220-You are user number 8 of 1000 allowed. 
< 220-Local time is now 08:50. Server port: 21. 
< 220-This is a private system - No anonymous login 
< 220-IPv6 connections are also welcome on this server. 
< 220 You will be disconnected after 15 minutes of inactivity. 
> USER [email protected] 
< 331 User [email protected] OK. Password required 
> PASS mypassword 
< 230 OK. Current restricted directory is/
> PWD 
< 257 "/" is your current location 
* Entry path is '/' 
> CWD portfolios 
* ftp_perform ends with SECONDARY: 0 
< 250 OK. Current directory is /portfolios 
> CWD csv 
< 250 OK. Current directory is /data/csv 
> EPSV 
* Connect data stream passively 
< 229 Extended Passive mode OK (|||43399|) 
* Hostname was NOT found in DNS cache 
* Trying 11.22.33.44... 
* Connecting to 11.22.33.44 (11.22.33.44) port 43399 
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#316) 
> TYPE I 
< 200 TYPE is now 8-bit binary 
> STOR data_private_allocation_protected-3.csv 
< 150 Accepted data connection 
* Remembering we are in dir "data/csv/" 
< 226-File successfully transferred 
< 226 0.196 seconds (measured here), 2.61 Kbytes per second 
* Connection #316 to host ftp.mywebsite.com left intact 
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation_protected-4.csv" 
* Hostname was NOT found in DNS cache 
* Trying 11.22.33.44... 
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#317) 
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ---------- 
< 220-You are user number 9 of 1000 allowed. 
< 220-Local time is now 08:50. Server port: 21. 
< 220-This is a private system - No anonymous login 
< 220-IPv6 connections are also welcome on this server. 
< 220 You will be disconnected after 15 minutes of inactivity. 
> USER [email protected] 
< 331 User [email protected] OK. Password required 
> PASS mypassword 
< 230 OK. Current restricted directory is/
> PWD 
< 257 "/" is your current location 
* Entry path is '/' 
> CWD portfolios 
* ftp_perform ends with SECONDARY: 0 
< 250 OK. Current directory is /portfolios 
> CWD csv 
< 250 OK. Current directory is /data/csv 
> EPSV 
* Connect data stream passively 
< 229 Extended Passive mode OK (|||48536|) 
* Hostname was NOT found in DNS cache 
* Trying 11.22.33.44... 
* Connecting to 11.22.33.44 (11.22.33.44) port 48536 
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#317) 
> TYPE I 
< 200 TYPE is now 8-bit binary 
> STOR data_private_allocation_protected-4.csv 
< 150 Accepted data connection 
* Remembering we are in dir "data/csv/" 
[1] "ftp://ftp.mywebsite.com/data/csv/data_coins_average-1.csv" 
< 226-File successfully transferred 
< 226 0.184 seconds (measured here), 2.77 Kbytes per second 
* Connection #317 to host ftp.mywebsite.com left intact 
* Hostname was NOT found in DNS cache 
* Trying 11.22.33.44... 
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#318) 
< 421 Too many connections (8) from this IP 
* We got a 421 - timeout! 
* Closing connection 318 
Error in function (type, msg, asError = TRUE) : 
    Uploading to a URL without a file name! 
+0

您是否嘗試關閉任何打開的連接('?closeAllConnections')? –

+0

嗨,我剛剛用'closeAllConnections()'關閉了'gzcon',但仍然有'421太多連接錯誤'。 – Florent

回答

0

這是一種解決方法,而不是我的問題的解決方案,但它就像一個魅力。我創建了一個ftp命令的文本文件,然後啓動Ubuntu ftp與R system()

ftpcommands.txt:

open ftp.mywebsite.com 
user [email protected] mypassword 
lcd ~/R/data/csv 
cd data/csv 
prompt 
mput *.csv 
bye 

然後R命令:

system("ftp -n < ~/R/data/ftpcommands.txt") 
0

它看起來像你打開每個文件一個新的連接,這是不如何文件傳輸協議打算使用。一些不同的解決方案:

  1. 在您的R代碼中打開較少的連接。
  2. 限制使用的最大併發連接數curl
  3. 增加配置中的併發連接的允許限制FTP服務器(做到這一點只有當你真的需要它,正如我所說,不打開每個文件的連接。)

檢查this detailed answer出以獲得更好的理解這一問題。不要理解我的錯誤,對於性能而言,同時使用多個連接可能會有好處,但對於每個文件來說都不是這樣,這確實太多了。大多數FTP客戶端使用2.