DownloadStringAsync 300網址並行 - C＃

我想用Web客戶端的DownloadStringAsync，一些URL的下載精美下載多個網址（約300），但對於其他一些下載的只是掛起，直到永遠。

問題：

我怎麼能趕上無法下載的問題的網址？
基於附加的代碼，我如何實現「DownloadStringAsync」方法的超時期限以跳過無法下載的Url？

我已經搜索瞭解決方案，但出現缺乏手段。然後嘗試在這裏搜索數據庫中的解決方案，結果相同。任何幫助表示讚賞。

這是代碼我使用：

public class ParallelDownloading 
    { 
     private int _totalDownloadedURLs = 0; 
     private ConcurrentQueue<DownloadFile> _queueToDownlaod; 
     private IList<Task> _downloadingTasks; 
     private Timer _downloadTimer; 

     public event EventHandler<OnFileDownloadCompletedEventArgs> OnDownloadURLCompleted; 

     private int _parallelDownloads; 

     public ParallelDownloading(int parallelDownloads) 
     { 
      _totalDownloadedURLs = 0; 
      _queueToDownlaod = new ConcurrentQueue<DownloadFile>(); 
      _downloadingTasks = new List<Task>(); 
      _downloadTimer = new Timer(); 

      _parallelDownloads = parallelDownloads; 

      ServicePointManager.DefaultConnectionLimit = parallelDownloads; 
     } 

     public void StartParallelDownloading() 
     { 
      _downloadTimer.Elapsed += new ElapsedEventHandler(DownloadTimer_Elapsed); 
      _downloadTimer.Interval = 800; 
      _downloadTimer.Start(); 
     } 

     public void EnqueueFileToDownload(DownloadFile file) 
     { 
      _queueToDownlaod.Enqueue(file); 
     } 

     void DownloadTimer_Elapsed(object sender, ElapsedEventArgs e) 
     { 
      StartDownload(); 
     } 

     private void StartDownload() 
     { 
      lock (_downloadingTasks) 
      { 
       if (_downloadingTasks.Count < _parallelDownloads && _queueToDownlaod.Count > 0) 
       { 
        DownloadFile fileToDownload; 
        if (_queueToDownlaod.TryDequeue(out fileToDownload)) 
        { 
         var task = new Task(() => 
         { 
          var client = new WebClient(); 
          client.DownloadStringCompleted += new DownloadStringCompletedEventHandler(client_DownloadStringCompleted); 
          client.DownloadStringAsync(new Uri(fileToDownload.Link), fileToDownload.Link); 

         }, TaskCreationOptions.LongRunning); 

         task.ContinueWith(DownloadOverCallback, TaskContinuationOptions.None); 

         _downloadingTasks.Add(task); 
         task.Start(); 
        } 
       } 
      } 
     } 

     void client_DownloadStringCompleted(object sender, DownloadStringCompletedEventArgs e) 
     { 
      _totalDownloadedURLs ++; 
      if (e.Error == null && e.Cancelled == false) 
      { 
       //do something 
      } 
      OnFileDownloadCompletedEventArgs evtArgs = new OnFileDownloadCompletedEventArgs(); 
      evtArgs.DownloadedFiles = _totalDownloadedURLs; 
      OnDownloadURLCompleted(this, evtArgs); 
     } 

     public void DownloadOverCallback(Task downloadingTask) 
     { 
      lock (_downloadingTasks) 
      { 
       _downloadingTasks.Remove(downloadingTask); 
      } 
     } 
    }

來源

2013-07-04 Frank Parsons

這可能是網絡或服務器問題。 – SLaks

'_totalDownloadedURLs ++'是一個競爭條件，它會在負載下搞砸你，因爲'client_DownloadStringCompleted'可以從不同線程同時調用。考慮'互鎖增量'。 – spender

@SLaks def不是網絡問題。 –

我1.How能趕上無法下載的問題的網址？爲此，您可以使用try，catch來查看發生了什麼

2.基於附加的代碼，如何實現'DownloadStringAsync'方法的超時時間段以跳過無法下載的Url？最簡單的方法是使用Timer在一段時間後調用CancelAsync或使用HttpWebRequest，它具有Timeout屬性

來源

2014-07-22 13:30:28

DownloadStringAsync 300網址並行 - C＃

回答

相關問題