2010-10-11 85 views
3

我想用Qt編寫一個程序,每天從一個站點下載大量的HTML網頁,大約5000個。下載完頁面之後,我需要使用DOM Query,使用WebKit模塊提取一些數據,然後將這些數據存儲在數據庫中。使用Qt批量下載網頁

哪種方法是最好的/正確的/有效的方法,特別是下載和分析階段?我如何處理這些請求以及如何創建「下載管理器」?

+1

考慮使用像'wget'這樣的外部二進制文件進行下載 – 2010-10-11 21:33:45

回答

0

這已經回答了,但這裏是使用你的要求的解決方案,並與QT這樣做。

您可以使用QT(特別是QNetworkManager,QNetworkRequests,QNetworkReply)製作(網站爬蟲)。我不確定這是否是處理此類任務的正確方法,但我發現利用多個線程可以最大限度地提高效率並節省時間。 (請有人告訴我,如果有另一種方式/或確認這是否是好的做法)

概念是工作的列表中排隊,並且工人將執行工作,接到信息後/處理它,然後繼續下一個項目。

類工人對象 類應該接受一個網址,程序和下載的URL的HTML數據,然後在接收時處理信息。

創建隊列和管理器隊列 我創建了一個QQueue <的QString> urlList以控制被處理併發物品的量和任務列表來完成的。

QQueue <String> workQueue; //First create somewhere a 
    int maxWorkers = 10; 


    //Then create the workers 
    void downloadNewArrivals::createWorkers(QString url){ 
checkNewArrivalWorker* worker = new checkNewArrivalWorker(url); 
workQueue.enqueue(worker); 
} 

    //Make a function to control the amount of workers, 
    //and process the workers after they are finished 

    void downloadNewArrivals::processWorkQueue(){ 
if (workQueue.isEmpty() && currentWorkers== 0){ 
    qDebug() << "Work Queue Empty" << endl; 
} else if (!workQueue.isEmpty()){ 
    //Create the maxWorkers and start them in seperate threads 
    for (int i = 0; i < currentWorkers && !workQueue.isEmpty(); i++){ 
     QThread* thread = new QThread; 
     checkNewArrivalWorker* worker = workQueue.dequeue(); 
     worker->moveToThread(thread); 
     connect(worker, SIGNAL(error(QString)), this, SLOT(errorString(QString))); 
     connect(thread, SIGNAL(started()), worker, SLOT(process())); 
     connect(worker, SIGNAL(finished()), thread, SLOT(quit())); 
     connect(worker, SIGNAL(finished()), worker, SLOT(deleteLater())); 
     connect(thread, SIGNAL(finished()), this, SLOT(reduceThreadCounterAndProcessNext())); 
     connect(thread, SIGNAL(finished()), thread, SLOT(deleteLater())); 
     thread->start(); 
     currentWorkers++; 
    } 
} 
} 

    //When finished, process the next worker 
    void downloadNewArrivals::reduceThreadCounterAndProcessNext(){ 
currentWorkers--; //This variable is to control amount of max workers 

processWorkQueue(); 
    } 


    //Now the worker 
    //The worker class important parts.. 
    void checkNewArrivalWorker::getPages(QString url){ 
QNetworkAccessManager *manager = new QNetworkAccessManager(this); 
QNetworkRequest getPageRequest = QNetworkRequest(url); //created on heap 
getPageRequest.setRawHeader("User-Agent", "Mozilla/5.0 (X11; U; Linux i686 (x86_64); " 
          "en-US; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1"); 
getPageRequest.setRawHeader("charset", "utf-8"); 
getPageRequest.setRawHeader("Connection", "keep-alive"); 
connect(manager, SIGNAL(finished(QNetworkReply*)), this, SLOT(replyGetPagesFinished(QNetworkReply*))); 
connect(manager, SIGNAL(finished(QNetworkReply*)), manager, SLOT(deleteLater())); 
manager->get(getPageRequest); 
} 

    void checkNewArrivalWorker::replyGetPagesFinished(QNetworkReply *reply){ 
QString data = reply->readAll(); //Here data will hold your html to process as needed... 
reply->deleteLater(); 
emit finished(); 


} 

後你會得到你的信息,我只是處理從QString的信息,但是我相信你能解決如何使用DOM解析器一旦你到這個階段。

我希望這是一個足夠的例子,足以幫助你。