如何讓我的rexml/nokogiri腳本運行得更快

Question

我有收集46344個xml鏈接的這個ruby腳本，然後在每個xml文件中收集16個元素節點。該過程的最後一部分是它將其存儲在一個CSV文件中。我的問題是需要很長時間。它需要超過1-2小時..

這是沒有鏈接的腳本，所有的XML鏈接，我不能提供鏈接beacuse其公司的東西..我希望它很酷。

這裏是腳本，它的工作原理，但它需要長期：

require 'rubygems' 
require 'nokogiri' 
require 'open-uri' 
require 'rexml/document' 
require 'csv' 
include REXML 

@urls = Array.new 
@ID = Array.new 
@titleSv = Array.new 
@titleEn = Array.new 
@identifier = Array.new 
@typeOfLevel = Array.new 
@typeOfResponsibleBody = Array.new 
@courseTyp = Array.new 
@credits = Array.new 
@degree = Array.new 
@preAcademic = Array.new 
@subjectCodeVhs = Array.new 
@descriptionSv = Array.new 
@visibleToSweApplicants = Array.new 
@lastedited = Array.new 
@expires = Array.new 

# Hämtar alla XML-länkar 
htmldoc = Nokogiri::HTML(open('A SITE THAT HAVE ALL THE LINKS')) 
# Hämtar alla länkar för xml-filerna och sparar dom i arrayn urls 
htmldoc.xpath('//a/@href').each do |links| 
    @urls << links.content 
end 

@urls.each do |url| 
    # Loop throw the XML files and grab element nodes 
    xmldoc = REXML::Document.new(open(url).read) 
    # Root element 
    root = xmldoc.root 
    # Hämtar info-id 
    @ID << root.attributes["id"] 
    # TitleSv 
    xmldoc.elements.each("/ns:educationInfo/ns:titles/ns:title[1]"){ 
    |e| @titleSv << e.text 
    } 
    # TitleEn 
    xmldoc.elements.each("/ns:educationInfo/ns:titles/ns:title[2]"){ 
    |e| @titleEn << e.text 
    } 
    # Identifier 
    xmldoc.elements.each("/ns:educationInfo/ns:identifier"){ 
    |e| @identifier << e.text 
    } 
    # typeOfLevel 
    xmldoc.elements.each("/ns:educationInfo/ns:educationLevelDetails/ns:typeOfLevel"){ 
    |e| @typeOfLevel << e.text 
    } 
    # typeOfResponsibleBody 
    xmldoc.elements.each("/ns:educationInfo/ns:educationLevelDetails/ns:typeOfResponsibleBody"){ 
    |e| @typeOfResponsibleBody << e.text 
    } 
    # courseTyp 
    xmldoc.elements.each("/ns:educationInfo/ns:educationLevelDetails/ns:academic/ns:courseOfferingPackage/ns:type"){ 
    |e| @courseTyp << e.text 
    } 
    # credits 
    xmldoc.elements.each("/ns:educationInfo/ns:credits/ns:exact"){ 
    |e| @credits << e.text 
    } 
    # degree 
    xmldoc.elements.each("/ns:educationInfo/ns:degrees/ns:degree"){ 
    |e| @degree << e.text 
    } 
    # @preAcademic 
    xmldoc.elements.each("/ns:educationInfo/ns:prerequisites/ns:academic"){ 
    |e| @preAcademic << e.text 
    } 
    # @subjectCodeVhs 
    xmldoc.elements.each("/ns:educationInfo/ns:subjects/ns:subject/ns:code"){ 
    |e| @subjectCodeVhs << e.text 
    } 
    # DescriptionSv 
    xmldoc.elements.each("/educationInfo/descriptions/ct:description/ct:text"){ 
    |e| @descriptionSv << e.text 
    } 
    # Hämtar dokuments utgångs-datum 
    @expires << root.attributes["expires"] 
    # Hämtar dokuments lastedited 
    @lastedited << root.attributes["lastEdited"] 

    # Lagrar dom i uni.CSV 
    CSV.open("eduction_normal.csv", "wb") do |row| 
    (0..@ID.length - 1).each do |index| 
     row << [@ID[index], @titleSv[index], @titleEn[index], @identifier[index], @typeOfLevel[index], @typeOfResponsibleBody[index], @courseTyp[index], @credits[index], @degree[index], @preAcademic[index], @subjectCodeVhs[index], @descriptionSv[index], @lastedited[index], @expires[index]] 
    end 
    end 
end 

Answer 1

如果它是網絡訪問，你可以啓動線程，並/或開始使用JRuby可以使用所有內核的處理器。如果你經常這樣做，你將不得不制定一個讀寫策略，在沒有阻塞的情況下爲你提供最好的服務。