Python簡單的多線程下載文件損壞

Question

這是我的第一篇文章。我一直在進行python編程，最近正在開發一個多線程下載器。但問題是我的文件（jpg是我的目標）被損壞。另外隨着followinf輸入：http://www.aumathletics.com/images_web/headerAUMLogo.jpg

它顯示錯誤

，而與輸入： http://www.nasa.gov/images/content/607800main_kepler1200_1600-1200.jpg

的文件被損壞。

下面是代碼： -

import os, sys, requests 
import threading 
import urllib2 
import time 

URL = sys.argv[1] 

def buildRange(value, numsplits): 
    lst = [] 
    for i in range(numsplits): 
    if i == 0: 
     lst.append('%s-%s' % (i, int(round(1 + i * value/(numsplits*1.0) + value/(numsplits*1.0)-1, 0)))) 
    else: 
     lst.append('%s-%s' % (int(round(1 + i * value/(numsplits*1.0),0)), int(round(1 + i * value/(numsplits*1.0) + value/(numsplits*1.0)-1, 0)))) 
return lst 

def main(url=None, splitBy=5): 
    start_time = time.time() 
    if not url: 
     print "Please Enter some url to begin download." 
     return 

fileName = "image.jpg" 
sizeInBytes = requests.head(url, headers={'Accept-Encoding': 'identity'}).headers.get('content-length', None) 
print "%s bytes to download." % sizeInBytes 
if not sizeInBytes: 
    print "Size cannot be determined." 
    return 

dataDict = {} 

# split total num bytes into ranges 
ranges = buildRange(int(sizeInBytes), splitBy) 

def downloadChunk(idx, irange): 
    req = urllib2.Request(url) 
    req.headers['Range'] = 'bytes={}'.format(irange) 
    dataDict[idx] = urllib2.urlopen(req).read() 

# create one downloading thread per chunk 
downloaders = [ 
    threading.Thread(
     target=downloadChunk, 
     args=(idx, irange), 
    ) 
    for idx,irange in enumerate(ranges) 
    ] 

# start threads, let run in parallel, wait for all to finish 
for th in downloaders: 
    th.start() 
for th in downloaders: 
    th.join() 



print 'done: got {} chunks, total {} bytes'.format(
    len(dataDict), sum((
     len(chunk) for chunk in dataDict.values() 
    )) 
) 

print "--- %s seconds ---" % str(time.time() - start_time) 

if os.path.exists(fileName): 
    os.remove(fileName) 



# reassemble file in correct order 
with open(fileName, 'w') as fh: 

    for _idx,chunk in sorted(dataDict.iteritems()): 
     fh.write(chunk) 

print "Finished Writing file %s" % fileName 
print 'file size {} bytes'.format(os.path.getsize(fileName)) 

if __name__ == '__main__': 
    main(URL) 

這裏的缺口可能是錯了，所以這裏是代碼引擎收錄（點）的COM/wGEkp878

我會很感激，如果有人能指出錯誤

編輯：由人提議

def buildRange(value, numsplits): 
    lst = [] 
    for i in range(numsplits): 
     first = i if i == 0 else buildRange().start(i, value, numsplits) 
     second = buildRange().end(i, value, numsplits) 
     lst.append("{}-{}".format(first, second)) 
    return lst 

誰能告訴我鋤保持part1 part2等名稱下載的零件文件等

Answer 1

事實證明，文件必須以二進制模式打開，'wb'而不是'w'。如果用'w'打開，會寫入一堆額外的字符。這與derpy windows與linux新行語義有關。如果你使用'wb'，它會準確地寫入你放入文件的內容。

編輯： 如果要存儲個人文件部分可以改變

# reassemble file in correct order 
with open(fileName, 'w') as fh: 
    for _idx,chunk in sorted(dataDict.iteritems()): 
     fh.write(chunk) 

print "Finished Writing file %s" % fileName 
print 'file size {} bytes'.format(os.path.getsize(fileName)) 

要

# reassemble file in correct order 
for _idx,chunk in sorted(dataDict.iteritems()): 
    with open(fileName + str(".part-") + str(_idx), 'wb') as fh: 
     fh.write(chunk) 

print "Finished Writing file %s" % fileName 
#print 'file size {} bytes'.format(os.path.getsize(fileName))