2017-06-16 48 views
1

蟒蛇福利局這裏具體的CSV行 - 我試圖格式化一組真毛CSV的我被送到,這樣我可以把它們變成一個很好的Postgres的表進行查詢和分析。爲了做到這一點,我首先使用csv.writer刪除空行和雙引號來封裝每個條目,潔淨他們。這裏是我的代碼如下所示:遍歷在Python輸出一個空白文件

import os 
import csv 
import glob 
from itertools import islice 

files = glob.glob('/Users/foo/bar/*.csv') 

# Loop through all of the csv's 
for file in files: 
    # Get the filename from the path 
    outfile = os.path.basename(file) 

    with open(file, 'rb') as inp, open('/Users/foo/baz/' + outfile, 'wb') as out: 

     reader = csv.reader(inp) 
     writer = csv.writer(out) 
     for row in reader: 
      if row: 
       writer.writerow(row) 
     out.close() 

它完美的罰款,並確切地做什麼,我想要它做的。輸出csv看起來不錯。接下來,我嘗試基本上砍掉了一定包含從一開始就和新潔淨CSV文件的末尾都完全不必要的垃圾行(省略第8行和最後2)。爲此,我真的不能確定,從代碼的這部分CSV輸出的一個原因(縮進一樣「與」塊更早)完全是空的:

with open('/Users/foo/baz/' + outfile, 'rb') as inp2, open('/Users/foo/qux/' + outfile, 'wb') as out2: 
    writer2 = csv.writer(out2) 
    reader2 = csv.reader(inp2) 
    row_count = sum(1 for row in reader2) 
    last_line_index = row_count - 3 
    for row in islice(reader2, 7, last_line_index): 
      writer2.writerow(row) 
    out2.close() 

我知道是因爲我「與」使用中,關閉()在每個塊的末尾是冗餘的 - 我嘗試了作爲一種方法尋找here之後。我也試圖把第二個「與」塊到不同的文件,並運行在運行第一「與」塊之後,但仍無濟於事。非常感謝您的幫助!

而且,這裏的整個文件:

import os 
import csv 
import glob 
from itertools import islice 

files = glob.glob('/Users/foo/bar/*.csv') 

# Loop through all of the csv's 
for file in files: 
    # Get the filename from the path 
    outfile = os.path.basename(file) 

    with open(file, 'rb') as inp, open('/Users/foo/baz/' + outfile, 'wb') as out: 

     reader = csv.reader(inp) 
     writer = csv.writer(out) 
     for row in reader: 
      if row: 
       writer.writerow(row) 
     out.close() 

    with open('/Users/foo/baz/' + outfile, 'rb') as inp2, open('/Users/foo/qux/' + outfile, 'wb') as out2: 
     writer2 = csv.writer(out2) 
     reader2 = csv.reader(inp2) 
     row_count = sum(1 for row in reader2) 
     last_line_index = row_count - 3 
     for row in islice(reader2, 7, last_line_index): 
       writer2.writerow(row) 
     out2.close() 

謝謝!

回答

2

有罪的一方是

row_count = sum(1 for row in reader2) 

reader2讀取所有的數據;現在當您嘗試for row in islice(reader2, 7, last_line_index)時,您不會收到任何數據。

而且,你可能看過很多空白行,因爲你打開該文件爲二進制;而不是做

with open('file.csv', newline='') as inf: 
    rd = csv.reader(inf) 
+0

這確實是問題!加快速度!你太快了!..... –

+0

啊哈!我不知道這個閱讀是一次性交易!非常感謝您的快速響應! – yungblud

1

您可以快速修復這樣的代碼(我評論的問題行了,因爲@Hugh博思韋爾說,你已經從變量reader2讀取所有數據):

import os 
import csv 
import glob 
from itertools import islice 

files = glob.glob('/Users/foo/bar/*.csv') 

# Loop through all of the csv's 
for file in files: 
    # Get the filename from the path 
    outfile = os.path.basename(file) 

    with open(file, 'rb') as inp, open('/Users/foo/baz/' + outfile, 'wb') as out: 

     reader = csv.reader(inp) 
     writer = csv.writer(out) 
     for row in reader: 
      if row: 
       writer.writerow(row) 
     out.close() 

    with open('/Users/foo/baz/' + outfile, 'rb') as inp2, open('/Users/foo/qux/' + outfile, 'wb') as out2: 
      writer2 = csv.writer(out2) 
      reader2 = csv.reader(inp2) 
      row_count = sum(1 for row in csv.reader(inp2)) #here you separately count the amount of rows without read the variable reader2 
      last_line_index = row_count - 3 
      for row in islice(reader2, 7, last_line_index): 
        writer2.writerow(row) 
      out2.close() 
+0

我很欣賞它的男人!你的解決方案也是完美的,休剛剛收到我的收件箱,速度更快:) – yungblud