我用這個函數。 其基本思想是讓生成器生成文件中的數字。
def iter_loadtxt(filename, delimiter=',', skiprows=0, read_range=None, dtype=float):
'''
Read the file line by line and convert it to Numpy array.
:param delimiter: character
:param skiprows : int
:param read_range: [int, int] or None. set it to None and the function will read the whole file.
:param dtype: type
'''
def iter_func():
with open(filename, 'r') as infile:
for _ in range(skiprows):
next(infile)
if read_range is None:
for line in infile:
line = line.rstrip().split(delimiter)
for item in line:
yield dtype(item)
else:
counter = 0
for line in infile:
if counter < read_range[0]:
counter += 1
else:
counter += 1
for item in line:
yield dtype(item)
if counter >= read_range[1]:
break
iter_loadtxt.rowlength = len(line)
data = np.fromiter(iter_func(), dtype=dtype)
data = data.reshape((-1, iter_loadtxt.rowlength))
return data
你看過http://pandas.pydata.org/嗎?我自己並沒有使用熊貓,但我認爲這對於這類事情很有用。 –
除了在用於創建閱讀器的文件對象上設置'buffering'參數還有更多嗎?所有的Python對象都不需要進一步的努力就可以支持懶惰評估,你只是想確保文件不會試圖將整個內容拉入內存。 –