熊貓read_csv和UTF-16

我有一個用UTF-16編碼的CSV文本文件（以便在別人使用Excel時保留Unicode字符），但是當用熊貓0.9.0做一個read_csv時，我得到這個神祕的錯誤：熊貓read_csv和UTF-16

df = pd.read_csv('data.txt',encoding='utf-16',sep='\t',header=0) 
df.head() 

--------------------------------------------------------------------------- 
Exception         Traceback (most recent call last) 
<ipython-input-18-85da1383cd9e> in <module>() 
----> 1 df = pd.read_csv('candidates-spanish.txt',encoding='utf-16',sep='\t',header=0) 
    2 df.head() 

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/io/parsers.pyc in read_csv(filepath_or_buffer, sep, dialect, header, index_col, names, skiprows, na_values, keep_default_na, thousands, comment, parse_dates, keep_date_col, dayfirst, date_parser, nrows, iterator, chunksize, skip_footer, converters, verbose, delimiter, encoding, squeeze, **kwds) 
248   kdict['delimiter'] = sep 
249 
--> 250  return _read(TextParser, filepath_or_buffer, kdict) 
251 
252 @Appender(_read_table_doc) 

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/io/parsers.pyc in _read(cls, filepath_or_buffer, kwds) 
198   return parser 
199 
--> 200  return parser.get_chunk() 
201 
202 @Appender(_read_csv_doc) 

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/io/parsers.pyc in get_chunk(self, rows) 
853   elif not self._has_complex_date_col: 
854    index = self._get_simple_index(alldata, columns) 
--> 855    index = self._agg_index(index) 
856 
857   elif self._has_complex_date_col: 

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/io/parsers.pyc in _agg_index(self, index, try_parse_dates) 
980     arr, _ = _convert_types(arr, col_na_values) 
981     arrays.append(arr) 
--> 982    index = MultiIndex.from_arrays(arrays, names=self.index_name) 
983   return index 
984 

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/index.pyc in from_arrays(cls, arrays, sortorder, names) 
1570 
1571   return MultiIndex(levels=levels, labels=labels, 
-> 1572       sortorder=sortorder, names=names) 
1573 
1574  @classmethod 

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/index.pyc in __new__(cls, levels, labels, sortorder, names) 
1254   assert(len(levels) == len(labels)) 
1255   if len(levels) == 0: 
-> 1256    raise Exception('Must pass non-zero number of levels/labels') 
1257 
1258   if len(levels) == 1: 

Exception: Must pass non-zero number of levels/labels

閱讀在行由符合csv.reader基於this example意味着我的數據是不是格式不正確的數據：

from io import BytesIO 
import csv 

with open('data.txt','rb') as f: 
    r = f.read().decode('utf-16').encode('utf-8') 
    for l in csv.reader(BytesIO(r),delimiter='\t'): 
     print l 

['Country', 'State/City', 'Title', 'Date', 'Catalogue', 'Wikipedia Election Page', 'Wikipedia Individual Page', 'Electoral Institution in Country', 'Twitter', 'CANDIDATE NAME 1', 'CANDIDATE NAME 2'] 
['Venezuela', 'N/A', 'President', '10/7/12', 'Hugo Rafael Chavez Frias', 'Hugo Ch\xc3\xa1vez', 'Hugo Ch\xc3\xa1vez', 'Hugo Chavez', 'Hugo Ch\xc3\xa1vez Fr\xc3\xadas', 'Hugo Chavez', 'Hugo Ch\xc3\xa1vez'] 
['Venezuela', 'N/A', 'President', '10/7/12', 'Henrique Capriles Radonski', 'Henrique Capriles Radonski', 'Henrique Capriles Radonski', 'Henrique Capriles Radonski', 'Henrique Capriles R.', 'Henrique Capriles', '']

有一些預處理，在read_csv一個另外的選擇或者在pandas.read_csv可以讀取utf-16文件之前需要做的其他事情？謝謝！

來源

2012-12-03 Brian Keegan

可以爲您郵寄/電子郵件一個版本的文本文件？我會看一看。 –

http://www.brianckeegan.com/data/candidates-spanish.txt –

這是一個錯誤，我想是因爲CSV讀者是傳回一個額外的空行的開頭。它的工作對我關於Python 2.7.3和0.9.1大熊貓，如果我這樣做：

In [36]: pd.read_csv(BytesIO(fh.read().decode('UTF-16').encode('UTF-8')), sep='\t', header=0) 
Out[36]: 
<class 'pandas.core.frame.DataFrame'> 
Int64Index: 50 entries, 0 to 49 
Data columns: 
Country        43 non-null values 
State/City       43 non-null values 
Title        43 non-null values 
Date        43 non-null values 
Catalogue       43 non-null values 
Wikipedia Election Page    43 non-null values 
Wikipedia Individual Page   43 non-null values 
Electoral Institution in Country 43 non-null values 
Twitter        43 non-null values 
CANDIDATE NAME 1     43 non-null values 
CANDIDATE NAME 2     16 non-null values 
dtypes: object(11)

我報告的錯誤的位置：https://github.com/pydata/pandas/issues/2418 在GitHub上掌握它的不幸引起了C-解析器段錯誤。我們會解決它。

現在，有趣的是：https://softwareengineering.stackexchange.com/questions/102205/should-utf-16-be-considered-harmful;）

來源

2012-12-03 22:42:58

如果Excel使用UTF-8很好地播放，那麼我會使用它！ :) –

現在所有設置在git master –

@BrianKeegan已經停止使用Excel：p –

from StringIO import StringIO 
import pandas as pd 

a = ['Venezuela', 'N/A', 'President', '10/7/12', 'Hugo Rafael Chavez Frias', 'Hugo Ch\xc3\xa1vez', 'Hugo Ch\xc3\xa1vez', 'Hugo Chavez', 'Hugo Ch\xc3\xa1vez Fr\xc3\xadas', 'Hugo Chavez', 'Hugo Ch\xc3\xa1vez'] 

pd.read_csv(StringIO('\t'.join(a)), delimiter='\t')

作品在這裏可以上傳數據的頭，所以我可以測試

來源

2012-12-03 19:39:17 locojay

Python3：

with open('data.txt',encoding='UTF-16') as f: 
    df = pd.read_csv(f)

來源

2015-06-26 12:43:49 avances123

您現在可以直接將'encoding ='UTF-16''直接傳遞給'pd.read_csv（）'，那麼不需要'open（） '。 –

熊貓read_csv和UTF-16

回答

相關問題