閱讀MSR意譯語料庫爲大熊貓

我從MSR下載MSR複述語料庫，並試圖把它載入數據幀，但得到了以下錯誤：閱讀MSR意譯語料庫爲大熊貓

import pandas as pd 
df = pd.read_csv(r'C:\MSRParaphraseCorpus\msr_paraphrase_test.txt', sep = '\t')

錯誤：

CParserError        Traceback (most recent call last) 
<ipython-input-10-35d992467320> in <module>() 
----> 1 df = pd.read_csv(r'C:\MSRParaphraseCorpus\msr_paraphrase_test.txt', sep = '\t') 

c:\python34\lib\site-packages\pandas\io\parsers.py in parser_f(filepath_or_buffer, sep, dialect, compression, doublequote, escapechar, quotechar, quoting, skipinitialspace, lineterminator, header, index_col, names, prefix, skiprows, skipfooter, skip_footer, na_values, na_fvalues, true_values, false_values, delimiter, converters, dtype, usecols, engine, delim_whitespace, as_recarray, na_filter, compact_ints, use_unsigned, low_memory, buffer_lines, warn_bad_lines, error_bad_lines, keep_default_na, thousands, comment, decimal, parse_dates, keep_date_col, dayfirst, date_parser, memory_map, float_precision, nrows, iterator, chunksize, verbose, encoding, squeeze, mangle_dupe_cols, tupleize_cols, infer_datetime_format, skip_blank_lines) 
    472      skip_blank_lines=skip_blank_lines) 
    473 
--> 474   return _read(filepath_or_buffer, kwds) 
    475 
    476  parser_f.__name__ = name 

c:\python34\lib\site-packages\pandas\io\parsers.py in _read(filepath_or_buffer, kwds) 
    258   return parser 
    259 
--> 260  return parser.read() 
    261 
    262 _parser_defaults = { 

c:\python34\lib\site-packages\pandas\io\parsers.py in read(self, nrows) 
    719     raise ValueError('skip_footer not supported for iteration') 
    720 
--> 721   ret = self._engine.read(nrows) 
    722 
    723   if self.options.get('as_recarray'): 

c:\python34\lib\site-packages\pandas\io\parsers.py in read(self, nrows) 
    1168 
    1169   try: 
-> 1170    data = self._reader.read(nrows) 
    1171   except StopIteration: 
    1172    if nrows is None: 

pandas\parser.pyx in pandas.parser.TextReader.read (pandas\parser.c:7566)() 

pandas\parser.pyx in pandas.parser.TextReader._read_low_memory (pandas\parser.c:7806)() 

pandas\parser.pyx in pandas.parser.TextReader._read_rows (pandas\parser.c:8423)() 

pandas\parser.pyx in pandas.parser.TextReader._tokenize_rows (pandas\parser.c:8297)() 

pandas\parser.pyx in pandas.parser.raise_parser_error (pandas\parser.c:20715)() 

CParserError: Error tokenizing data. C error: Expected 5 fields in line 34, saw 6

所以我看了看線34，看起來非常好。

fname = r'C:\MSRParaphraseCorpus\msr_paraphrase_test.txt' 
    with open(fname, encoding="utf8") as f: 
     content = f.readlines() 

    content[34]

輸出：

'0\t1268500\t1268733\tAgainst the Japanese currency, the euro was at 135.92/6.04 yen against the late New York level of 136.03/14.\tThe dollar was at 117.85 yen against the Japanese currency, up 0.1 percent.\n'

有什麼想法？

P.S.添加一些更多的細節，因爲SO會讓我發帖而沒有詳細信息...

來源

2017-05-29 user1700890

34行實際上是內容[33]和該行的問題是，最後一個字段有一個開始報價，但沒有結束。你需要做pd.read_csv（r'C：\ MSRParaphraseCorpus \ msr_paraphrase_test.txt'，sep ='\ t'，quotechar =''）讓csv閱讀器忽略引號。不作爲答案張貼，因爲在解決該問題之後嘗試加載文件還有其他問題。額外的問題不會跳出來，錯誤不是很清楚。但我會看起來更多一點。 – clockwatcher

問題在於第34行的開引號（正如我的評論中提到的）。通過傳入csv.QUOTE_NONE來禁用csv閱讀器的引用。嘗試：

import csv 
import pandas as pd 

df = pd.read_csv(r'C:\MSRParaphraseCorpus\msr_paraphrase_test.txt', sep = '\t', quoting=csv.QUOTE_NONE)

來源

2017-05-29 03:35:29 clockwatcher

閱讀MSR意譯語料庫爲大熊貓

回答

相關問題