如何從warc文件中讀取記錄的子集

我試圖用Python從Common Crawl解析.warc文件。如何從warc文件中讀取記錄的子集

由於文件很大，我想從頭幾個記錄的樣本/子集開始。

如何截斷文件以僅包含第一個X行，同時保留已安裝的換行符/回車符？

這是我已經試過：

head -n 250 oldfile > newfile這消除了一些需要解析該文件的回報。下面是我的錯誤，如果我嘗試使用這個文件在我的Hadoop作業（與warc包看完）：

Traceback (most recent call last): 
     File "test.py", line 46, in <module> 
     TagGrabber.run() 
     File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/job.py", line 461, in run 
     mr_job.execute() 
     File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/job.py", line 479, in execute 
     super(MRJob, self).execute() 
     File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/launch.py", line 151, in execute 
     self.run_job() 
     File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/launch.py", line 214, in run_job 
     runner.run() 
     File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/runner.py", line 464, in run 
     self._run() 
     File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/sim.py", line 173, in _run 
     self._invoke_step(step_num, 'mapper') 
     File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/sim.py", line 264, in _invoke_step 
     self.per_step_runner_finish(step_num) 
     File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/local.py", line 152, in per_step_runner_finish 
     self._wait_for_process(proc_dict, step_num) 
     File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/local.py", line 268, in _wait_for_process 
     (proc_dict['args'], returncode, ''.join(tb_lines))) 
    Exception: Command ['sh', '-ex', 'setup-wrapper.sh', '/var/cc-mrjob/venv/bin/python', 'test.py', '--step-num=0', '--mapper', '/tmp/test.root.20150520.071726.549519/input_part-00000'] returned non-zero exit status 1: 
    Traceback (most recent call last): 
     File "test.py", line 46, in <module> 
     TagGrabber.run() 
     File "/tmp/test.root.20150520.071726.549519/job_local_dir/0/mapper/0/mrjob.tar.gz/mrjob/job.py", line 461, in run 
     mr_job.execute() 
     File "/tmp/test.root.20150520.071726.549519/job_local_dir/0/mapper/0/mrjob.tar.gz/mrjob/job.py", line 470, in execute 
     self.run_mapper(self.options.step_num) 
     File "/tmp/test.root.20150520.071726.549519/job_local_dir/0/mapper/0/mrjob.tar.gz/mrjob/job.py", line 535, in run_mapper 
     for out_key, out_value in mapper(key, value) or(): 
     File "/var/cc-mrjob/mrcc.py", line 33, in mapper 
     for i, record in enumerate(f): 
     File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/warc/warc.py", line 390, in __iter__ 
     record = self.read_record() 
     File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/warc/warc.py", line 373, in read_record 
     header = self.read_header(fileobj) 
     File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/warc/warc.py", line 331, in read_header 
     raise IOError("Bad version line: %r" % version_line) 
    IOError: Bad version line: 'WARC/1.0\n'

同＃1，但tail命令
同＃1，但之後使用tr或sed替換丟失的換行符或^M（回車）字符。這導致warc程序包仍然抱怨預計的回車或換行符不到位。
unix2dos oldfile

來源

2015-05-20 okoboko

看看warc python lib，它不會一次讀取整個.warc文件，而是一次讀取一條記錄。你需要截斷什麼？一個誠實的問題，也許移動通過網絡或一些這樣的？ –

添加到前面的「它不會讀取整個.warc」，使用warc lib僅執行「讀取N條第一條記錄」非常簡單：'islice（warc_file，N）'，如果這正是您想要的。 –

@Ilja謝謝 - 這就是我一直在尋找的。你可以添加作爲答案？ – okoboko

這將是很難正確處理新行，因爲.warc文件可能包含二進制數據，以及。截斷也可能會產生損壞的.warc文件，例如python庫相信內容長度標頭是有效的。

warc python lib從.warc文件中一次只讀取一條記錄（避免一次讀取整個文件到內存），因此只能使用python處理子集。例如：

import warc 
from itertools import islice 

N = 10 
warc_file = warc.open('/path/to/file.warc') 
for record in islice(warc_file, N): 
    do_stuff_with(record)

來源

2015-05-21 10:07:00

如何從warc文件中讀取記錄的子集

回答

相關問題