將地圖縮減作業的輸出記錄到文本文件中

Question

我一直使用此jobclient.monitorandprintjob（）方法將地圖縮小作業的輸出打印到控制檯。我的用法是這樣的：

job_client.monitorAndPrintJob(job_conf, job_client.getJob(j.getAssignedJobID())) 

它的輸出如下（印在控制檯上）：

13/03/04 07:20:00 INFO mapred.JobClient: Running job: job_201302211725_10139<br> 
13/03/04 07:20:01 INFO mapred.JobClient: map 0% reduce 0%<br> 
13/03/04 07:20:08 INFO mapred.JobClient: map 100% reduce 0%<br> 
13/03/04 07:20:13 INFO mapred.JobClient: map 100% reduce 100%<br> 
13/03/04 07:20:13 INFO mapred.JobClient: Job complete: job_201302211725_10139<br> 
13/03/04 07:20:13 INFO mapred.JobClient: Counters: 26<br> 
13/03/04 07:20:13 INFO mapred.JobClient: Job Counters<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Launched reduce tasks=1<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Aggregate execution time of mappers(ms)=5539<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Total time spent by all reduces waiting after reserving slots (ms)=0<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Total time spent by all maps waiting after reserving slots (ms)=0<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Launched map tasks=2<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Data-local map tasks=2<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Aggregate execution time of reducers(ms)=4337<br> 
13/03/04 07:20:13 INFO mapred.JobClient: FileSystemCounters<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  MAPRFS_BYTES_READ=583<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  MAPRFS_BYTES_WRITTEN=394<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  FILE_BYTES_WRITTEN=140219<br> 
13/03/04 07:20:13 INFO mapred.JobClient: Map-Reduce Framework<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Map input records=6<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Reduce shuffle bytes=136<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Spilled Records=22<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Map output bytes=116<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  CPU_MILLISECONDS=1320<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Map input bytes=64<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Combine input records=13<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  SPLIT_RAW_BYTES=180<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Reduce input records=11<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Reduce input groups=11<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Combine output records=11<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  PHYSICAL_MEMORY_BYTES=734961664<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Reduce output records=11<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  VIRTUAL_MEMORY_BYTES=9751805952<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Map output records=13<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  GC time elapsed (ms)=0<br> 

我想上面的輸出/日誌在一個文本文件要打印，而不是控制檯。有什麼建議麼？

Answer 1

在您的HADOOP_HOME/conf中，您可能會發現一個文件名爲：log4j.properties。我相信你可以配置在哪裏以及如何登錄。

準確地說，你必須使用一個滾動文件附加目的地，所以你應當取消註釋（只是刪除#）從log4j.properties文件中以下行：

# Rolling File Appender 
# 

#log4j.appender.RFA=org.apache.log4j.RollingFileAppender 
#log4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file} 

# Logfile size and and 30-day backups 
#log4j.appender.RFA.MaxFileSize=1MB 
#log4j.appender.RFA.MaxBackupIndex=30 

#log4j.appender.RFA.layout=org.apache.log4j.PatternLayout 
#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} - %m%n 
#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n 

和自定義其他參數你喜歡。

有關log4j配置的更多信息，請參閱here。