我很新的Map/Reduce原理和python mrjob框架,我寫了這個示例代碼,它工作正常,但我想知道我可以改變什麼它使它「完美」/更高效。如何優化這個MapReduce函數,Python,mrjob
from mrjob.job import MRJob
import operator
import re
# append result from each reducer
output_words = []
class MRSudo(MRJob):
def init_mapper(self):
# move list of tuples across mapper
self.words = []
def mapper(self, _, line):
command = line.split()[-1]
self.words.append((command, 1))
def final_mapper(self):
for word_pair in self.words:
yield word_pair
def reducer(self, command, count):
# append tuples to the list
output_words.append((command, sum(count)))
def final_reducer(self):
# Sort tuples in the list by occurence
map(operator.itemgetter(1), output_words)
sorted_words = sorted(output_words, key=operator.itemgetter(1), reverse=True)
for result in sorted_words:
yield result
def steps(self):
return [self.mr(mapper_init=self.init_mapper,
mapper=self.mapper,
mapper_final=self.final_mapper,
reducer=self.reducer,
reducer_final=self.final_reducer)]
if __name__ == '__main__':
MRSudo.run()
這聽起來更像是http://codereview.stackexchange.com的問題。 – 2013-04-05 20:34:52
感謝您指出了這一點,但我正在詢問如何提高此代碼的性能。 – Vor 2013-04-05 20:35:54
是的,這正是[codereview是有](http://codereview.stackexchange.com/faq)。 – 2013-04-05 20:36:52