在Prolog中計算字詞頻率

Question

我搜索了Google無濟於事。這裏的任何人都可以告訴我如何在Prolog中對術語頻率進行編碼？越「邏輯」越好。 給定一個文本文件，忽略非字母數字字符，檢測單詞，刪除停用詞（可能在外部文件中給出），計算文件中每個單詞的出現次數並輸出按照降低頻率排序的「word：freq」。

謝謝！

（這不是功課，順便說一句，這是我做的一個項目）

Answer 1

這裏的東西，讓你開始：

它假設的話是每個輸入的單獨的行文件

無法處理 「停止詞」

沒有實現那種

; Print the term frequency for words in file Path 
print_freq_from_file(Path) :- 
    open(Path, read, Stream), 
    read_file(Stream, Words), 
    calc_freq(Words, Freq), 
    sort_freq(Freq, SortedFreq), 
    print_freq(SortedFreq). 

; calc_freq(Words,Freq) Freq is the term frequency of the words in list Words 
calc_freq([], []). 
calc_freq([Word|Words], Freq) :- 
    calc_freq(Words, FreqRest), 
    add_word(Word, FreqRest, Freq). 

; add_word(Word, Freq1, Freq2) Freq2 is Freq1 with Word added 
add_word(Word, [], [(Word,1)]). 
add_word(Word, [(Word,N)|Rest], [(Word,N1)|Rest]) :- 
    N1 is N+1. 
add_word(Word, [Term|Rest], [Term|NewRest]) :- 
    add_word(Word, Rest, NewRest). 

; Print the given term frequency 
print_freq([]). 
print_freq([(W,N)|Rest]) :- 
    write(W), write(' : '), write(N), nl, 
    print_freq(Rest). 

Answer 2

對於過濾停止的話，一個好的辦法是通過assert：

read_sw(Path) :- 
    open(Path, read, Stream), 
    read_words_file(Stream, Words), 
    add_sw(Words). 

add_sw([]). 
add_sw([Word|Rest]) :- 
    assertz(stop_word(Word)), 
    add_sw(Rest). 

一旦做到這一點，你可以用stop_word/1檢查一個詞是停用詞，並忽略它。舉例來說，如果你正在實施斯科特·亨特的回答，您可以在calc_freq/2增加一個附加條款，以篩選出那些話：

calc_freq([], []). 
calc_freq([Word|Words], Freq) :- 
    stop_word(Word), !, 
    calc_freq(Words, Freq). 
calc_freq([Word|Words], Freq) :- 
    calc_freq(Words, FreqRest), 
    add_word(Word, FreqRest, Freq). 

Answer 3

我用這個答案SWI-Prolog的豐富庫支持，因此可能是不合適的案件。

當然，你是如何解決任何編程任務強力地什麼在你選擇的語言，從你使用它的能力和可用的影響。

這裏我使用library(assoc)，如果你的系統錯過了它，那麼你可以使用列表模擬，或者使用assert/retract。

這個片段只是顯示了一個（慣用）的方式計數單詞，計算頻率可以很容易地與庫（聚合）或一些算術，可能你想嘗試記下自己在鍛鍊語言...

/* File: frequency_of_words.pl 
    Author: Carlo,,, 
    Created: May 23 2012 
    Purpose: http://stackoverflow.com/questions/10711483/calculating-term-frequency-in-prolog 
*/ 

:- module(frequency_of_words, [count_words/2, count_words/1]). 
:- [library(assoc)]. 

count_words(File, Assoc) :- 
    empty_assoc(Empty), 
    open(File, read, Stream), 
    frequency_of_words(Stream, Empty, Assoc, ""), 
    close(Stream). 
count_words(File) :- 
    count_words(File, Assoc), 
    assoc_to_list(Assoc, List), 
    maplist(writeln, List). 

frequency_of_words(Stream, SoFar, Words, CurrWord) :- 
    get_code(Stream, Code), 
    ( Code == -1 
    -> update_dictionary(SoFar, Words, CurrWord) 
    ; use_character(Code, SoFar, Updated, CurrWord, NextWord), 
     frequency_of_words(Stream, Updated, Words, NextWord) 
    ). 

update_dictionary(SoFar, SoFar, Word) :- 
    skip_word(Word). 
update_dictionary(SoFar, Updated, Codes) :- 
    atom_codes(Word, Codes), 
    (get_assoc(Word, SoFar, CountSoFar) ; CountSoFar = 0), 
    WordCount is CountSoFar + 1, 
    put_assoc(Word, SoFar, WordCount, Updated). 

use_character(Code, SoFar, Updated, CurrWord, NextWord) :- 
    ( word_character(Code) 
    -> Updated = SoFar, 
     NextWord = [Code|CurrWord] 
    ; reverse(CurrWord, Forward), 
     update_dictionary(SoFar, Updated, Forward), 
     NextWord = "" 
    ). 

word_character(Code) :- 
    [Code] @>= "A", [Code] @=< "Z" ; 
    [Code] @>= "a", [Code] @=< "z" ; 
    [Code] @>= "0", [Code] @=< "9" ; 
    [Code] == "_". 

skip_word(""). % a trick on EOF or consecutive blanks: not really a skipword 
skip_word("is"). 

測試：

?- count_words('frequency_of_words.pl'). 
0-2 
1-3 
2012-1 
23-1 
9-1 
A-1 
Author-1 
Carlo-1 
Code-14 
Codes-2 
CountSoFar-3 
Created-1 
CurrWord-6 
EOF-1 
Empty-2 
File-3 
Forward-2 
Full-2 
List-2 
May-1 
NextWord-5 
Purpose-1 
SoFar-11 
StackOverflow-1 
Stream-6 
Updated-7 
Word-5 
WordCount-2 
Words-3 
Z-1 
_-1 
a-3 
answer-1 
assoc-1 
assoc_to_list-1 
atom_codes-1 
blanks-1 
close-1 
consecutive-1 
count_words-2 
empty_assoc-1 
frequency_of_words-5 
get_assoc-1 
get_code-1 
library-1 
maplist-1 
module-1 
not-1 
on-1 
open-1 
or-1 
pl-1 
put_assoc-1 
read-1 
really-1 
reverse-1 
skip_word-3 
skipword-1 
trick-1 
update_dictionary-4 
use_character-2 
word_character-2 
writeln-1 
z-1