如何提取Scala中的bigrams和trigrams？

Question

假設這些都是我的文檔：

very pleased product . phone lightweight comfortable sound quality good house yard . 
quality construction phone base unit good . ample supply cable adapter . plug computer soundcard . 
shop unit mail rebate . unit battery pack hold play time strap carr headphone adapter cable perfect digital copy optical. component micro plug stereo connector cable micro plug rca cable . 
unit primarily record guitar jam session . input plug provide power plug microphone . decent stereo mic need digital recording performance . mono mode double recording time . 
admit like new electronic toy . digital camera not impress . 

，我想提取所有的雙字母組，並從每個句子的出現次數卦，每一個文件內。

，我已經試過：

case class trigram(first: String, second: String,third: String) { 
    def mkReplacement(s: String) = s.replaceAll(first + " " + second + " " + third, first + "-" + second + "-" + third) 
} 

def stringToTrigrams(s: String) = { 
    val words = s.split(".") 
    if (words.size >= 3) { 
    words.sliding(3).map(a => tigram(a(0),a(1),a(2))) 
    } 
    else 
    Iterator[tigram]() 
} 

val conf = new SparkConf() 
val sc = new SparkContext(conf) 
val data = sc.textFile("docs") 

val trigrams = data.flatMap { 
    stringToTrigrams 
}.collect() 

val trigramCounts = trigrams.groupBy(identity).mapValues(_.size) 

，但它不顯示任何卦？

Answer 1

def stringToTrigrams(s: String) = { 
    val words = s.split(".") 
    if (words.size >= 3) { 
    words.sliding(3).map(a => trigram(a(0),a(1),a(2))) 
    } else Iterator[trigram]() 
} 

IIUC，這個函數是把上面的整個文件，然後把文件拆分成「。」。這是你的第一個問題。調用split（「。」）不會做你認爲它做的事情。你實際上是在通配符而不是「」上分割。像你要的那樣。將其更改爲「\」。你會把文件分成幾個句子。

一旦完成，我們需要通過簡單地分割空格來分割句子，我建議通過做_.split(\\s+)來分割所有的空格。現在，你應該能夠通過的話來分析，並使用功能卦像這樣：

def stringToTrigrams(s: String) = { 
    val sentences = s.split("\\.") 
    sentences flatMap { sent => 
    val words = sent.split("\\s+").filter(_ != "") 
    if (words.length >= 3) 
     words.sliding(3).map(a => trigram(a(0), a(1), a(2)) 
    else Iterator[trigram] 
    } 
} 

希望這有助於。