如何使用bigrams和trigrams使用整齊文本

Question

我想使用tidytext同時使用bigram和trigram。我可以使用什麼代碼來查找2個和3個單詞。

這是隻使用雙字母組代碼：

library(tidytext) 
library(janeaustenr) 

austen_bigrams <- austen_books() %>% 
    unnest_tokens(bigram, text, token = "ngrams", n = 2) 

austen_bigrams 

Answer 1

如果你看一下?unnest_tokens，它會告訴你...是傳遞給標記生成器的參數。對於n元語法，這是tokenizers::tokenize_ngrams，如果你看一下它的幫助文件，它有一個n_min參數，所以你可以做

library(magrittr) 
library(tidytext) 
library(janeaustenr) 

austen_bigrams <- austen_books() %>% 
    head(1000) %>% # otherwise this will get very large 
    unnest_tokens(bigram, text, token = "ngrams", n = 3, n_min = 2) 

austen_bigrams 
#> # A tibble: 19,801 x 2 
#>     book    bigram 
#>     <fctr>     <chr> 
#> 1 Sense & Sensibility    sense and 
#> 2 Sense & Sensibility sense and sensibility 
#> 3 Sense & Sensibility  and sensibility 
#> 4 Sense & Sensibility and sensibility by 
#> 5 Sense & Sensibility  sensibility by 
#> 6 Sense & Sensibility sensibility by jane 
#> 7 Sense & Sensibility    by jane 
#> 8 Sense & Sensibility  by jane austen 
#> 9 Sense & Sensibility   jane austen 
#> 10 Sense & Sensibility  jane austen 1811 
#> # ... with 19,791 more rows