Question
Asked By – Shifu
I’m looking for a way to split a text into n-grams.
Normally I would do something like:
import nltk
from nltk import bigrams
string = "I really like python, it's pretty awesome."
string_bigrams = bigrams(string)
print string_bigrams
I am aware that nltk only offers bigrams and trigrams, but is there a way to split my text in four-grams, five-grams or even hundred-grams?
Thanks!
Now we will see solution for issue: n-grams in python, four, five, six grams?
Answer
Great native python based answers given by other users. But here’s the nltk
approach (just in case, the OP gets penalized for reinventing what’s already existing in the nltk
library).
There is an ngram module that people seldom use in nltk
. It’s not because it’s hard to read ngrams, but training a model base on ngrams where n > 3 will result in much data sparsity.
from nltk import ngrams
sentence = 'this is a foo bar sentences and i want to ngramize it'
n = 6
sixgrams = ngrams(sentence.split(), n)
for grams in sixgrams:
print grams
This question is answered By – alvas
This answer is collected from stackoverflow and reviewed by FixPython community admins, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0