Asked By – Shifu
I’m looking for a way to split a text into n-grams.
Normally I would do something like:
import nltk from nltk import bigrams string = "I really like python, it's pretty awesome." string_bigrams = bigrams(string) print string_bigrams
I am aware that nltk only offers bigrams and trigrams, but is there a way to split my text in four-grams, five-grams or even hundred-grams?
Now we will see solution for issue: n-grams in python, four, five, six grams?
Great native python based answers given by other users. But here’s the
nltk approach (just in case, the OP gets penalized for reinventing what’s already existing in the
There is an ngram module that people seldom use in
nltk. It’s not because it’s hard to read ngrams, but training a model base on ngrams where n > 3 will result in much data sparsity.
from nltk import ngrams sentence = 'this is a foo bar sentences and i want to ngramize it' n = 6 sixgrams = ngrams(sentence.split(), n) for grams in sixgrams: print grams