IT Tutorial

Introduction to Natural Language Processing in Python – (Simple text preprocessing)

Why preprocess ?

 

Text preprocessing with Python:

from nltk.corpus import stopwords

text = """The cat is in the box. The cat likes the box.
The box is over the cat."""
tokens = [w for w in word_tokenize(text.lower())
if w.isalpha()]
no_stops = [t for t in tokens
if t not in stopwords.words('english')]
Counter(no_stops).most_common(2)

In the previous article, the results of a similar sample were different. We got more meaningful results in this example.

You can read the previous article below

Introduction to Natural Language Processing in Python – (Words counts with bag-of-words )

 

 3,088 views last month,  2 views today

Exit mobile version