Description
text2token is a nodejs module that breaks down a corpus of text into lines and tokens.
text2token alternatives and similar modules
Based on the "Natural Language Processing" category.
Alternatively, view text2token alternatives based on common mentions on social networks and blogs.
-
nlp.js
An NLP library for building bots, with entity extraction, sentiment analysis, automatic language identify, and so more -
leven
Measure the difference between two strings with the fastest JS implementation of the Levenshtein distance algorithm
SurveyJS - Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App
* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest.
Do you think we are missing an alternative of text2token or a related project?
README
text2token
is a nodejs module that breaks down a corpus of text into lines and tokens.
Install
$ npm install text2token
Usage
The module has one method: text2token
, which returns an object that contains a list of each line
in your text file, as well as a list of all unique tokens
.
$ node
>
> var lib = require('text2token');
> var converted = lib.text2token('./src/bigtext.txt')
> converted.tokens
[ '©',
'2015',
'GitHub,',
'Inc.',
'Terms',
'Privacy',
'Security',
..........
> converted.lines
[ '© 2015 GitHub, Inc. Terms Privacy Security Contact Help',
'Status API Training Shop Blog About Pricing',
'The quick brown fox jumped over the lazy dog'
.......
MIT License 2015-2016 © Andy Craze & Contributors
*Note that all licence references and agreements mentioned in the text2token README section above
are relevant to that project's source code only.