Popularity
0.4
Stable
Activity
0.0
Stable
1
3
2

Description

text2token is a nodejs module that breaks down a corpus of text into lines and tokens.

Programming language: JavaScript
License: MIT License
Tags: Utility     Text     Natural Language Processing     String     Token     Nlp     Tokens     Strings    
Latest version: v1.1.1

text2token alternatives and similar modules

Based on the "Natural Language Processing" category.
Alternatively, view text2token alternatives based on common mentions on social networks and blogs.

Do you think we are missing an alternative of text2token or a related project?

Add another 'Natural Language Processing' Module

README

travis build Codecov version license semantic-release

text2token

is a nodejs module that breaks down a corpus of text into lines and tokens.

Install

$ npm install text2token

Usage

The module has one method: text2token, which returns an object that contains a list of each line in your text file, as well as a list of all unique tokens.

$ node
> 
> var lib = require('text2token');

> var converted = lib.text2token('./src/bigtext.txt')

> converted.tokens
  [ '©',
  '2015',
  'GitHub,',
  'Inc.',
  'Terms',
  'Privacy',
  'Security',
  ..........

> converted.lines

[ '© 2015 GitHub, Inc. Terms Privacy Security Contact Help',
  'Status API Training Shop Blog About Pricing',
  'The quick brown fox jumped over the lazy dog'
 .......

MIT License 2015-2016 © Andy Craze & Contributors


*Note that all licence references and agreements mentioned in the text2token README section above are relevant to that project's source code only.