Skip to content

Add tokenization of words #6

@mattlockyer

Description

@mattlockyer

Adding a tokenization library or making this an optional input could really help.

For instance, what if I wanted to search only top sentences related to "food" for recipes or "locations" for scanning destinations in blog posts?

I realize I can do this using the chain phrase myself but basic NLP entity extraction would be nice to have so we can group by broader categories.

Also, the term "chain phrase" implies that the words will be found in order, like in a "chain". It would be trivial and somewhat useful to add a "preserve order" option so we can proximity search within phrases.
e.g. "captain james kirk" should have preserve order so we can find "captain, james t. kirk" but we don't need "kirk captain james". Suggestion.

Otherwise the library works really well, it's clean and straightforward, thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions