Part of creating “smart” web applications is understanding the semantics of what people type in, which implies using natural language processing.  Natural language processing software examines unstructured documents, and generates structured metadata that computers can handle.

Our application needed to understand phrases that people enter into a web browser.  We found three different approaches to handling this unstructured text:

SaaS APIs

These are hosted applications. All offer limited services at no charge, commercial services are generally pretty expensive. The major players appear to be:

Zemanta: offers an API with automatic tagging, among many other features.

OpenCalais: while it is by no means “open”, this API is powered by Reuters – which means that their “corpus” (body of words understood by the system) was composed using one of the world’s largest and most accurate volumes of text.

Alchemy API: offers automated categorization, tagging, keywords, etc.

NLP Toolkits

These are open-source toolkits (APIs that you can install on your own server) for analysis of unstructured text. Learning how to apply one of these might take a considerable effort – someone would have to learn at least the basics of NLP, to apply this software, or you might choose to hire a consultant with the the skills to develop this part of the application.

NLTK.org: a library written in Python, started in 2005, has been slowly creeping towards release 1.0 for the past year or so. While relatively young, it may be based on newer research than some of the more mature NLP libraries. Many corpora, grammar collections and trained models ready to use.

GATE: General Architecture for Text Engineering. Stable and proven toolkit for Java – this project started in 1995. Countless subprojects leverage this toolkit for various purposes.

FreeLing: Widely used toolkit in C++, with APIs for Java, PERL and Python. Online demos of this library demonstrate graphically how a short sentence can be broken down to a kind of tree-structure (nested subject/object, verb/adverb, etc.)

These are just a few examples – there are so many toolkits, and applications using these toolkits, that it would be impossible to make a choice based on a superficial analysis. To make a qualified choice, we would need to study at least the basics, or we would need the help of someone who knows enough about it to make a recommendation based on our needs.

Roll-your-own

Using e.g. MySQL, the Porter stemmer, a stop-word list and various other techniques to roll a basic search engine. Perhaps throw in a Bayesian text similarity measurement, to help rank the results and create stronger/weaker links between tables of keywords and posts.

It’s not NLP, and it’s not “web 3.0″, or “the semantic web” that everyone is buzzing about these days – because it does not understand semantics, and this will not yield the same kind of results – NLP systems “understand” unstructured text, where words like “not” and “really” can reverse or amplify the meaning of a subject – whereas anything you can roll on your own would most likely just recognize and consider these words “stop words” (ignoring them).