Unified Framework Brings Fresh Approach to Text Classification

RTI International’s gobbli bridges natural language processing techniques and real-world challenges to innovate in the text classification field.

How did we get here?

Imagine having to completely relearn how to read every time you picked up a new book. Until recently, this was the process used by machine learning algorithms to solve problems like sentiment analysis or document classification. These algorithms—unable to carry information between tasks or understand anything about general language—were only capable of solving the narrow problem defined by the data they learned from.

In the last year, there’s been a complete paradigm shift in how these problems are solved, known by experts as the ImageNet moment. For example, computer vision now allows machines to detect pneumonia from X-rays, classify diabetic retinopathy in fungus photographs, and catalog satellite imagery into residential areas for surveys. But while the computer vision applications have had time to flourish, the world of text is just getting a first taste of potential improvements.

At the heart of recent improvements is a concept known as transfer learning. In machine learning, an algorithm uses patterns in example data to learn statistical rules that associate an input with an output. For each new task, a model has to relearn unique rules for that specific task and can’t carry information between problems. Transfer learning solves this problem by first modeling how an entire language works and then fine-tuning this language model for specific tasks. In this way, the algorithm is able to learn from text similar to how humans learn: using our general understanding of language applied to a specific task, without having to relearn every time we start a new task.

Proven success in practice

This unified framework has already proven useful on text classification projects. The RTI International Employee Survey used gobbli to consistently evaluate models and develop a final model that provided suggestions for how free-text responses should be coded into thematic categories. The suggested codes helped the team by pointing out the most likely relevant themes for a given response, while the qualitative team read through every comment and reviewed the suggestions for accuracy.

“The automated codes really helped speed up the coding process this year," said Dawn Ohse, RTI Employee Survey Project Manager. "It was surprising the number of times it coded some fairly nuanced sentiment accurately. Given its high predictive accuracy it has great potential to make the coding process less and less time intensive in future administrations of the survey.”

The survey is a great example of the type of problem where transfer learning shines. Using 3,000 responses from the previous year's survey we built an algorithm that made coding suggestions for 25 unique codes. Typically, this type of problem sees poor results due to the complexity of the themes and the low number of examples to learn from. However, because transfer learning takes advantage of a language model, we can see much better performance from fewer examples.

Open source for future advancement

As is standard practice in the machine learning research community, gobbli will be released as open source on RTI International's GitHub. Releasing this software as open source has several benefits, including improving the quality of the software and helping advance the field by allowing others to apply gobbli to their own problems and contribute back their improvements.

Transfer learning shows great potential for solving text classification problems in situations with low amounts of data. The unified interface with gobbli, will allow applied practitioners to use and evaluate text classification methodologies and deliver the best solution for the problem at hand.

Visit RTI International's Github today to download gobbli or check out this post for more technical information on getting started.