Are You Ready for Intelligent Document Processing?

Don’t call it “OCR” anymore.” Intelligent Document Processing (IDP), or what analyst firm Deep Analysis refers to as “cognitive capture,” is a far cry from the traditional method of applying brute force OCR to documents to create searchable content. Indeed, with the proliferation of digital documents, increasing OCR is unnecessary.

This begs the following question: if IDP software isn’t identical to the OCR-focused document capture of the past and promises increased performance, are we missing something? Are organizations even prepared to reap the benefits? To address those questions, we must first examine two significant distinctions between document capture and IDP software:

  1. Greater use of machine learning
  2. Ability to make sense of documents beyond forms.

Consider the following in reverse order.

The Evolution of Document Capture Expanded Document Support

When we look back at the history of document capture, we see that it all began with the simple process of converting scanned documents to searchable data. To accomplish this, software converted scanned text into computer text using optical character recognition (OCR), and it did so on a page-by-page basis, converting each page. Additionally, there was the option to manually add index data, all in the name of making documents easier to find. Then businesses realized they could use the same software to automate business processes requiring forms. Rather than performing OCR on each word on each page, a “template” could be created that instructed the software on which words to scan. Form data such as names, addresses, and other critical information would be retrieved and incorporated into a business process. Insurance claims data could be directly entered into a workflow. This worked exceptionally well due to the high level of standardization – you only needed a few templates to cover the various form layouts. Simple as that. Indeed, it worked so well that enterprising organizations began investigating alternative document-intensive processes that did not involve forms.

However, it became exceedingly difficult. Unlike a standard form, other document-based data is not nearly as consistent. Often referred to as “semi-structured documents,” the required data could be located in a hundred different locations and in a variety of different formats, necessitating the creation of a hundred or more templates. Nonetheless, for processes that generate many documents, spending several hundred hours developing and optimizing templates for each variant was worthwhile. However, the progression stalled – other business processes that could benefit lacked sufficient document volume to justify the configuration expense.

To overcome the template’s rigidity, software vendors developed more tolerant, flexible rule-based approaches. Rather than creating templates by drawing zones around fields, fields were located using their labels or keywords. The label “Date:” can be used to locate the remittance date value. “PO #” can be used to locate a purchase order number. Additionally, rules could be used to address the issue of automatically identifying documents. Rather than manually sorting documents and employing separator pages or barcodes, keywords could be used to distinguish a remittance from a fax cover page, for example.

While a rules-based approach meant that organizations no longer had to create and manage hundreds of templates and reduce manual preparation, this technique did require a significant amount of analysis of a lot of sample data. And the complexity grew significantly as simple templates were replaced by more complex rules that could include coding regular expressions. Hiring professional services to do all this work became the norm to get around the complexity. Some advances, like the ability to create knowledgebases of rules based upon user feedback, were made to make things simpler, but overall, the systems weren’t spared from this additional complexity. Again, adoption within organizations slowed. Organizations were content to apply document capture to the most expensive, least complex processes.

While a rules-based approach eliminated the need for organizations to create and manage hundreds of templates and reduced manual preparation, it did require extensive analysis of a large amount of sample data. And the complexity increased significantly as simple templates were phased out in favor of more complex rules that may include regular expression coding. To avoid the complexity, it became the norm to hire professional services to handle all of this work became the norm. While some advancements, such as the ability to create knowledgebases of rules based on user feedback, were made to simplify things, the systems were not spared from this added complexity. Again, adoption slowed within organizations. Organizations were content to use document capture to automate the simplest, most expensive processes.

Thus, the current state of document capture is primarily limited to the adoption of forms processing and some high-volume processing of semi-structured documents, mainly invoices. In maintenance mode – similar to the old COBOL programs. And when changes are required, most organizations rely heavily on a cadre of professional services staff to perform the necessary work – becoming experts in these systems is a bridge too far. Something must change, as documents aren’t going away.

To learn more, visit