Intelligent Document Processing (IDP) refers to a set of tools and solutions based on deep learning techniques to automate document processing. Leveraging the latest artificial intelligence and computer vision techniques, IDP is capable of handling documents (e.g., e-mail text, PDFs, and scans) of any nature and converting them into structured data. The IDP automates the processing of information contained in documents by understanding what the document is about, what information it contains, extracting it, and automatically making it usable to the process or department of interest.

IDP differs from optical character recognition (OCR). In fact, traditional OCR solutions aim to transform a scanned document into machine-readable text. IDP solutions not only read documents, but extract, classify and export relevant data (key value data, tables, images, etc.) to enable further processing or actions taken based on the results. This is possible through the synergy of different technologies such as OCR, Computer Vision, NLP and RPA. These technologies, used together, enable the highest rate of automation.

Often, a company lacks the specific skills required to implement such solutions at the expense of final system performance. For this reason, Intelligent Document Processing solutions are the solution to such shortcomings: they encapsulate specific skills and technologies in a single product that is easy to use and integrate.

In fact, IDP solutions tend to be "non-invasive" and easily integrated into existing systems, business applications and platforms. They also often offer a pre-configured range of off-the-shelf solutions up to more complex, customized implementations. Offering pre-built use cases makes it possible to automate or improve the quality of a process in less time than traditional solutions: from several months to a few days. Ownership costs of the solution are also greatly reduced, requiring little or no data for setup and minimal effort for integration.

Here are some examples of use cases that often come pre-configured: invoice processing, customer onboarding, mortgage file processing, contract or purchase receipt processing.

Intelligent Document Processing: the steps

Image pre-processing

In many Intelligent Document Processing solutions, the first step is to pre-process the image of the document that has been received (e.g., via scan or e-mail). Pre-processing allows OCR/ICR algorithms to improve performance and retain a "normalized" version of the image. The ultimate goal of pre-processing is to improve the quality and readability of the image. Typical pre-processing operations may involve image binarization, rotation angle correction, image resolution standardization, and several other operations. Some solutions do not require this step in order to process the document, but the step is still useful for preserving a more readable and clearer version of the document.

Text identification from the document and layout analysis

IDP solutions use Computer Vision techniques to understand the structure of the document and identify elements such as text, tables, and images. Typically this phase can be divided into:

• Layout analysis: step required to identify document structure (e.g., paragraphs and headings), tables and images contained;
• OCR: required for reading the document. Typically useful for further processing involved in the process.

During this process, IDP solutions essentially create a machine-readable version of the document. This version is then ready for later automated analysis. Solutions that do not use text for subsequent steps can skip or simplify this step.

Document classification

The process of classifying the document involves assigning each of its pages, or the document as a whole, a category automatically.

The classification of a document can be done by following different methodologies:

• through transcription and subsequent analysis of the text contained within;
• Through image analysis of the document;
• with hybrid techniques that involve analyzing both the text and its image.

Both supervised and unsupervised machine learning techniques can be used in the intelligent document processing workflow. The unsupervised approach has a lower cost in the setup phase (no data labeling phase is required) but typically offers lower accuracy. Based on the algorithm used, the model can also provide the user with a reliability score (Confidence Score) to represent the model's confidence with respect to its predictions. Depending on the technologies used, this step may be enabling for extraction or an optional step.

Information extraction

The extraction of information contained in the document is a fundamental step, necessary for the automation of processes related to document processing. In this step, the same methodologies presented for document classification (text analysis, image analysis, or both) can be used with their advantages or disadvantages.

In document processes, among the most time-consuming and costly operations is precisely the extraction of key information and subsequent manual entry.

This step aims to automatically transform the unstructured data in the document into structured data that can be easily used by the next steps or processes below. Many types of information such as: tables, images, signatures, etc. can be extracted in this step.

Validation of results

In most cases, Intelligent Document Processing solutions include a scoring or confidence mechanism useful for reviewing data identified as potentially erroneous. Thus, to ensure data accuracy and integrity, IDP platforms leverage human review, external databases and pre-configured vocabularies useful for validating data extracted from documents. This process not only ensures data quality, but improperly processed data can be collected by enabling continuous learning of the system(Human in the loop & Continuous learning).

Enrichment of results

Another important step, before making the data usable, is theenrichment of the extracted data. Typically, external databases or services are leveraged to add information to that extracted from the document in order to have more detailed and higher quality data. An example would be the look up on an external system from a company name to check the health of a company.

Integration

A final relevant aspect involves integration with the systems from which the data come and with the systems that will need to make use of them. IDP systems often offer very simple ways of no-code integrations, implemented directly through the platform or through the use of RPA tools. This step is critical to make the integration process easy and to have an end-to-end solution that interfaces with the adopted business tools.

Benefits

Intelligent Document Processing solutions enable companies to achieve several benefits:

• Direct cost savings
  Leveraging scalable, high-performance architectures reduces time and cost, dramatically lowering the effort to process large volumes of data;
• Reducing repetitive manual tasks
  With Artificial Intelligence, the need for manual intervention to process documents is minimized;
• Superior data quality
  Through continuous learning and validation, data quality improves, dramatically lowering errors;
• Start processing data quickly
  IDP solutions are easily integrated through RPA mechanisms, often 5-10 times faster and easier to integrate than other approaches;
• Process any document
  Thanks to Artificial Intelligence, it is possible to manage structured, semi-structured and unstructured documents of all kinds;
• Complete Automation
  Because of the simplicity with which the IDP integrates with other areas of the enterprise, a fully integrated RPA solution is easily achieved without the need for costly upgrades;
• Improved Productivity
  IDP helps organizations increase productivity and reduce time spent on repetitive tasks, improving the quality of the work environment;
• Easy to use for businesses
  With pre-packaged use cases to choose from, launching and integrating more common use cases is easier and faster.

myBiros and benefits

myBiros is an Intelligent Document Processing solution that enables automatic document processing. Core functionalities are information extraction and automatic document classification.

All this is offered through a prebuilt set of ready-to-use APIs for common use cases and the ability to retrain the entire pipeline (both the OCR engine and the document interpretation system) for custom cases. It is very easy to integrate myBiros into any application through the use of APIs and easy interaction with RPA systems.

By leveraging advanced deep learning techniques that analyze multimodal features, it is possible to process all document types with a single solution. The system uses pre-trained models, data-augmentation techniques, and for that reason can be trained with a small volume of data allowing even processes involving a small volume of documents to be automated.

This solution includes a scoring mechanism: the system reduces false positives by enabling the ability to review low confidence data while minimizing errors. Interaction with a human user enables the system to correct errors while continuing to train the system so that past mistakes are not repeated(human in the loop and continuous learning). Finally, the high scalability of the cloud-based architecture makes it possible to process highly variable masses of documents without having to allocate expensive resources in advance.

The features mentioned so far allow myBiros to perform optimally on any document allowing you to automate a wide variety of processes. If you are curious about how myBiros works in order to simplify document processing, contact us and try our demo. We are ready to help you!

‍