Intelligent Document Processing (IDP) refers to a set of tools and solutions based on deep learning techniques to automate document processing. Leveraging the latest artificial intelligence and computer vision techniques, IDP is capable of handling documents (e.g. e-mail text, PDFs and scanned documents) of any nature and converting them into structured data. The IDP automates the processing of information contained in documents: it understands what the document is about, what information it contains, extracts it and makes it automatically available to the process or department of interest.
IDP differs from optical character recognition (OCR). In fact, traditional OCR solutions aim to transform a scanned document into machine-readable text. IDP solutions not only read documents, but also extract, classify and export relevant data (key value data, tables, images, etc.) to enable further processing or action based on the results. This is possible thanks to the synergy of different technologies such as OCR, Computer Vision, NLP and RPA. These technologies, used together, enable the highest automation rate.
Often, a company lacks the specific skills required to implement such solutions, to the detriment of the final performance of the system. For this reason, IDP’s solutions are the solution to these shortcomings: they encapsulate specific skills and technologies in a single product that is simple to use and easy to integrate.
Infatti, le soluzioni IDP tendono ad essere “non invasive” e facilmente integrabili nei sistemi, nelle applicazioni aziendali e nelle piattaforme In fact, IDP solutions tend to be ‘non-invasive’ and easily integrated into existing systems, business applications and platforms. IDPs also often offer a pre-configured range of ready-to-use solutions up to more complex, customized implementations. Offering pre-built use cases makes it possible to automate or improve the quality of a process in less time than traditional solutions: from several months to a few days. Ownership costs of the solution are also greatly reduced, requiring little or no data for setup and minimal effort for integration.
Here are some examples of use cases that often come pre-configured: invoice processing, customer onboarding, mortgage processing, contract processing or purchase receipt processing.
Intelligent Document Processing: steps
Image pre-processing
In molIn many IDP solutions, the first step is to pre-process the image of the document that has been received (e.g. by scanning or e-mail). Pre-processing improves the performance of OCR/ICR algorithms and preserves a ‘normalised’ version of the image. The goal of pre-processing is to improve the quality and readability of the image. Typical pre-processing operations may involve image binarisation, rotation angle correction, image resolution standardisation and other operations. Some solutions do not require this step in order to process the document, but the step is nevertheless useful for preserving a more readable and clearer version of the document.
Text identification from the document and layout analysis
IDP solutions use Computer Vision techniques to understand document structure and identify elements such as text, tables and images. Typically, this step can be divided into:
- Layout analysis: step required to identify the structure of the document (e.g. paragraphs and headings), tables and images contained;
- OCR: necessary for reading the document. Typically useful for further processing in the process.
During this process, IDP solutions essentially create a machine-readable version of the document. This version is thus ready for subsequent automatic analysis. Solutions that do not use text for subsequent steps can skip or simplify this step.
Document classification
The document classification process involves assigning each of its pages, or the document as a whole, a category automatically.
The classification of a document can be carried out using different methods:
- through transcription and subsequent analysis of the text contained within it;
- through image analysis of the document;
- through hybrid techniques that involve analyzing both the text and its image.
Both supervised and unsupervised machine learning techniques can be used in the intelligent document processing workflow. The unsupervised approach has a lower cost in the setup phase (no data labelling is required) but typically offers lower accuracy. Based on the algorithm used, the model can also provide the user with a Confidence Score to represent the model’s confidence with respect to its predictions. Depending on the technologies used, this step can either be an enabler for extraction or an optional step.
Information extraction
Extraction of information contained in the document is a fundamental step, necessary for the automation of processes related to document processing. In this step, the same methodologies can be used as those presented for document classification (text analysis, image analysis or both) with their relative advantages or disadvantages. One of the most time-consuming and costly operations is the extraction of key information and subsequent manual input.
This step aims to make this step automatic, transforming the unstructured data in the document into structured data that can be easily used by subsequent steps or underlying processes. Many types of information such as tables, images and signatures can be extracted in this step.
Validation of results
In the majority of cases, IDP solutions provide a scoring or confidence mechanism useful for reviewing data identified as potentially erroneous. Thus, to ensure data accuracy and integrity, IDP platforms utilize human review, external databases and pre-configured vocabularies to validate data extracted from documents. This process not only ensures data quality, but incorrectly processed data can be collected by enabling continuous learning of the system (Human in the loop & Continuous learning).
Enrichment of results
Another important step, before making the data usable, is the enrichment of the extracted data. Typically, external databases or services are exploited to add information to that extracted from the document in order to have more detailed and higher quality data. An example would be the look-up on an external system, starting with the name of a company, to check the health status of a company.
Integration
A final relevant aspect involves integration with the systems from which the data originate and with the systems that are to use them. IDP systems often offer very simple modes of no-code integration, implemented directly via the platform or with the help of RPA tools. This phase is essential to make the integration process easy and to have an end-to-end solution that interfaces with the adopted business tools.
Benefits
Intelligent Document Processing solutions enable companies to achieve several benefits:
- Direct cost savings
Leveraging scalable, high-performance architectures reduces time and costs, dramatically lowering the effort to process large volumes of data; - Reduction of manual processing
Artificial Intelligence minimizes the need for manual intervention to process documents; - Best data quality
Continuous learning and validation improves data quality, drastically reducing errors; - Start processing data quickly
IDP solutions are easily integrated via RPA mechanisms, often 5-10 times faster and easier to integrate than other approaches; - Process any document
Artificial Intelligence makes it possible to manage all kinds of structured, semi-structured and unstructured documents; - Enterprise-level automation
Thanks to the simplicity with which the IDP integrates with other areas of the company, a fully integrated RPA solution is easily reali<ed without the need for costly upgrades; - Improved productivity
IDP helps organizations increase productivity and reduce time spent on repetitive tasks, improving the quality of the working environment. - Business user friendly
With per-build use cases to choose from, launching and integrating more common use cases is quicker and easier;
myBiros and benefits
myBiros is a high-performance, user-friendly and versatile Intelligent Document Processing solution that enables automatic document processing. Core functionalities are information extraction and automatic document classification.
All this is offered via a prebuilt set of ready-to-use APIs with pre-trained templates for the most common use cases and the possibility to re-train the entire pipeline (both OCR engine and document interpretation system) for custom cases. It is very easy to integrate myBiros into any application through the use of APIs and easy interaction with RPA systems.
By exploiting advanced deep learning techniques that analyze multimodal features, it is possible to process all document types in a single solution. The system uses pre-trained models and data-augmentation techniques and can therefore be trained with a small volume of data, allowing processes involving a small volume of documents to be automated.
This solution includes a scoring mechanism: the system reduces false positives by enabling the possibility of reviewing low confidence data while minimising errors. Interaction with a human user enables the system’s errors to be corrected while continuing to train it so that it does not repeat past mistakes (Human in the loop and continuous learning). Finally, the high scalability of the cloud-based architecture makes it possible to process highly variable volumes of documents without having to allocate expensive resources in advance.
The features mentioned so far allow myBiros to perform optimally in the processing of bills. Succeeding effectively and quickly in identifying all relevant information. If you are curious about how myBiros works in order to simplify bill processing, contact us and try our demo. We are ready to help you!