Scenario
We recently had an Insurance company come to us with a common business problem . They need to process 100,000+ forms/month which are received in various formats, structured and unstructured and at enterprise scale in order to search, validate and drive intelligent automation of business processes. Requirement is an automated process of extracting business critical data while maintaining the capability for human intervention to ensure data extraction quality with the core ROI to reduce the amount of time the user spends manually processing documents and searching for relevant content..
Solution
Implement an enterprise scale Knowledge Mining solution. What does this look like technically? Azure Cognitive Search integrated with AI Services such as Form Recognizer and other Cognitive Services. An ASP.Net MVC web application to serve as the front-end for searching forms and changing data scraped from forms. Maintain large amount of model-form relationships so various document types can be stored together.
Architecture Diagram
Setup Azure
You will need to do some setup in Azure before we begin. Create the following resources in your Azure Portal:
- Search Service
- Storage Account (Blob Container for storing your documents)
- App Service to house your Azure Function
- Standard Cognitive Services
- Form Recognizer Cognitive Service
Step 1: Image Document Classification
Utilizing the Computer Vision API from cognitive services we can automatically detect a particular form and give it a unique classification ID. By doing this forms that come into the pipeline can be processed and assigned ID’s to be matched with the OCR model in the next step. See below 2 examples of documents passed to the Computer Vision API where it recognizes its pre-designated type and gives it a probability of match.
Step 2:
Utilize the OCR Form Tool located at https://github.com/microsoft/OCR-Form-Tools/. The labeling tool is used to create a model against a single form. This would match a single document classification from Step 1. The model ID returned upon creation would be paired with a classification ID from Step 1 and stored within a SQL database table to be looked up later by an Azure Function.
After you train your model make sure that you check to make sure that it has a status of ready. You can use the following REST API call. At the time of this writing Form Recognizer has two active API’s. Make sure to use v2.0.
Step 3: Create the Azure Function
Close the repository located at https://github.com/Azure-Samples/azure-search-power-skills. We’ll be using the Vision Project. The code here needs some modification as it was implemented for Forms Recognizer API v1.0. Open AnalyzeForm.cs and replace the AnalyzeForm class with the below code.
Step 4: Create the Skillsets
Cognitive Services Skillset: This will be used by the indexer to populate the index with the following data:
- Key Phrases
- Locations
- Persons
- Content (This includes all scraped text as well as an image of your document for preview rendering)
- Metadata_Storage_Path (The url of your document)
- Metadata_Storage_Name (The name of your document)
- Search Score (Used to rank likelihood of a match to a particular search. Defaulted to 1 (100%), this score may be manipulated manually or by other services to rank search results.)
Form Recognizer Skillset: This will be used by the indexer to populate the custom fields you define in your index that match the fields from your OCR model.
Step 4: Create Search Index
Create the index with both standard OOB index fields and custom fields.
Step 5: Create Search indexers
Create Form Recognizer Indexer. Indexer’s run upon creation. Make sure to go to the Azure Portal to see if your documents are getting indexed.
Create Cognitive Services Indexer. Indexer’s run upon creation. Make sure to go to the Azure Portal to see if your documents are getting indexed.
Step 6: Setup your Web App
To get a jump start on your project we’ve modified the Knowledge Mining Solution Accelerator located at https://docs.microsoft.com/en-us/samples/azure-samples/azure-search-knowledge-mining/azure-search-knowledge-mining/. The web app created by Microsoft provides a working template utilizing Cognitive Services. What we have done is added the capability to both show the data that is mined with Forms Analyzer. With any data mining it makes sense that users may need to modify what AI predicts is the correct data or because information has been updated after the document has been created.
Clicking on our Sample forms provides the basis for our UI.
The data in our input boxes here comes from Forms Analyzer pulling the data off the pdf. We’ve added the capability to click on a tag (circled in red) which will display the data pulled from Cognitive Services. This way you can verify the data in context. This matches the data from Forms Analyzer with the data from Cognitive Services.
In order to accomplish this, you must add a Controller in your MVC app with supporting classes. Here is the code you will need to implement within your web application.
Summary
In conclusion, the integration of various AI services for Knowledge Mining has the potential to become a game changer for companies with a large ROI. The combination of the automated functionality for gathering data with human intervention to make modifications presents a robust solution to maximize the accuracy of data with the speed of full automation. After this data is obtained and processed there is an endless set of possibilities to report on this data with the Knowledge Store (Power BI), BPA and Logic Apps to report on data and continue LOB applications.
Contact us to learn more!
