Find out more with SciForce free checklist
The client works in the Financial Technology (FinTech) sector, focusing on digital invoicing and automated expense management for transportation services. They act as a middleman offering technical solutions to taxi services, helping them manage invoices more efficiently and keeping their financial records well-organized.
They help turn physical taxi receipts into a standardized digital format, dealing with various receipt types and photo qualities.
Client's key challenge is dealing with receipts in different formats from various taxi services and turning a poor photo of a receipt into a clear image to understand and extract the data from.
1. Variability of Data
Taxi receipts came in numerous formats with unique layouts, languages, and structures. In our case, we had to work with receipts in Swedish and English — a language scope the contract automation case shares, where Swedish-language agreements had to be parsed with clause-level precision despite equally inconsistent document structures.
2. Poor Photo Quality
Photos of receipts often have issues such as low resolution, poor lighting, blurriness, and visual artifacts like shadows and reflections, creating challenges for computer vision and OCR for financial documents. Additionally, most receipt photos are taken at an angle, making it harder for computer vision for document analysis to recognize the values.
3. Data Accuracy
Ensuring that the extracted data from receipts was accurate and reliable for tracking expenses, despite differences in receipt layouts, inconsistent field formats (such as dates and prices), language variations, low-quality photos.
4. Model Training Resources
Training large models like LLMs required significant computational resources, including high-performance GPUs, extensive memory, and long training times, making it costly and time-consuming.
5. Integration and AI Infrastructure Scalability
Integrating the processed receipt data into existing financial and management systems, and scaling the solution to handle large volumes of receipts, potentially reaching millions per month.
Dataset Сollection
We collected a dataset around 10.000 receipts that included a wide range of receipt formats in English and Swedish languages. It was used to train the model models to recognize and adapt to these variations, ensuring accurate processing.
Image Enhancement
Implemented advanced image enhancement techniques, including noise reduction and upsampling with GANs, to improve image quality before processing.
LLM for Structured Data Extraction
Used OCR technology like Tesseract and trained LLMs to improve text extraction and classification accuracy. Implemented post-processing steps to correct errors and maintain data integrity.
Machine Learning Model Integration
Employed LoRA-based training to focus on a small portion of the model's weights, optimizing resource usage while maintaining high accuracy.
Data Integration
Choose relational databases for structured data storage and seamless integration with existing systems, allowing for efficient data retrieval and manipulation.
Image Enhancement
Enhances the quality of receipt images by reducing noise and performing upsampling using Generative Adversarial Networks (GANs). This also includes geometric transformations to make the receipts easy to read, as if they were scanned.
AI for Invoice Management
Trains a model to accurately detect and delineate the boundaries of receipts, ensuring that only relevant information is captured. Utilizes a dataset with labeled check boundaries, trained using YOLO.
Text Extraction
Uses Optical Character Recognition (OCR) to extract text from the processed images within the detected boundaries, ensuring accurate text recognition.
Data Structuring
Organizes the extracted text into a structured format, categorizing elements such as order lists and VAT calculations.
Post-processing
Identifies and corrects errors in the extracted data. This includes editing the image if necessary and ensuring keywords correspond to the correct statements, handling the most frequent errors.
Feedback Loop
Implemented a feedback loop allowing users to report inaccuracies easily. This feedback was reviewed, added to the dataset, and used to retrain the model, continuously improving accuracy and handling variability.
1. Image Enhancement
Edited the source images to improve quality, including noise removal and upsampling using Generative Adversarial Networks (GANs). This step ensured that low-resolution images were enhanced to a higher resolution, making them suitable for further processing.
2. Classical Image Processing Techniques and Geometric Transformations
Applied geometric transformations to align and correct receipt images, making them look like scanned documents. Used model-based image enhancement (GANs/diffusion) only for edge cases to maintain cost-effectiveness.
1. Model Training
Collected a comprehensive dataset containing various receipt images with clearly labeled check boundaries. This dataset was used to train a YOLO model, designed to detect the edges and boundaries of receipts. The training process involved teaching the model to recognize different receipt formats and accurately identify their perimeters.
2. Detection
Deployed the trained YOLO model to automatically detect and delineate the boundaries of receipts in real-time. The model accurately localized the receipts within the images, drawing precise bounding boxes around them.
1. Text Extraction
We used Tesseract for Optical Character Recognition (OCR) to extract text from processed images within the accurately detected receipt boundaries. The receipt recognition AI models were configured to recognize various fonts and languages commonly found on receipts, ensuring high accuracy in text extraction.
For more complex tasks requiring maximum accuracy, it’s possible to use high-end OCR models via APIs, such as Microsoft Vision 3.0.
2. Text Processing
Converted the extracted text from taxi receipts into organized, structured data formats. This included categorizing details such as ride dates, pickup and drop-off locations, fare amounts, tip amounts, total charges, and VAT calculations. The field taxonomy here is relatively predictable — every receipt has a date, a fare, a total. The spec sheet extraction case shows what structuring looks like when the field taxonomy itself is unknown until the document is parsed.
1. LLM Training
Trained a Large Language Model (LLM) named Gemma 2B to enable LLM-based structured data extraction from documents, accurately categorizing data from taxi receipts into specific groups, such as ride details, fare breakdowns, and VAT information. Employed LoRA-based (Low-Rank Adaptation) training techniques to optimize the model by focusing on specific attention layers.
2. Dataset Preparation
Compiled a high-variance dataset consisting of taxi receipts from various services and regions. This dataset included a wide range of templates with variations in structures, ride details, date formats, currency symbols, and other receipt elements, to teach the model accurately deal with variability in real receipts.
Identified and corrected common errors in the extracted data from taxi receipts. This involved adjusting misrecognized text, such as correcting fare amounts, dates, and location names. Edited images to remove artifacts or distortions that could affect text accuracy.
- YOLO (You Only Look Once): Trained a model to accurately detect and delineate the boundaries of taxi receipts within images.
The solution removed the need for manual data entry, increasing the efficiency of AI receipt processing software by 80% and cutting the costs by 50%.
The end-to-end AI document automation for fintech platforms enabled the client to scale their operations effectively, handling millions of receipts per month.
The accuracy of data extraction improved by 80%, enhancing the reliability of AI in financial operations. This led to higher client satisfaction and trust, resulting in increased client retention and loyalty.
Quicker processing times led to faster reimbursements for taxi drivers and companies. This reduced the waiting period for payments, improved cash flow, and allowed for more efficient financial planning and operations. We’ve upgraded the taxi receipt management system for a FinTech client. Our AI solution made processing 80% faster and more accurate, even with tough receipt formats and bad image quality
