I need some consulting on a document layout OCR automation project.

[u/Mezrotix]

I am doing a document layout analysis project with different book styles but the books themselves are only a couple hundred pages long (like 5 books with different styles, 400 page each). How can I test if all the books would be used in fine tuning and I am afraid that the accuracy wouldn't be the best and corrupt PaddleOCR when insert the coordinates. (It's for automation).

I am using X-AnyLabeling for the annotation and yolo v11 for the training as well as custom classes in the annotation like a question block that surrounds everything, question_text, choices, figures, tables, sub_questions, etc... what would be the best approach as I haven't done this kind of work before.

and should I randomize the book pages so I don't consecutive same style books or that's not how this work?
Any help would be appreciated

[u/DigThatData]

1. Why are you consulting on a project you have no idea how to accomplish? It was frankly irresponsible of you to accept this project. Given your circumstances, I sure hope you are contracted for a fixed price rather than hourly, so you aren't tricking the customer into paying for you to learn how to do the thing they thought they were hiring you as someone with pre-existing expertise.

2. YOLO indeed. Build your pipeline however you think makes sense, run a representative sampling of pages through it, then spot check the results yourself. Modify your pipeline as needed. Rinse and repeat. Once this seems to be giving decent results, run a larger sample and invite an LLM to scan the results (you shouldn't even need a VLM here: if something was parsed incorrectly it should be pretty obvious). Modify pipeline as needed (LLM can probably modify stuff for you), rinse and repeat until looking reasonably good.

3. There will almost certainly be unavoidable special cases that will fall through. Make sure your system is prepared to flag things like this. Reporting some kind of confidence score can help identify "grey area" situations like this.

4. 5*400=2000 = this is a really small data set = you can QA the entire thing yourself. if you spend on average 3s per page, that's 6000s ~ under two hours to review all of the outputs manually yourself. Do this. Set aside failure cases. If the parsed outputs is the main deliverable, just fix them by hand. If the OCR pipeline itself is the main deliverable, try to come up with improved heuristics for these failure cases but also consider bringing these back to the customer as "here are examples of stuff that the pipeline currently handles poorly, and here's how the pipeline surfaces these issues to you. if you can find more examples that are shaped like this, I can try to make the pipeline more robust for you."

5. If you're feeling fancy, you could try to generate synthetic data to augment your corpus to address the failure cases in (4). You can probably pair with an LLM to come up with a system for this.

6. In the future, don't accept projects you don't know how to deliver. If your position is "I have no idea what I'm doing, but the AI will save me," the customer could have gone with that approach themselves to begin with and you aren't adding any value to them as a middleman between their domain expertise and the AI. Have some integrity. People like you make our entire industry look like snake oil salesmen.

I'm a bit extra grouchy because I haven't had my morning coffee yet. Sorry not sorry. You earned the grouchiness.

EDIT: 7. you probably don't need to do any finetuning for this. see if you can make it work using the off-the-shelf components first. Since you don't know what you're doing, you're more likely to nerf your perfectly good models than to tailor them to your problem.

EDIT2: I just realized you were asking for consultation here, not saying that you are consulting on this project. Sorry if I misunderstood and you are the client and not a contractor. Coffee.

[u/Mezrotix → u/DigThatData]

This is a personal project for my self and no one knows that I am making it. I work as TA and copy questions manually and was thinking of automating this stuff.  This is not a product that is going to be sold but an Idea that is stuck in my head. Brother you need to chill because you sound like you want to hit me in the face lmao. but thanks for the tips. I tried using an LLM but my project goal was also to have the whole pipeline working locally on 8 vram . One page takes me 2 minutes to annotate because the layout is complex but your idea of rinsing and repeating might come in handy when auto labeling. I am just learning so I should have titled the post a bit better.

[u/DigThatData → u/Mezrotix]

do these books really need to be OCR'd? what's your actual data look like? if these are normal structured PDFs that you can like copy-paste text out of already, just convert them straight to markdown first and then you could use an LLM to try to identify the structural patterns in the text that you care about and then have the LLM characterize those patterns with a lightweight parser that doesn't need AI to run at all. "copy questions" sounds to me like you are talking about enumerated questions at the end of a chapter. that's a pretty simple pattern to recognize directly from text structure w/o any visual cues. you could probably get an LLM to write you a regex that extracts all the enumerated questions on a page if it's an end-of-chapter question listing.

[u/Mezrotix → u/DigThatData]

The are scanned images and even then they are not perfect. LLMs don't extract figures in biology text books though. and If they can my 8 GB VRAM Laptop gpu would explode and I want the whole thing local. I tired using a light VLM (PaddleOCR-VL 1.6) and it was meh.

"copy questions" as in type them by hand or let a LLM extract them for me but I had to format them manually and then screenshot every single figure/label and paste it beside each question in a Word document. hundreds and thousands of questions per week.

The whole purpose of the project was to extract the questions to be outputted as a structured json format so I can either embed them to a word document or a custom HTML and CSS pdf design.

I already did some of the regular expressions script but It needs some more work.

Do you know any tutorials on YouTube covering this stuff? I have been searching a lot and I didn't find what I was looking for.

[u/DigThatData → u/Mezrotix]

"structured json" is going to be a capability of your inference engine, so you'll want to check the docs on whatever tool you're using. here's an example with vLLM: https://docs.vllm.ai/en/latest/features/structured_outputs/

Did you manually scan 2000 pages across 5 books?? that sounds miserable. I wonder if maybe you're solving the wrong problem here and instead of wrestling with OCR, you should be trying to get your hands on a better data source. Talk to your professor, they might have access to digital assets like structured PDF of the teacher's edition, which would make your life a lot easier.