Why convert PDFs to Markdown before using them with an LLM?

Markdown keeps headings, lists, tables and links while removing layout noise such as page numbers, running headers and styling. The same content takes fewer tokens, and headings give natural places to split documents into chunks for retrieval.

Can scanned PDFs be converted to Markdown?

Only with OCR, because a scanned PDF contains pictures of text rather than text. OCR output needs checking, especially for numbers, tables and names. Text-based PDFs, where you can select words, convert directly.

Should I convert the PDF or the original Word file?

Convert the original Word (.docx) file when you have it. Word records real headings, lists and table cells, while a PDF only records where text sits on the page, so the structure has to be guessed.

What chunk size should I use for RAG?

There is no single best size. Split at headings first, split long sections further, and test retrieval with real user questions to choose a size that works for your documents and embedding model.

How to Convert PDF and Word to Markdown for RAG and AI

If you want a retrieval-augmented generation (RAG) system or an AI assistant to use your PDFs and Word documents well, convert them to Markdown before you index them. Markdown keeps the structure a model and a retriever depend on (headings, lists, tables and links) and drops the layout, styling and repeated page furniture that only add noise. The steps are short: check what kind of file you have, convert it, check the structure, clean it up, then split it into chunks along its headings.

Why Markdown is a good format for RAG

Retrieval quality depends heavily on what goes into the index. Three properties make Markdown a practical intermediate format:

Structure survives. A heading is still a heading and a table is still a table. Plain-text extraction from a PDF flattens all of that into one stream of lines.
Less noise per token. HTML carries tags, attributes, scripts and styles; PDFs carry page numbers and running headers on every page. Markdown carries almost nothing but content.
Natural chunk boundaries. Headings give you sensible places to split a document, and the heading path (for example "Billing > Refunds > Eligibility") makes useful metadata to store with each chunk.

To see the size difference on a real page, we ran one of our own posts, RAG vs. fine-tuning, through our HTML to Markdown converter. The page's HTML file is 35,126 bytes, most of it CSS, navigation and markup. The article as Markdown is 4,626 characters, about 13 percent of the original. Using the common rule of thumb of about four characters per token for English, that is roughly 1,200 tokens instead of nearly 9,000 for the same content. Your ratio will differ from page to page, but the direction is consistent.

Step 1: check what kind of file you have

Before converting anything, find out whether the text is really text:

Text-based PDF: you can select and copy words in a PDF viewer. These convert directly.
Scanned PDF: each page is a picture, so selecting text does nothing. There is no text to extract without OCR (optical character recognition), which adds its own errors to check.
Word document (.docx): if you have the original .docx, convert that instead of a PDF exported from it. Word files record real headings, lists and table cells; a PDF only records where text sits on a page.

Step 2: convert the file

Our free File to Markdown converter handles PDF, DOCX, HTML, XLSX, PPTX and CSV in the browser, so documents never leave your machine. There are also dedicated pages for the most common formats: PDF to Markdown, Word to Markdown, Excel to Markdown and PowerPoint to Markdown.

For large volumes, run the same kind of conversion in your ingestion pipeline with a library. The principle does not change: extract structure where the format records it, and rebuild it carefully where the format does not.

Step 3: check the structure, not just the text

A conversion can contain every word and still be wrong for retrieval. Open the Markdown and look at the outline:

Heading levels. PDFs have no headings, so converters infer them from font size. A caption in a large font can become a heading, and a small-font section title can become a paragraph. Fix the hierarchy by hand where it matters.
Tables. PDFs do not record table cells, so table text often arrives as loose lines. If a table carries facts people will ask about (prices, limits, dates), rebuild it as a Markdown table or convert the source spreadsheet instead.
Reading order. Two-column layouts can interleave lines from both columns. Read a page or two of the output to catch it.
Page furniture. Running headers, footers and page numbers repeated on every page should be gone; they otherwise land in the middle of sentences and chunks.

For Word files, headings come from Word's heading styles. If the author made headings by enlarging and bolding text instead of using Heading 1 or Heading 2, the converter cannot tell them apart from body text. Applying the styles in Word and converting again is usually faster than fixing the Markdown.

Step 4: clean up what the model should not see

Remove content that will only compete with the real answers at retrieval time: tables of contents, cover pages, legal boilerplate repeated in every document, and changelog sections that describe old behaviour. Keep anything a user could reasonably ask about, including notes and exceptions. A short YAML front matter block with the title, source file and date (our converter can add one) makes it easy to cite the source and to spot stale documents later.

Step 5: chunk along the headings

Split each document at its headings first, then split long sections further. Store the heading path with each chunk, so a passage about "eligibility" still carries the fact that it is about refunds. There is no universal best chunk size: the right size depends on your documents, your embedding model and the questions people ask, so test retrieval on a set of real questions before settling on one. We cover the failure modes in why RAG gives wrong answers from the right documents.

A quick quality check

Pick ten questions your users actually ask, find the passage that answers each one in the converted Markdown, and check that it reads correctly on its own. If a human cannot answer from that passage, the retriever and the model will struggle too.

How we approach this

When we build retrieval systems, document conversion is one of the first things we inspect, because problems introduced here cannot be fixed by a better model later. We convert from the most structured source available, check heading hierarchy and tables on a sample of every document type, and keep the conversion step in the pipeline rather than as a one-off, so updated documents go through the same process.

How to convert PDF and Word files to Markdown for RAG and AI