cli: how to extract a table from a page?

[aborruso]

Hi,
first of all thank you for this great tool.

I have seen that via python I can use find_tables. Is there a way to do something similar via cli and extract a table from a PDF?

Thank you

[JorjMcKie]

You mean py -m pymupdf gettext ...?
No, not yet.

You could build a little script and invoke it via cli:

import pymupdf4llm
import sys
import pathlib

filename=sys.argv[1]
textfile=filename.replace(".pdf", ".md")

data = pymupdf4llm.to_markdown(filename)

pathlib.Path(textfile).write_bytes(data.encode())

This will convert the complete file to Markdown text. Any table therein, too.

[aborruso]

Thank you very much @JorjMcKie

I have tried using this pdf file.pdf.

I have this

And I obtain this (below).
What I would like to understand how to solve is mainly the mapping of rows. In the first table of this pdf I have about 10 rows, while in output I have 3 (with header).

How can I optimize the script?

Thank you

[JorjMcKie]

The problem here is that the gridlines are dotted using such a small length that they fall below an inbuilt limit - and are thus ignored.
We would have to change the table detector to be more forgiving ...

[aborruso]

@JorjMcKie Is there no stream method? it works well with tables that do not have grid lines. It relies on text alignment to identify columns.

[JorjMcKie]

Unpack & Copy this into your PyMuPDF installation folder :
table.zip

Then try again.

[aborruso]

Hi @JorjMcKie you are very kind. Thank you

I have copied in:

• /home/aborruso/.local/lib/python3.11/site-packages/fitz
• /home/aborruso/.local/lib/python3.11/site-packages/fitz
• /home/aborruso/.local/bin/pymupdf

I have used this code

import pymupdf4llm
import sys
import pathlib

filename="file.pdf"
textfile=filename.replace(".pdf", ".md")

data = pymupdf4llm.to_markdown(filename)

pathlib.Path(textfile).write_bytes(data.encode())

And I have the same wrong result: https://gist.github.com/aborruso/ba0f3d3431b9f75de95dfeecf40a9e65

But I probably didn't understand something.

[JorjMcKie]

This script:

import pymupdf4llm
data=pymupdf4llm.to_markdown("file.pdf", page_chunks=True)

Delivers this MD in data[0]["text"]:
file.md
I think this is close to perfect ...

[aborruso]

It works, it's really impressive thank you!!!

Now I need to work on how to export a markdown (CSV would be even better) for each table.
Because I will have to work on a multipage pdf, and then do filtering, transformation and cleaning operations on the tables.

Currently it is all in one variable, and the separator between tables is \n\n.

"text": "|REGIONE|CITTA' METROPOLITANA|COMUNE|NUMERO COMUNI|CAPOLUOGO (CAP)|15.000 abitanti (SUP)|INFERIORI  15.000 abitanti (INF)|MOTIVI DIVERSI DA SCADENZA NATURALE (*)|RINNOVO PER SCADENZA NATURALE (#)|al 31/12/2021      D.P.R. 20/01/2023|Rilevazione al 31/12/2023|Rilevazione al 31/12/2023|\n|---|---|---|---|---|---|---|---|---|---|---|---|\n|||CORVARA|1|||INF|||206|1|510|\n|||FARINDOLA|1|||INF|||1.357|2|1.578|\n|||LETTOMANOPPELLO|1|||INF|||2.713|3|2.710|\n|||MONTEBELLO DI BERTONA|1|||INF|||883|2|1.139|\n|||MONTESILVANO|1||SUP||||53.402|52|44.600|\n|||MOSCUFO|1|||INF|||3.092|3|2.851|\n|||ROSCIANO|1|||INF|||4.038|6|3.697|\n|||SALLE|1|||INF|||270|1|698|\n|||VICOLI|1|||INF|||381|1|495|\n||Totale||17|1|2|15|0|0|209.303|267|185.620|\n\n|TERAMO|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|\n|---|---|---|---|---|---|---|---|---|---|---|\n||ANCARANO|1|||INF|||1.811|2|1.620|\n||ARSITA|1|||INF|||756|1|874|\n||ATRI|1|||INF|*||10.064|14|9.986|\n||CAMPLI|1|||INF|||6.630|11|6.568|\n||CANZANO|1|||INF|||1.794|3|1.653|\n||CASTIGLIONE MESSER RAIMONDO|1|||INF|||2.052|4|1.917|\n||CELLINO ATTANASIO|1|||INF|||2.274|3|2.531|\n||CERMIGNANO|1|||INF|||1.459|3|1.839|\n||COLLEDARA|1|||INF|||2.097|4|2.281|\n||CORROPOLI|1|||INF|||5.108|5|4.467|\n||FANO ADRIANO|1|||INF|||257|2|277|\n||GIULIANOVA|1||SUP||||23.442|23|21.806|\n||MONTEFINO|1|||INF|||967|2|915|\n||MORRO D'ORO|1|||INF|||3.560|3|3.232|\n||MOSCIANO SANT'ANGELO|1|||INF|||9.088|9|8.365|\n||PENNA SANT'ANDREA|1|||INF|||1.635|2|1.778|\n||PINETO|1|||INF||#|14.538|11|13.363|\n||ROCCA SANTA MARIA|1|||INF|||477|1|537|\n||SANT'EGIDIO ALLA VIBRATA|1|||INF|||9.804|7|8.244|\n||SANT'OMERO|1|||INF|||5.112|7|4.767|\n||TORANO NUOVO|1|||INF|||1.490|2|1.280|\n||TORRICELLA SICURA|1|||INF|||2.460|4|2.658|\n||TOSSICIA|1|||INF||#|1.258|3|1.388|\n|Totale||23|0|1|22|1|2|108.133|126|102.346|\n\n|ABRUZZO|98|1|3|95|2|2|424.264|537|405.785|\n|---|---|---|---|---|---|---|---|---|---|\n\n\n-----\n\n"

Really great job, thank you

[JorjMcKie]

Well, you are not forced to use pymupdf4llm. You could simply do table extraction with pymupdf - the llm package itself is based on it.
So for each page, you can extract the tables, and convert each one to a pandas DataFrame.

DataFrames support around 20 different output formats - among them are (also) markdown, but in addition CSV, Excel, ...

DataFrame tables can be joined or manipulated in all sorts of ways.

We have written our own markdown output for tables, to have it independent from pandas (respectively tabulate). This makes it easily usable in the LLM / RAG field (no external dependency).
It also produces smaller output data volume - another advantage for LLM / RAG usage.
Tabulate produces nice-looking output instead - it just has a different use case target.

So, here is a code skeleton for processing tables directly in pymupdf:

for page in doc:
    for table in page.find_tables(...):
        md_text = table.to_markdown()  # pymupdf's own MD output
        df = table.to_pandas()  # make a pandas DataFrame from the table
        df.to_csv(...)  # CSV output
        df.to_markdown() # ... requires package tabulate
        df.to_excel(...)  # Excel
        dataframe_list.append(df)  # if you need to store it for later manipulation

[aborruso]

A really great result.

The input file: input.pdf

The output: data.csv

After extracting the tables, I had to apply several cleaning rules, but I could never have done it without this great tool. Yay fitz and PyMuPDF

Thank you very much

[JorjMcKie]

Very pleased to read these compliments - thank you!

I am just preparing the official fix. For this purpose, I am converting this Discussions post back to an issue for better reference in the future change log.
Please simply ignore this bookkeeping activity.

[aborruso]

Please simply ignore this bookkeeping activity.

I will be proud of this

[thangarajdeivasikamani]

How to add page number in the MD file as per PDF Page?

[JorjMcKie]

@thangarajdeivasikamani - What do you mean?

[thangarajdeivasikamani]

Example my pdf has 10 pages.

While converting from PDF to MD format. Is there any way configure page1, page 2, page 3 like that at header of MD file or each page number?.
This will really helpful identify source page while LLM answer the question.

[JorjMcKie]

I see what you mean now.
Your question is off topic in the context of this discussion thread. The problem here was a missed table detection, not MD text output of tables or pages.

You should ask this on the Discussions of repository pymupdf4llm. In fact, it has been answered there already: simply use option page_chunks=True with method pymupdf4llm.to_markdown().

[Ishwari-31]

I am using the same code to extract tables from a pdf "import pymupdf4llm
md_text = pymupdf4llm.to_markdown("attached_table_links.pdf")
print(md_text)" But the thing is my table content has hyperlinks(Embedded links) inside it and in output i want those hyperlinks(Embedded links) which will be clickable as well. It will be great if you could answer this question
attached_table_links.pdf

[JorjMcKie]

Please enter this as an enhancement request in the issues tab.
Incorporating hyperlink information and more (e.g. image references) as part of table cell extraction would entail supporting this kind of thing as part of the table module.
Currently, only table cell text is returned by the table module.