get_drawings() can't detect line

[HeroadZ]

Please provide all mandatory information!

Describe the bug (mandatory)

get_drawings() can't detect line

To Reproduce (mandatory)

Here is the code you written in the past issues for recognizing tables by the lines. But after I update the version of Pymupdf, I can't get lines but all rects.

def get_table_location(page: fitz.Page) -> [fitz.Rect]:
    """
    Get the location of tables in page
    by finding horizontal lines with same length

    Parameters
    ----------
    page: page object of pdf

    Returns
    -------
    table_rects: rectangles that contain tables
    """

    # make a list of horizontal lines
    # each line is represented by y and length
    hor_lines = []
    paths = page.getDrawings()
    pprint(paths)
    for p in paths:
        for item in p["items"]:
            if item[0] == "l":  # this is a line item
                p1 = item[1]  # start point
                p2 = item[2]  # stop point
                if p1.y == p2.y:  # line horizontal?
                    hor_lines.append((p1.y, p2.x - p1.x))  # potential table delimiter

    # find whether table exists by number of lines with same length > 3
    table_rects = []
    # sort the list for ensuring the correct group by same keys
    hor_lines.sort(key=lambda x: x[1])
    # getting the top-left point and bottom-right point of table
    for k, g in groupby(hor_lines, key=lambda x: x[1]):
        g = list(g)
        if len(g) >= 3:  # number of lines of table will always >= 3
            g.sort(key=lambda x: x[0])  # sort by y value
            top_left = fitz.Point(0, g[0][0])
            bottom_right = fitz.Point(page.rect.width, g[-1][0])
            table_rects.append(fitz.Rect(top_left, bottom_right))

    return table_rects

this is the sample file.
https://drive.google.com/file/d/1ww1ZiBdQLWTHPmNbadTWXA_LU-rUW2tt/view?usp=sharing

Expected behavior (optional)

Detect lines and detect tables

Your configuration (mandatory)

• Operating system, windows 10
• Python version 3.8.5
• PyMuPDF version 1.18.9

[JorjMcKie]

Please tell me which was the PyMuPDF version, where this file was processed as you expected it.
I tried 1.18.1, 1.18.9 and 1.18.5: no difference.

[JorjMcKie]

In addition, looking at the second page for example:
This page has no lines - only rectangles.
And (apparently) only filled rectangles with no border line. Did not check them all however.

[JorjMcKie]

If I draw the rectangles in red found on second page, this picture comes out:

So what you think are lines, really are rectangles with a very small height (0.48) less than half a point.

[HeroadZ]

Unbelievable! They are rectangles.
Could you please check https://drive.google.com/file/d/1Pp679QHgEXYuYubN8PCCDvB4IXsmzlw_/view?usp=sharing
I thought they were all lines.
And actually, even now I can't tell the difference between them. lines and rects with small height .

[JorjMcKie]

For reasons which remain his secret, the PDF creator decided to draw thin horizontal bars, instead of lines.
You can easily find out yourself:

• If a path (an item of the list page.get_drawings()) is a rectangle, then there is only one item in path["items"], which then looks like ("re", fitz.Rect(...)).
• In contrast, a path can have multiple lines and curves in its path["items"].
• For all paths, there is the key/value path["rect"] which is the rectangle envelopping the complete drawing, which this path represents.
• You can check the rectangle height as usual: path["rect"].height - it is a fitz.Rect.
• So my suggestion would be to treat rectangles like lines, if they obey certain criteria (please confirm for your use case), e.g.
  • path["rect"].height < some threshold, if path["item"][0][0] == "re".
  • no border color
  • but fill color

[JorjMcKie]

As I wrote: the criteria may be too strict, a border color may also be ok, as well as a missing fill color.
Your "line" start and end points in the "rect" case could be fitz.Rect.bl and fitz.Rect.br respectively - or the tl / tr rect corners.

[HeroadZ]

Thank you very much!
I try to convert these rectangles into lines. And I found that their length may not be the same.
For example, left is rect.y0, right is the length of rect.

[(351.40997314453125, 424.74597930908203),
 (367.4899597167969, 424.74597930908203),
 (383.6899719238281, 424.74597930908203),
 (399.76995849609375, 424.74597930908203),
 (415.8499755859375, 425.4659729003906)]

The length of last line is a bit longer for unknown reasons. Maybe I should add some tricks.

[JorjMcKie]

yes - apply some rounding or whatever.
The rect width is given by rect.x1 - rect.x0. If you round this to integer, you would be getting 425.
Who / what created that table?

[HeroadZ]

I created that table by Word 2101 and exported it as a pdf for test.
Round could only support for a range(-0.5, 0.5). I made it with // and *
For example,

>>> 424.74597930908203//2*2
424.0
>>> 425.4659729003906//2*2
424.0

With this, a larger range is allowed.

[JorjMcKie]

ahhh - now see.
I also have a PDF convertede from word. And the same phenomenon: lines in Word are implemented as slim rectangles in PDF.