Scraping data with Tabula

Many tender notices or award notifications are published in the PDF format. PDFs are highly versatile filetypes, and can contain text and images. Unlike, say, Word documents, copy and pasting text and numbers from a PDF document rarely works. When a scanned document is shared as a PDF, copy and paste won't work at all.

Here is an example from the Department of Labour - you can download the original file here.

This PDF contains a table, with the bid number, tender value and winning organisation. But how do we copy that into a spreadsheet when the document is actually just an image with no "text"?

Using Tabula and OCR

The answer is that we need an application which can apply Optical Character Recognition (OCR). OCR is a technique by which computers can "read" images and look for letters and numbers. In this case, we are going to use Tabula, an application which is specifically designed to extract tables from PDFs using OCR.

You can download Tabula here. You'll find installation instructions on the same site. Once you have installed it, Tabula doesn't run like a normal desktop application, it is accessed through your web browser (usually by browsing to http://127.0.0.1:8080)

Download the PDF file we showed you above. Once you have Tabula running, open it using the Browse button in this view.

Once you've imported the file, click on Extract Data. Your screen should look like this.

Tabula is very easy to use. Simply use your mouse to drag a selection rectangle around the table you want to extract.

And then click Preview & Export Extracted Data. Your screen should look like this.

In this instance, you can see that the table hasn't extracted cleanly. The text is split over rows that don't appear in the original, making it hard to analyse. You can change the way Tabula performs its OCR by clicking on the Stream/Lattice buttons.

Changing the method to Lattice gives us exactly what we want, an error free table in the preview view. From here, we can click Export and save the extracted table as a CSV file.

This example use a very simple (although somewhat poor quality) PDF, but Tabula can extract large tables too. It can even extract tables that are split over multiple pages of a PDF.