Scraping data with Tabula

Much data is published in the PDF format. PDFs are highly versatile filetypes, and can contain text and images. Unlike, say, Word documents, copy and pasting text and numbers from a PDF document rarely works. When a scanned document is shared as a PDF, copy and paste won't work at all.

Here is a simple example from the Department of Labour - you can download the original file here.

This PDF contains a table, with the bid number, tender value and winning organisation. But how do we copy that into a spreadsheet when the document is actually just an image with no "text"?

Using Tabula

The answer is that we need an application such as Tabula, which is specifically designed to extract tables from PDFs

You can download Tabula here. You'll find installation instructions on the same site. Once you have installed it, Tabula doesn't run like a normal desktop application, it is accessed through your web browser (usually by browsing to http://127.0.0.1:8080)

Download the PDF file we showed you above. Once you have Tabula running, open it using the Browse button in this view.

Once you've imported the file, click on Extract Data. Your screen should look like this.

Tabula is very easy to use. Simply use your mouse to drag a selection rectangle around the table you want to extract.

And then click Preview & Export Extracted Data. Your screen should look like this.

In this instance, you can see that the table hasn't extracted cleanly. The text is split over rows that don't appear in the original, making it hard to analyse. You can change the way Tabula performs its data extracion by clicking on the Stream/Lattice buttons

.

This example use a very simple (although somewhat poor quality) PDF, but Tabula can extract large tables too. It can even extract tables that are split over multiple pages of a PDF.

The only thing it cannot do is Optical Character Recognition (OCR). OCR is a technique by which computers can "read" images and look for letters and numbers. It may work with some scanned documents, but not all. If not, you will likely need to use paid for software, such as Adobe Acrobat or an application based on the open source Tesseract libraries.

Try it with this National Senior Certificate (Matric) report from South Africa's Department of Basic Education. It's a 222 page document, but almost all the important data is in identical tables from page 13 on. In this case, you can extract an awful lot of data at once, but you may well still need to clean it a little once it has been imported into your spreadsheet app.