Guten Tag! 👋 

Many greetings from Munich, Germany.

Last week we extracted data from one PDF with {ellmer}. But often we don’t want to get data out of one single PDF. Instead, we want to use multiple PDFs. Today, let’s combine {ellmer}, {fs}, {purrr} and {pdftools} to do that in one smooth pipeline. 🥳 

Getting the PDF paths

As you know, the {fs} package allows you get paths to many files in a directory. For example, I have a couple of PDFs inside a pdfs/ directory:

As you can probably guess from these paths, these are the PDF penguin reports that we generated a while back. Back then, I used Quarto & Typst. But nowadays I just use Typst. In any case, that’s how one of the PDFs looks:

Get the text content from all PDFs

Next, we can get the text content from all PDFs. The magical {purrr} package makes it really easy to apply the pdftools::pdf_text() function iteratively. That’s perfect to iterate over the paths_pdf vector we’ve just created.

This looks pretty ugly. So let’s remove all extra white spaces with stringr::str_squish(). Once again, we can use {purrr} to iterate over all those texts.

Ahh that looks much better.

Do many LLM calls

Now, you might think that we can also use {purrr} to iterate over all these pages. And you’re right. We could. But that’s not the efficient way to do it.

You see, {ellmer} allows us to run data extractions in parallel. We just have to pass a list of prompts to parallel_chat_structured(). Here’s how that looks:

Sweet! With that we have looped over our entire content and extracted information from it. Even better, this was much faster than running 3 LLM calls one after the other. With parallel_chat_structured(), you will asynchronously sent a bunch of llm calls all at once.

Even better: This happens all in the same R process. So you didn’t even have to think about async stuff like the things that the {mirai} package allows you to do. And {ellmer} will even take care of sending LLM calls in bits and pieces. Let me demonstrate that with the progress bar that you’ve seen above. I’ll simply repeat a bunch of my data and send more requests to the LLM:

Have a look how the progress bar evolves. Row-by-row {ellmer} will tell you that:

• 15 LLM calls are queued, 0 have to be retried, 9 calls are waiting on a reply and 6 already finished

• 0 queued, 0 retries, 10 still waiting, 20 finished

• 0 queued, 0 retries, 1 still waiting, 29 finished

• All done

So as you can see we can throw a whole bunch of data at {ellmer} and it will process that all in parallel for us. Pretty sweet, isn’t it?

Alrighty, that was everything I wanted to cover this week. Hope you enjoyed it. Feel free to hit the reply button to tell me about how you liked this newsletter episode.

And don’t forget about the “Report Automation With Typst” course. Offer is still available until Sunday.

See you next week,
Albert

Whenever you’re ready, there are three I can help you:

1. Automate Your Data Reports: This course helps data analysts eliminate manual copy-paste reporting by automating PDF reports end-to-end, saving hours every cycle and preventing costly mistakes. (Using the lovely Typst language 😍)

2. Generate Insights in Minutes, not Hours: This comprehensive course teaches you to handle data faster, smarter, and more efficiently.

3. Bespoke Data Science Solutions: I’ve helped clients build their own data science solutions. Whether building custom web apps, PDF reports, AI automations or teaching workshops, I’ve got you covered. You can reach out to me via this form (or simply hit reply to this email)