Larry Hudson

Putting my Strava activities on a single map

2024-08-23T00:00:00Z

I’m excited to share a small project I’ve been working on: a web app that takes your Strava workout activiites and puts them on a single map using Astro, the Strava API and the Mapbox SDK.

After moving to Paris in early May this year, I have been walking a lot, as a way to stay healthy and to explore Paris and learn about its geography. I’ve been walking around the 18th, 9th, 10th and 11th arrondissements in particular.

I thought it would be interesting to make a small web app that takes my Strava activities and renders them on a single map, as a way to visualise the areas that I have been exploring, and encourage myself to check out more.

Here’s a screenshot of my activities on a map:

Decoding polyline strings

When you retrieve your list of activities from the Strava API, it returns the map information as a long encoded string:

{ "map" : { "id" : "a1410355832", "polyline" : "ki{eFvqfiVqAWQIGEEKAYJgBVqDJ{BHa@jAkNJw@Pw@V{APs@^aABQAOEQGKoJ_FuJkFqAo@{A}@sH{D...", "resource_state" : 3, "summary_polyline" : "ki{eFvqfiVsBmA`Feh@qg@iX`B}JeCcCqGjIq~@kf@cM{KeHeX`@_GdGkSeBiXtB}YuEkPwFyDeAzAe@..." }, }

These encoded strings follow the Encoded Polyline Algorithm Format. I used the @mapbox/polyline npm library to turn these strings into coordinates that can be rendered on a map.

Aider’s AI pair programming

I’ve been using Aider a lot in the last couple of weeks. It’s an open-source AI coding assistant that lives in the terminal. I’ve been using it with Anthropic’s Claude 3.5 Sonnet model and I’ve been really impressed with the results. It’s been a big productivity boost to help me get stuff done.

Check out the GitHub repo

If you’re curious about how this project works and want to give it a try, check out the GitHub repository for more details and the source code.

GitHub: https://github.com/larryhudson/astro-strava-map

Giving Claude 3.5 Sonnet extra abilities with custom tools

2024-07-03T00:00:00Z

I’m excited to share a demo chatbot interface built with Next.js that allows you to chat with Claude 3.5 Sonnet (a competitor to ChatGPT and the current state of the art large language model) and create custom tools that extend its capabilities. By connecting Claude to a Weaviate vector database, the chatbot has the ability to save notes for future reference, and search for relevant information to be more helpful.

You can check out the project on GitHub.

As a side note, in the last couple of weeks I’ve been playing with Claude 3.5 Sonnet a lot, and I’m super impressed with its abilities. I’m having a lot of fun creating little web apps and tiny projects. It feels like you can make anything with it, as long as you can think it through piece by piece.

Project highlights

This demo project showcases several key features and technologies:

Anthropic API Integration:
Custom Tool Integration: This allows the AI to perform actions beyond its training data, such as searching through notes, fetching real-time information, or interacting with external systems.
Weaviate Vector Database Integration: Weaviate, a vector database enables semantic search capabilities
Dynamic Note Saving and Retrieval: The assistant can save and retrieve notes, creating a growing knowledge base.
Streaming Interface: The chatbot provides real-time, streaming responses This is implemented using server-sent events
Next.js Framework: Built on Next.js using the App Router model.

Tool use with Claude

In this project, I’ve added a few custom tools that Claude can choose to use within the chat interface.

Tools are defined with a name, description, and input schema in the API request.
Claude can decide when to use tools or be instructed to use specific tools.
The process involves Claude requesting tool use, your application executing the tool, and optionally sending results back to Claude.

For a deep dive into tool use with the Anthropic API, check out these resources:

Weaviate vector database integration

Semantic Search: Weaviate allows Claude to perform semantic searches rather than simple keyword matching.
Efficient Information Retrieval: The vector database enables quick and efficient retrieval of relevant information from a large corpus of data.
Dynamic Knowledge Base: new information can be stored in Weaviate, allowing the assistant to expand its knowledge over time.

Limitations and drawbacks of this approach

One limitation of this approach is cost. The more tools that you share with Claude, and the more output that these tools create, use up tokens, which makes your API usage more expensive. While the latest AI models have large context windows, the costs add up, and would be a concern if you were trying to make an app for a large user base.

If you were making a production app, you would need to pass the cost onto the user, or implement severe rate limits which would be annoying for the user.

However, as Ethan Mollick says, you should always treat the current AI model as the worst one you will ever use again, so abilities are going to keep getting better. And costs will go down. So it’s worth experimenting to push these models to get the most out of them.

Running the project locally

To get this project up and running, you can find detailed instructions in the GitHub repository README.

Ideas for future improvements

Scheduled workflows - I like the idea of being able to create specific tasks that the assistant would perform periodically - eg. monitoring websites and generating summaries, or keeping an eye on to do lists and creating plans for what needs to get done. I think doing automation in this way, using natural language, has a lot of potential.
Text-to-speech integration - I think it would be great to be able to talk to the assistant using speech, and to listen to the assistant using a ‘read aloud’ feature. That could reduce the amount of time the user needs to type into a text box. I like the ChatGPT app for this reason.
Claude’s Artifacts feature - I’m a big fan of the Claude web UI’s Artifacts feature, which is how you can collaborate with Claude on a document or a piece code. If it’s a web app, you can preview it without needing to copy and paste it elsewhere. I like the idea of integrating a code editor (like Sandpack) into the chatbot interface.

Let me know what you think

If this project is interesting to you, I’d love to hear what you think! Please feel free to reach out.

A Neovim shortcut for quickly extracting Twig components

2024-06-13T00:00:00Z

In the last week or so I’ve been getting up to speed learning how to make websites with Craft CMS. Craft is a PHP-based content management system that uses Twig templates.

In Twig, you can include smaller templates inside your template by doing something like this:

{{ include('partials/Header') }}

I’m using Neovim as my code editor - I’ve been using it for the past 6-12 months. Neovim is based on the original Vim editor that has been around since the early nineties, but is extremely customisable thanks to its Lua scripting engine. Lua is a flexible and lightweight programming language that is relatively straightforward to learn.

Building a basic homepage template in Twig, I found myself writing a lot of ‘include’ statements and separating components into separate Twig files. I was keen to make this more efficient if possible.

What you can do with Lua scripting in Neovim

Within your Neovim configuration file, it’s possible to assign keyboard shortcuts (or ‘keymaps’) to execute certain functions. These may be ‘built in’ functions, or custom functions you write yourself.

Within a custom function, you can automate key presses (so one ‘key map’ can execute a series of key presses), but you can also do more complicated things with files in the current project folder.

Extracting components in Twig files

I thought it would be interesting to create a custom shortcut where:

I could select some code, hit the space bar and then hit ‘et’ (standing for ‘[e]xtract [t]emplate’).
The editor would then ask me to type the name of the component that I want to save.
It would save a new Twig file with the selected code in it, at templates/partials/<Component name>.twig
It would replace the current selected text with an include statement like this:

{{ include('partials/<Component name>') }}

With a little help from ChatGPT, I was able to get this working fairly quickly!

Here’s the basic Lua code to get this working:

function ExtractTwigTemplate() -- Do not run if this is not a Twig file if vim.bo.filetype ~= 'twig' then print 'This is not a Twig file' return end -- Ask the user for the template filename local filename = vim.fn.input 'Enter new template filename: ' if filename == '' then print 'Filename cannot be empty!' return nil end -- Get the selected lines local start_line = vim.fn.line "'<" local end_line = vim.fn.line "'>" local component_lines = vim.fn.getline(start_line, end_line) -- If only one line is selected, convert it to a table (like an array in Lua) if type(component_lines) == 'string' then component_lines = { component_lines } end -- Write the lines to the new template file local component_filepath = string.format('templates/partials/%s.twig', filename) vim.fn.writefile(component_lines, component_filepath) -- Delete the selected lines vim.cmd "'<,'>d" local include_line = string.format("{{ include('partials/%s') }}", filename) -- Insert the include line vim.fn.append(start_line - 1, include_line) end -- Set the key map so that the function runs when the user presses <Leader>et in visual mode vim.api.nvim_set_keymap('x', '<Leader>et', ':lua ExtractTwigTemplate()<CR>', { noremap = true, silent = true })

To get this working, I added this code to the end of my init.lua file in my Neovim configuration folder. You can read more about configuring Neovim using Lua here.

So once this was added to my config file, I could select some code in a Twig file, hit space then ‘et’, and it would ask me for the component name. When I enter the component name and hit enter, it saves the selected code into a new file and replaces it in the current file with the ‘include’ statement. Handy!

Here’s a GIF showing how the keyboard shortcut works in practice:

(Side note: I used Keycastr to show the key strokes in the bottom corner of the screen, macOS’ native Screenshot tool to record the video, and Gifski to turn the video into a GIF.)

Taking it further with component props

Often when you’re creating components, you want to pass in variables to be included in the component. In Twig, you can do that like this, by passing in an object as the second parameter:

{{ include('partials/Card', { title: 'Title', description: 'description' } }}

I wanted to extend my shortcut so that it would ask for a list of ‘props’ that would then be included in the component.

This meant adding another question that the editor would ask after receiving the component name.

The shortcut would need to, for each prop:

add a line to the ‘include’ statement setting the prop (eg. title: '',)
add a line to the top of the component file itself, to show that the prop is avaiable within the component (eg. {% set title = title|default('') %}

the ‘object’ lines that are inserted into the ‘include’ statement (eg. , but I also wanted to include the props at the top of the component file itself, to show what props are available to the component:

So if I create a component called ‘Card’ with the props ‘title’ and ‘description’, the output should be:

{# where my selected code was #} {{ include('partials/Card', { title: '', heading: '', }) }} {# inside templates/partials/Card.php #} {% set title = title|default('') %} {% set description = description|default('') %} {# Original selected code goes here #}

This was a little bit more fiddly, because it required writing new lines of text and manipulating the Lua table of strings before writing the component file. But I’m happy with where this ended up. Here’s the updated code:

function ExtractTwigTemplateWithProps() if vim.bo.filetype ~= 'twig' then print 'This is not a Twig file' return end -- Ask the user for the template filename local filename = vim.fn.input 'Enter new template filename: ' if filename == '' then print 'Filename cannot be empty!' return nil end -- Define include_lines as a table so we can add props to it if we need local include_lines = { string.format("{{ include('partials/%s' }}", filename) } -- Get the selected lines local start_line = vim.fn.line "'<" local end_line = vim.fn.line "'>" local component_lines = vim.fn.getline(start_line, end_line) -- If only one line is selected, convert it to a table (like an array in Lua) if type(component_lines) == 'string' then component_lines = { component_lines } end -- Ask the user for a list of props local props_string = vim.fn.input 'Enter a list of properties, separated by commas, or leave blank to skip: ' if props_string ~= '' then -- Redefine include_lines, to open the props object include_lines = { string.format("{{ include('partials/%s', {", filename) } local props = vim.fn.split(props_string, ',') -- For each prop, write at the top of the component_lines for _, prop in ipairs(props) do -- Trim whitespace around prop local trimmed_prop = vim.fn.trim(prop) table.insert(component_lines, 1, string.format("{%% set %s = %s|default('') %%}", trimmed_prop, trimmed_prop)) table.insert(include_lines, string.format(" %s: '',", trimmed_prop)) end -- Close the props object at the end of the 'include' statement table.insert(include_lines, '}) }}') end -- Write the lines to the new template file local component_filepath = string.format('templates/partials/%s.twig', filename) vim.fn.writefile(component_lines, component_filepath) -- Delete the selected lines vim.cmd "'<,'>d" -- Insert the include lines vim.fn.append(start_line - 1, include_lines) end -- Set the key map so that the function runs when the user presses <Leader>et in visual mode vim.api.nvim_set_keymap('x', '<Leader>et', ':lua ExtractTwigTemplateWithProps()<CR>', { noremap = true, silent = true })

Here’s another GIF showing how the updated shortcut runs:

What else is possible?

I should point out that I am very new to creating my own shortcuts using Lua in Neovim, so I might not be doing things in the ideal way.

If you’ve made some of these shortcuts yourself, I would love to hear about them. Feel free to reach out.

If you think this sounds interesting but you don’t use Neovim, I would recommend giving it a shot! There is a learning curve when you start using a Vim-based editor, but once you get into the rhythm, you can navigate around code and get things done more efficiently.

Focusmate feels like a magic trick for my brain

2024-05-18T00:00:00Z

I have been trying out Focusmate - a web app that matches you with another person for a virtual coworking call - and it is helping me get a lot more done.

Moving to Paris

My partner and I are now living in Paris! As of 18 May 2024, we’ve now been here for about two weeks.

I have left my full-time job at the Information Access Group in Melbourne, Australia. So I’ve been adjusting to managing my own time, currently split between job search (currently looking for a full stack engineer role!), French language learning and side projects.

It has been a difficult transition in some ways - I’ve felt a slump in motivation. But in the last couple of days I’ve been trying something out which has been helping a lot - the web app Focusmate.

What is Focusmate?

Focusmate is a web app for virtual coworking - it matches you with another person for a 25 minute video call. At the start of the video call, you say what you are working on this session, then at the end of the session, you say how you went.

During the call, I put my mic on mute, and put the other person’s video in ‘picture-in-picture’ mode in the top right corner of my screen.

It sounds a bit strange to have a video call with a random stranger, but in the ~25 sessions I’ve had so far, everyone has been really nice and encouraging. It feels like a nice little community of likeminded people, who are using the app to help them get stuff done. People are studying for exams, writing press releases, making website wireframes.

I’m surprised how well Focusmate is working for me. Right now, it feels like a magic trick for my brain to click into ‘doing’ mode.

If you struggle with procrastination, or sometimes have a hard time following through on the things that you need or want to do, I would strongly recommend giving it a try.

How and why does it work?

I’m not entirely sure why Focusmate works so well for me, but here are a few ideas off the top of my head. It will be interesting to see if these ideas change over time.

It feels like Focusmate is helping my brain switch into a more active ‘doing’ mode where I can focus on one thing at a time, whereas before, if I was working on something that I’m not excited about, I would end up scrolling on social media. When I’m in a Focusmate session, I feel like I am staying on track for longer.
For me personally, impressing other people is a big motivator for me. I want to impress the other person at the end of the session when I tell them how I went. I don’t want to let them down.
Because you need to tell the other person what you’re working on at the start of the session, it forces you to clarify your task, at least enough to be able to describe it. It also forces you to pick one thing.
Because the session is only 25 minutes, thinking about what you can get done in 25 minutes, it forces you to break a big task down into smaller, more achievable chunks.
I think working with a total stranger can be a good thing, because all you know about the person is what they are doing for the next 25 mins, and that’s all they know about you. I think this helps the sessions feel light.

Let’s see how long this feeling lasts

Right now, I’m feeling really excited about how much I can get done with Focusmate. It feels like more is possible than before.

However, I have been excited about other productivity systems in the past! So I don’t expect this feeling to stay the same forever.

My motivation comes and goes in waves, so it will be interesting to see how Focusmate works for me when the motivation is ebbing.

I’m making a reminder for myself in a months’ time (18 June 2024) to add an update to this post.

Give it a try

I don’t think Focusmate is for everyone. Everyone has different things that motivate them. If you don’t struggle with motivation or procrastination, then you probably don’t need to try it.

But from my experience, procrastination is a difficult thing to talk about - it might be something you struggle with but no-one else knows. For example, if you work in a hybrid environment, then you might feel less productive working at home than when you work in the office.

If you ever struggle with these issues, I would strongly recommend giving Focusmate a try. You can do 3 sessions per week for free, or pay US$10 per month for unlimited sessions. It is a bit cheaper if you pay annually.

You can use my referral link here to get one month of unlimited sessions for free.

I’m keen to hear what you think. You can email me at larryhudson@hey.com.

Update after a couple of months

It’s now Wednesday 24 July 2024 - a couple of months after I wrote this post. It took me a while to get around to writing this update.

In the last month or so, I haven’t been using Focusmate as much as I did when it was new to me. The novelty factor has worn off, so it takes more effort for me to book in sessions now.

However, on the days where I use Focusmate, I feel more productive and focused. It’s easier for me to make progress on the things that I need to do, and stay ‘on the rails’, versus when I am just motivating myself.

The challenge for me now is to have the discipline to reach for Focusmate and use it consistently. The difficult thing is, when I need Focusmate the most (when I am feeling unproductive), is when my brain is not reaching for it. So I need to work on that.

How text-to-speech can make your content more accessible

2024-03-13T00:00:00Z

In March 2024, I wrote an article for the Information Access Group newsletter about how text-to-speech can make your content more accessible, aimed at content authors and readers.

For this article, we implemented a ‘Listen’ button that allows you to listen to the article while you read. This is the same functionality that the Easy Read HTML product uses.

You can read the article on the Information Access Group website.

A web app for taking screenshots of maps using Protomaps and MapLibre

2024-02-20T00:00:00Z

At the start of May, my partner and I will be moving from Australia to live in Paris for a year. We’re really excited! I’ve been trying to learn a bit more French before our move, so I’ve been using the spaced repetition app Anki to learn vocabulary.

I’m also keen to learn a bit of European geography, to help with navigation. I’ve been using this shared Anki deck to help me memorise the locations of European countries on a map.

For example, where on the map is Croatia?

Show the answer

Last month, I came across the open source mapping libraries Protomaps and MapLibre. These libraries make it possible to integrate maps into a web app using OpenStreetMap’s dataset. I thought it would be interesting to try making a little web app to generate screenshots of maps with locations highlighted. This could help me learn landmarks in Paris, and make me a bit more confident when travelling.

After a bit of experimentation with the Protomaps and MapLibre libraries, and a lot of help from Bishal Sapkota’s geojson.app project, I’ve got a little demo up and running! In the blog post below, I’ll explain how I got it working, and how you can try it out for yourself.

Experimenting with Mapbox

To get a feel for how mapping libraries work in a web app, I started with the examples on the Mapbox website. Mapbox is a popular set of APIs and services, and has some good documentation for getting started.

Because my app is a hobby project with only a couple of users, my usage would fit in Mapbox’s free tier. Mapbox only becomes expensive if your app has more than a few thousand users.

But I wanted to learn more about the open-source mapping libraries Protomaps and MapLibre, just for my own knowledge, and to work out what is possible with OpenStreetMap’s dataset.

Poking around the geojson.app project

Bishal Sapkota’s geojson.app project is a great example of a React app using Protomaps and MapLibre:

It uses MapLibre for the interactive map on the frontend. MapLibre renders the map into a <canvas> element and visualises the GeoJSON data using markers, lines and polygons.
It uses Protomaps to self-host the vector map tiles. As you navigate around the map, it loads the tile data from tiles.geojson.app.

You can learn more about the project in Bishal’s recent talk ‘Beyond Mainstream Maps’ at MelbJS.

I learned a lot by poking around the source code of Bishal’s project. By learning how to get a map working, and how to draw markers and polygons, I was able to get a head start on my own project.

Self-hosting the vector map tiles

In order to get my own app up and running, I needed to self-host my own vector map tiles. Here’s what the process involved:

using the pmtiles CLI to extract the vector map data for the Paris region, from the global OpenStreetMap dataset. The pmtiles file for the whole world is about 110GB, but the Paris region is only 10MB.
uploading my pmtiles file to a Cloudflare R2 bucket using the rclone CLI.
creating a Cloudflare worker to serve the HTTP requests on a subdomain. When the requests come in to pmtiles.larryhudson.net, the Cloudflare worker renders the data from the pmtiles file in the bucket.

I followed these instructions on the Protomaps website.

I had a little bit of difficulty getting this working. One step that I missed initially, was setting the ‘ALLOWED_ORIGINS’ environment variable, so that requests from my localhost:4321 development server would be allowed. Overall, this process took me around 30 mins to set up, so it wasn’t too bad.

OpenStreetMap’s search API

OpenStreetMap has a search API called Nominatim that allows you to search for locations and get their coordinates. It makes it fairly easy to lookup data for a specific place or region, and get GeoJSON data that can be visualised on a map. For example, here’s a search URL to look up the polygon data for the 11th arrondissement of Paris.

OpenStreetMap allows developers to use the Nominatim API for free, but their usage policy only allows one request per second. While this is ok for a hobby project, if you need to make more requests, they have alternatives on their wiki.

OpenStreetMap is a real gift for developers who want to do interesting things with maps - I’m really interested in experimenting more.

It was fairly straightforward integrating a search box into my web app. Because the Nominatim API returns GeoJSON data, I was able to visualise that GeoJSON data using the functionality within Bishal’s example.

Generating images from ‘canvas’ elements

The last part of the puzzle was taking screenshots of maps, so that I can use them in Anki flashcards.

I’ve recently been playing with the html2canvas library after finding out about it in Andrew Walpole’s great example for generating social media images for blog posts here.

It turns out, if you can render a <canvas> element, it’s fairly easy to turn that into a screenshot using JavaScript. You just need to get the <canvas> element from the DOM and then do canvas.toDataURL('image/png').

Because the MapLibre library renders the map as a <canvas> element, I was almost there. I just needed to enable the preserveDrawingBuffer option so that the canvas would be able to be rendered to an image. This option is disabled by default for performance reasons.

One ‘gotcha’ that I ran into was with map markers - if I added markers using the MapLibre library, they were not rendered in the canvas, because they are added as separate DOM elements. To get around this, I added a ‘layer’ for each marker to the map. You can see that in the source code here.

An example flashcard

Where on the map is the Arc de Triomphe?

Show the answer

Video walkthrough and GitHub repo

You can view a video walkthrough of the web app here:

You can also explore the source code for the project on GitHub.

Let me know what you think

If this is interesting to you, I’d love to hear what you think. Have you made something similar? Do you have any other ideas for interesting use cases for mapping in a web app?

Feel free to reach out on LinkedIn or Twitter.

How can we use AI to do better thinking, not skip thinking?

2024-02-15T00:00:00Z

Ethan Mollick recently wrote an article demonstrating what you can do with AI in under a minute. He shows that you can write a product launch, plan a course syllabus, design a kitchen and do market research all in under a minute. It’s a great demonstration of what we can generate with AI tools today.

I thought this was a great insight from Ethan’s article:

When a middle manager writes a weekly report on the status of a major initiative, the report may not be the point. Instead, it serves as a signal that the middle manager has done their job, speaking to the relevant employees, keeping an eye on the status of the project, and making corrections as needed. And it has always worked well enough - a senior manager could tell at a glance if the report was seemingly substantive (showing effort) and well-written (showing quality). But now every employee with Copilot can produce work that checks all the boxes of a formal report without necessarily representing underlying effort.

I think this is a big deal - while previously the ‘output’ (the report or summary) was proof of underlying thinking, now these outputs can be generated without much thinking at all. While these AI tools allow us to jump straight to the end result, they work better when we do more thinking and give them more context. In this blog post, I want to focus on how AI tools can help us do better thinking, rather than skip thinking.

Thinking and output

When thinking about work, it can be useful to break a task into two components:

thinking: considering the problem, making a plan, making decisions
output: a report, a presentation, a meeting agenda.

There are certain tasks that don’t require much thinking. You might have a meeting agenda that you need to prepare. You know the topics that need to be covered, and you know the order that they need to be covered in. If you have a template for the meeting agenda, the work is mostly about filling in the blanks.

There are other tasks that require more thinking. You might have an idea for a new way of working that will save time. You need to think about the potential benefits and challenges of using this new approach, and write a pitch to present to your manager.

With the recent wave of AI tools such as ChatGPT, people can jump straight to the output without doing much thinking. While this means we can get things done a lot faster, we need to be careful to not skip the ‘thinking’ part.

AI can create better outputs if you do more thinking

For an AI to create a good output, it needs to be given contextual information in its prompt. Without the proper context, it will generate a generic output that might not be relevant to your specific situation. Here’s an example of a generic prompt:

Write a list of pros and cons of using Jira for task management.

If you take some time to think about the task, prepare some rough notes, and then use those notes in your prompt, the AI will be able to generate a much better output. Here’s an example of a more contextual prompt:

I’m considering using Jira for task management. I work at an agency where we make websites for clients. Some of our team members are not technical and some are. We are currently organising tasks using a combination of Trello and Google Sheets. We are having trouble keeping track of tasks and changing deadlines. We need software that can help us manage tasks, deadlines, and deal with external clients. Can you write a list of pros and cons of using Jira for task management in this context?

Tools like ChatGPT can really shine when you give them more context. I think this is what sets them apart from traditional search engines.

AI tools can also help with the thinking part

While AI tools are great for generating outputs, they can also help you with the thinking part.

If you’re trying to decide between two different approaches, you can ask ChatGPT to write a summary arguing for one approach, and another summary arguing for the other approach. You can then read through those outputs and think about them deeply. In this way, ChatGPT is not making the decision for you, but is helping you to think about the problem from different angles.

ChatGPT can also generate a list of ideas for you to consider. For example, if you want to create an app that helps people save money, you could ask ChatGPT to generate a list of 20 different app ideas that help people save money. In this way, ChatGPT can be more like a generative design tool that prepares many options or combinations of ideas.

I have been using the brainstorming app Brainstory to help me think deeper about my ideas. In the app, an AI coach asks me thought-provoking questions and I answer the questions using my voice. The app transcribes the conversation and generates a concise summary. I think this is a great example of how AI can help users think deeper, rather than skip thinking.

It’s worth noting that it is still early days for AI in the workplace. As time goes on, AI tools are going to do more and more of the thinking for us. Tasks that require ‘thinking’ today, such as finding relevant contextual information and making connections between different pieces of information, will be completed by AI in the future. So our concept of ‘thinking’ will keep changing over time.

Let me know what you think

I think it’s fascinating to think about how AI tools are changing the way we work.

If this is interesting to you, feel free to reach out on LinkedIn or Twitter. I’d love to hear what you think, and if you have any ideas for using AI tools to help you think better.

Using Weaviate's generative search for brainstorming

2024-02-10T00:00:00Z

Lately I’ve been interested in generative search - the idea of searching through a database of text items by semantic similarity and then generating new text based on the search results. I think this is a really powerful idea.

I’ve created astro-weaviate-brainstorm - an example web app that demonstrates this concept using the Astro framework and the Weaviate vector database. The web app integrates vector search, which facilitates text search based on semantic meaning, with a large language model to generate new text based on search results.

This project is heavily inspired by the brainstorming app Brainstory, a web app that guides you through a brainstorming session by asking thought-provoking questions. At the end of the brainstorming session, it generates a concise summary for you. This encourages you to think deeper and can help you turn your idea into something tangible and shareable. If you want a real brainstorming app, try out Brainstory!

In this blog post, I’ll introduce the idea of generative search, and then talk about how you can get started trying out the astro-weaviate-brainstorm project.

What is generative search?

Generative search (also called Retrieval Augmented Generation or RAG) is a concept that uses vector search to find semantically similar text in a database, and then uses a large language model to generate new text based on the search results. Let’s break that down a bit:

vector search is a way to search through a database of text items by semantic similarity, rather than exact keywords. For example, the text strings “I love dogs” and “I love cats” are very similar semantically. Vector search can find these similarities between text items, and can make it easier to find related items to a search query.
a large language model is a machine learning model that can generate new text based on a prompt. For example, if you give it the prompt “Summarise the key ideas from this text” along with a piece of text, it can generate a summary of the text. This is the technology that is behind ChatGPT.

By combining vector search with a large language model, generative search can be a really powerful tool. In the context of a brainstorming app, it could be used to:

find similar ideas in different brainstorms
find insights or connections between brainstorms that you might not think are related.

You can find out more about generative search in the Weaviate docs.

An example web app using Astro and Weaviate

I’ve created a new side project called astro-weaviate-brainstorm as a way to explore the possibilities of generative search. It’s a fairly simple web app using the Astro web app framework and the Weaviate vector database. The web app demonstrates the concept of generative search, and how it can be used in a brainstorming app.

Astro is my favourite way to create simple web apps using Node.js. The framework’s .astro component files make it really easy to write pages and reusable components using HTML and CSS. Astro’s ‘fenced’ section at the top of the file --- component syntax makes it easy to include server-side logic at the top of the file, which makes it easy to work with HTML forms.

Weaviate is an open source vector database that can be used to store and search through text items by semantic similarity. While it can be deployed using Docker, for this project I am using Weaviate’s embedded client, which runs the database within the same process as the Astro web app. This is a great fit for small projects where you don’t want to set up more complicated infrastructure. All you need to bring is an OpenAI API key, which generates the text embeddings and powers the large language model.

Getting started with astro-weaviate-brainstorm

To get the project running on your computer:

clone this git repository
change into the project directory and run npm install to install the Node.js dependencies
duplicate the sample .env.sample file and rename it to .env, and then add your OpenAI API key to the .env file
run npm run dev to start the local development server
open your web browser and navigate to http:/localhost:4321

Video walkthrough

I’ve also recorded a video walkthrough where I talk through the idea and demonstrate a few things with the web app. I’m trying out a few different ways to share my side projects, so let me know if you like the video format. You can view the video on YouTube below:

Let me know if this is interesting to you

I’m keen to hear if this concept is interesting to you, and if you have any ideas for how it could be used. I think generative search could be a really powerful tool for brainstorming, and I’m excited to explore it further.

If this idea is interesting to you, feel free to share it on Twitter or LinkedIn.

Active reading with Readwise, Brainstory and Anki

2024-01-28T00:00:00Z

Lately I’ve been interested in active reading techniques and how we can turn the interesting information that we read into useful knowledge.

A lot of these ideas are influenced by Michael Nielsen’s article Augmenting Long-term Memory and Andy Matuschak’s ideas about self-guided learning.

In this blog post, I’ll talk about a few apps that I’ve been using to help me read more actively, think through ideas with brainstorming, and resurface those ideas with spaced repetition: Readwise, Brainstory and Anki. By using these tools, it’s possible to get more value out of the content that we read.

Readwise is great for active reading

Readwise is an app that lets you save highlights from the books, articles or documents you’re reading. The app then resurfaces those highlights over time, so you can retain the insights and important information.

Readwise has a separate app called Readwise Reader, which allows you to store articles, documents and other types of content that you want to read. It gives you a nice interface to do that reading across your devices, and makes it easy to highlight important information, which syncs over to the main Readwise app.

As an aside, Readwise Reader’s mobile app also has a great text to speech function, which means you can listen to your articles while you go for a walk or do the dishes. This is huge for me - it’s actually quite similar to my astro-sqlite-tts-feed side project that I have been working on.

Readwise Reader works great with longform YouTube videos too. When you add a YouTube video, the video plays in an embedded player at the top of the screen, and a transcript automatically scrolls below the video, where you can quickly highlight important information. This makes video watching more active, and makes it easier to save the key points. I did this with Dwarkesh Patel’s interview with Andy Matuschak. I was able to save quite a few insights from their discussion about self-guided learning.

Thinking deeper with Brainstory

Brainstory is a web app that guides you through a brainstorming session by asking thought-provoking questions. At the end of the brainstorming session, it generates a concise summary for you. This encourages you to think deeper and can help you turn your idea into something tangible and shareable.

In this Andy Matuschak interview, he mentions that the most important rule of active reading is asking questions about the content and answering them. When he mentioned that, I immediately thought of Brainstory, which I’ve been playing around with for the last couple of months.

I think Brainstory can be a useful tool in the active reading process. If you highlight interesting information in an article, you can copy and paste those highlights into the start of a brainstorming session, and then ask Brainstory to help you think through each highlight.

The process of thinking through each highlight can help you transform the content from something passive that someone else has written, into something that is more relevant and applicable to your own experience. For example, I did a Brainstory about the highlights in that Andy Matuschak video, and it helped me relate one of the insights to my own experience. You can read that Brainstory here.

I’m hoping that by transforming the original ideas into something new, I can retain the information more. If you’re interested, you can read my Brainstory about the Andy Matuschak video here.

This is just one possible use case for Brainstory. I think the idea of a guided brainstorming session, which helps you think deeper and then generates an ‘output’ summary that you can share and get feedback on, is really powerful. I’m excited to play around more with it and see where else it could be useful.

Spaced repetition with Anki for resurfacing ideas

One more thing that I’ve been exploring is spaced repetition for memorisation and resurfacing ideas. With an app like Anki, you can create flashcards and test yourself, and the app will retest you over longer and longer periods of time, to help the information stay in your long-term memory. If this is interesting to you, I’d recommend checking out the article Augmenting Long-term Memory. Use Readwise Reader if you want to listen to it using text to speech.

I think spaced repetition is another important part of the puzzle when it comes to active reading. After you read some material and highlight the interesting parts, and then ask yourself questions, you need to revisit the material in order for it to stay in long-term memory.

While spaced repetition is most popular for learning foreign language vocab, I’m keen to explore how useful it could be for resurfacing ideas in order to make new connections. For example, a few topics that I’m interested in are asynchronous communication in the workplace, using text-to-speech to listen to text content, and productivity techniques to do focused work. If I have flashcards related to these ideas, and I’m constantly circulating these ideas, then that could help me make new connections and insights.

Putting it all together

I’m pretty excited about the potential of combining active reading with Readwise, deeper thinking with Brainstory and spaced repetition with Anki. Hopefully it will help me get more value out of the information that I’m reading.

I actually planned this blog post in a Brainstory session. Brainstory has a great feedback function where you can share your brainstorm, and then other people can do a Brainstory in response. If the ideas in this blog post are interesting to you, you can do a Brainstory in response here. It will be a guided brainstorming session where it will ask you thought-provoking questions.

Turning a scanned PDF into an audiobook with Azure Document Intelligence and OpenAI APIs

2023-12-17T00:00:00Z

I’ve been experimenting with turning scanned PDFs into audiobooks using:

the Microsoft Document Intelligence API to extract text from the PDF
the OpenAI GPT-4 API to clean up the text
the OpenAI Text-to-Speech API to turn the cleaned text into audio.

I’ve found these APIs work really well together, and I’m excited about the potential here. If you like listening to long-form content, then I think this method is worth exploring.

In this post, I’ll walk through how I’m using these APIs to turn a scanned PDF into an audiobook. For each part, I’ll share a Node.js code example.

Extracting text from a PDF using the Document Intelligence API

Microsoft’s Document Intelligence API makes it pretty straightforward to extract text from a PDF. I’ve been really impressed with the quality of the text extraction even when the PDF is a low quality scan.

You can have a play with the API using the Document Intelligence Studio in your web browser.

To get started using the Document Intelligence API, you’ll need to set up an Azure account and create a new ‘resource’ for the Document Intelligence API.

It has a free tier that allows you to extract text from 500 pages per month. You can find out more information about pricing on the Azure website.

A few things to note here:

the Document Intelligence API only extracts text from a single page at a time, so in my Node script, I create a new PDF for each page using the pdf-lib library, then send each ‘page buffer’ to the API.
in my Node script, I write a text file for each page of the PDF, so that each text file can be sent to the GPT-4 API for correction.
rather than extracting text from every page in the PDF, I split up the original PDF into chapters. I didn’t want to use up the free quota too quickly, so I only extracted text from the pages that I needed.

Node.js code example for extracting text from a PDF

import fs from 'fs'; import path from 'path'; import { PDFDocument } from 'pdf-lib'; import { DocumentAnalysisClient, AzureKeyCredential, } from '@azure/ai-form-recognizer'; import 'dotenv/config'; import pMap from 'p-map'; // Example usage: // node extract-text-from-pdf.js ./my-pdf.pdf const [pdfPath] = process.argv.slice(2); const extractedPages = await extractTextFromFullPdf(pdfPath); const pdfFilename = path.basename(pdfPath, '.pdf'); writeTextFiles(extractedPages, pdfFilename); // this function takes a PDF path, and for each page, creates an individual PDF // and extracts the text using the Document Intelligence API async function extractTextFromFullPdf(pdfPath) { const fullPdfBytes = fs.readFileSync(pdfPath); const fullPdfDoc = await PDFDocument.load(fullPdfBytes); const pageCount = fullPdfDoc.getPageCount(); const pageNumbers = Array.from({ length: pageCount }, (_, i) => i + 1); const extractedPages = await pMap( pageNumbers, async (pageNumber) => { const pdfPageBuffer = await getPdfForIndividualPage(pdfPath, pageNumber); const pdfData = await extractTextFromPagePdf(pdfPageBuffer); const content = pdfData.content; return { pageNumber, content }; }, { concurrency: 2 }, ); return extractedPages; } // the Document Intelligence API only supports extracting text from a single // page at a time, so this function creates a new PDF for a single page async function getPdfForIndividualPage(pdfPath, pageNumber) { const existingPdfBytes = fs.readFileSync(pdfPath); const fullPdfDoc = await PDFDocument.load(existingPdfBytes); const singlePagePdf = await PDFDocument.create(); const pageIndex = pageNumber - 1; const [copiedPage] = await singlePagePdf.copyPages(fullPdfDoc, [pageIndex]); singlePagePdf.addPage(copiedPage); const singlePagePdfBytes = await singlePagePdf.save(); const singlePagePdfBuffer = Buffer.from(singlePagePdfBytes); return singlePagePdfBuffer; } // this function takes a PDF buffer and sends it to the Document Intelligence // API and returns the result async function extractTextFromPagePdf(pdfPageBuffer) { // environment variables should be in .env file const endpoint = process.env.AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT; const apiKey = process.env.AZURE_DOCUMENT_INTELLIGENCE_KEY; const documentAnalysisClient = new DocumentAnalysisClient( endpoint, new AzureKeyCredential(apiKey), ); const poller = await documentAnalysisClient.beginAnalyzeDocument( 'prebuilt-read', pdfPageBuffer, ); const pdfDataResult = await poller.pollUntilDone(); return pdfDataResult; } // this function writes a text file for each page of the PDF function writeTextFiles(extractedPages, pdfFilename) { const outputFolderName = 'extracted-pages'; if (!fs.existsSync(outputFolderName)) { fs.mkdirSync(outputFolderName); } for (const { pageNumber, content } of extractedPages) { const txtFilename = `${pdfFilename}_${pageNumber}.txt`; const txtFilePath = path.join(outputFolderName, txtFilename); fs.writeFileSync(txtFilePath, content); } }

Cleaning up the extracted text with GPT-4

While the Document Intelligence API does a good job of extracting text from the page, there can be a few issues with the extracted text:

if sections of the page are blurry, the text can be garbled
if the scanned page contains characters that shouldn’t be on the page (eg. they are on the edge of an adjacent page), then those characters will show up in the extracted text.

Here’s a screenshot of a page that includes these issues:

Here’s what the Document Intelligence API extracted from that page. For quite a few lines, the start of the line is garbled, and there are extra characters from the previous page:

of Underground, Collapsing and the double-album Disaster ot. - all came from it. Some claim that they split from music altoes- gherto continue in a purely political way, but kept up the illu- sion for years with seemingly new LPs. The first Amon Düül records are extraordinary classics and extremely raw, like coned Orcs playing neverending versions of The Mothers' Return of the Son of Monster Magnet" and the Stooges at LA.Blues". But they are dosed with a higher level of vibe than es any other freakout records - relentless, uplifting and full of the le crodest gimmicks that all work perfectly. Amon Düül did not re stay long, but they laid the beginnings of Krautrock with their ic music, and with one particular song on Psychedelic Undera emund. The name of the song translated as "Mama Düül and it er Sauerkraut band Start Up!" With that title, the lazy British d S ock press at last had something to latch on to. Aha, we'll call it Krautrock ... e 1 e The First Rumblings of Kosmische Music W. Germany was full of supposed 'head' groups by now. But any of them still did not sound remotely German, slavishly trying to be Hard Rach !!! Others, like Embryo, Emergency and Birth Control, mixed obvious Teutonics into unsuccessful fusions with British/American rock and jazz. But in the mean- ame, Amon Düül II, the musical half of the commune, had recorded an amazing free-flowing LP called Phallus Dei, for the British Liberty label. Its overtly mysterious sleeve first con- Booted me when I was 13 and standing in Tamworth Wool- worth's. I was with my Welsh grandfather, whom I asked about the meaning of Phallus Dei. "Bloody Hell, Don't tell your mother." he snorted. "That means God's cock!" And with the release of that 20 minute title track, both branches of Amon Dual had proved their commitment to the new cosmic political commune scene. This record was very extreme, both the chim- og sound and dizzy two-colour sleeve like something from the 13th Floor Elevators' International Artists label in Texas. And something else again was stirring in Cologne. The Stockhausen/Psychedelia-inspired Can were now a five-piece recording at Schloss Norvenich, the castle home of their patron, 13 `

I experimented with how to clean up the text, including using the GPT-4 Vision API to try to extract text from blurred sections. The Vision API is very powerful but it’s not suited to straightforward text extraction - it is better suited to asking questions about an image.

In the end, I found that asking GPT-4 to clean up the extracted text worked pretty well. The system prompt that I landed on was:

The supplied text has been extracted from a blurry page. There may be characters from the edge of adjacent pages that need to be deleted. Correct the text so that it makes sense.

Here’s the cleaned up text for the example page above:

"Underground, Collapsing" and the double-album "Disaster" all came from it. Some claim that they split from music altogether to continue in a purely political way but kept up the illusion for years with seemingly new LPs. The first Amon Düül records are extraordinary classics and extremely raw, like stoned Orcs playing never-ending versions of The Mothers' "Return of the Son of Monster Magnet" and the Stooges' "LA Blues". But they are dosed with a higher level of vibe than any other freak-out records - relentless, uplifting and full of the oddest gimmicks that all work perfectly. Amon Düül did not stay long, but they laid the beginnings of Krautrock with their music, and with one particular song on "Psychedelic Underground". The name of the song translated as "Mama Düül and her Sauerkraut band Start Up!" With that title, the lazy British rock press at last had something to latch on to. Aha, we'll call it Krautrock... The First Rumblings of Kosmische Music West Germany was full of supposed 'head' groups by now. But many of them still did not sound remotely German, slavishly trying to be Hard Rock. Others, like Embryo, Emergency and Birth Control, mixed obvious Teutonics into unsuccessful fusions with British/American rock and jazz. But in the meantime, Amon Düül II, the musical half of the commune, had recorded an amazing free-flowing LP called "Phallus Dei", for the British Liberty label. Its overtly mysterious sleeve first confused me when I was 13 and standing in Tamworth Woolworth's. I was with my Welsh grandfather, whom I asked about the meaning of Phallus Dei. "Bloody Hell, Don't tell your mother." he snorted. "That means God's cock!" And with the release of that 20-minute title track, both branches of Amon Düül had proved their commitment to the new cosmic political commune scene. This record was very extreme, both the chimeric sound and dizzy two-color sleeve like something from the 13th Floor Elevators' International Artists label in Texas. And something else again was stirring in Cologne. The Stockhausen/Psychedelia-inspired Can were now a five-piece recording at Schloss Norvenich, the castle home of their patron.

I’m pretty happy with that result! While it’s not perfect (I can see that it says ‘oddest gimmicks’ where it should say ‘crudest gimmicks’), this makes a page that was previously unlistenable into something that will work reasonably well in the audiobook form.

A couple of things to keep in mind here:

if the Document Intelligence API has missed a word because it is too blurry, GPT-4 will guess what the word would be. In a book I was reading, it was mentioning a person called ‘Ulli Pop’, but GPT-4 guessed the name was ‘Iggy Pop’, which was a surprise.
If the extracted text includes swear words, GPT-4 will filter them out. This may be avoidable by tweaking the system prompt.
As anything with untrusted input, prompt injection is a possibility - if you are extracting text from a document that is telling a robot to do something, that may trip up GPT-4.

Node.js code example for correcting text with GPT-4

import fs from 'fs'; import path from 'path'; import 'dotenv/config'; // Example usage: // node correct-text.js ./my-text.txt const [inputFilePath] = process.argv.slice(2); // read the text file const inputText = fs.readFileSync(inputFilePath, 'utf8'); // correct the text const correctedText = await correctText(inputText); // check output folder exists const outputFolderName = 'corrected-pages'; if (!fs.existsSync(outputFolderName)) { fs.mkdirSync(outputFolderName); } // write the output file const inputFileName = path.basename(inputFilePath, '.txt'); const outputFilePath = path.join(outputFolderName, inputFileName + '.txt'); fs.writeFileSync(outputFilePath, correctedText); // this sends the text to the GPT-4 API and returns the corrected text. async function correctText(text) { const openAiUrl = 'https://api.openai.com/v1/chat/completions'; const systemPrompt = 'The supplied text has been extracted from a blurry page. There may be characters from the edge of adjacent pages that need to be deleted. Correct the text so that it makes sense.'; const requestBody = { model: 'gpt-4', messages: [ { role: 'system', content: systemPrompt, }, { role: 'user', content: text, }, ], }; const openAiResponse = await fetch(openAiUrl, { method: 'POST', headers: { // environment variables should be in .env file Authorization: `Bearer ${process.env.OPENAI_API_KEY}`, 'Content-Type': 'application/json', }, body: JSON.stringify(requestBody), }); const responseJson = await openAiResponse.json(); const correctedText = responseJson.choices[0].message.content; return correctedText; }

Converting the text into audio with OpenAI’s text to speech API

In the past year, I’ve been experimenting with a few different text to speech tools. Microsoft Azure’s neural text to speech API has been my go-to. I’ve also been impressed by the quality of the ElevenLabs API, but its pricing is not feasible for large documents.

This week I’ve been playing around with OpenAI’s text to speech API. It only has a few voices at this stage, but the quality is very good, and the pricing is comparable with Azure, which makes it a good option for this project.

It seems that this latest generation of text to speech APIs, including ElevenLabs and OpenAI, are more flexible in adapting the voice to suit the content. I’ve been listening to two different books with different styles (Krautrocksampler by Julian Cope and Kraftwerk by Pascal Bussy), using the same ‘Onyx’ voice from the OpenAI API, and it’s interesting how the voice changes to suit the content.

Here’s an example from Krautrocksampler, which has a more informal style:

And here’s an example from Kraftwerk, which is more formal:

It’s worth noting the OpenAI API has a max input length of 4096 characters. This means we need to break our text into chunks, get audio for each chunk, and then join the chunks together.

Node.js code example for converting text to speech

import fs from 'fs'; import path from 'path'; import pMap from 'p-map'; import 'dotenv/config'; // Example usage: // node convert-text-to-speech.js ./my-text.txt const [inputPath] = process.argv.slice(2); const inputText = fs.readFileSync(inputPath, 'utf8'); const textChunks = breakTextIntoChunks(inputText); const audioBuffer = await convertTextChunksToAudio(textChunks); const inputFilename = path.basename(inputPath, '.txt'); const outputFilePath = `./${inputFilename}.mp3`; fs.writeFileSync(outputFilePath, audioBuffer); process.exit(0); // break text into chunks that are no longer than 4000 chars function breakTextIntoChunks(text) { const sentenceSeparator = '.\n'; let chunks = []; let currentChunk = ''; // split text into sentences, and filter out empty sentences const sentences = text.split(sentenceSeparator).filter(Boolean); for (let sentence of sentences) { if (currentChunk.length + sentence.length > 4000) { chunks.push(currentChunk); currentChunk = ''; } currentChunk += sentence + sentenceSeparator; } if (currentChunk.length > 0) { chunks.push(currentChunk); } return chunks; } async function convertTextChunksToAudio(chunks) { const audioBuffers = await pMap(chunks, convertTextToAudio, { concurrency: 2, }); // join the audio buffers into a single audio buffer const audioBuffer = Buffer.concat(audioBuffers); return audioBuffer; } async function convertTextToAudio(text) { const textIsTooLong = text.length > 4000; if (textIsTooLong) { console.log( 'Warning: text is too long, only converting first 4000 characters', ); } const textToConvert = text.slice(0, 4000); const ttsApiUrl = 'https://api.openai.com/v1/audio/speech'; const requestBody = { model: 'tts-1', input: textToConvert, voice: 'onyx', }; const OPENAI_API_KEY = process.env.OPENAI_API_KEY; const audioResponse = await fetch(ttsApiUrl, { method: 'POST', headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${OPENAI_API_KEY}`, }, body: JSON.stringify(requestBody), }); const audioArrayBuffer = await audioResponse.arrayBuffer(); const audioBuffer = Buffer.from(audioArrayBuffer); return audioBuffer; }

An extra tip: splitting a long PDF into smaller chapters

If you’re working with a big PDF, your costs for using these APIs will add up quickly.

Rather than converting a full book PDF in one go, I have been splitting up the PDF into chapters. That way, I can create audio for chapters as I get to them.

Below is a Node.js code example for creating a smaller PDF from a PDF, using the ‘start’ and ‘end’ page numbers. a PDF into chapters. It uses the pdf-lib library to create a new PDF using the start and end page numbers, and allows me to set a ‘page number offset’ in the case the page number in the bottom corner of the page doesn’t line up with the actual page number.

If I wanted to extract pages 20 to 30 from a PDF, and page 20 is page 25 in the PDF, I could use the script this way use the script like this: node extract-pages-from-pdf.js ./my-pdf.pdf 20 30 5

That would save a new PDF called my-pdf_pages_20_to_30.pdf, ready for extraction.

Node.js code example for extracting pages from a big PDF into a smaller PDF

import fs from 'fs'; import path from 'path'; import { PDFDocument } from 'pdf-lib'; // Usage: node extract-pages-from-pdf.js [pdf-path] [start-page-num] [end-page-num] [page-number-offset] // Example usage: node extract-pages-from-pdf.js ./my-pdf.pdf 0 5 5 const [pdfPath, startPageNumStr, endPageNumStr, pageNumberOffsetStr] = process.argv.slice(2); // arguments are strings, so we need to convert them to integers const startPageNum = parseInt(startPageNumStr, 10); const endPageNum = parseInt(endPageNumStr, 10); // page number offset is added to the page number, // in case the page number in the bottom corner of the page is different to the actual page number const pageNumberOffset = parseInt(pageNumberOffsetStr, 10) || 0; const newPdfBytes = await extractPdfPages( pdfPath, startPageNum, endPageNum, pageNumberOffset, ); const pdfFilename = path.basename(pdfPath, '.pdf'); const newPdfTitle = `${pdfFilename}_pages_${startPageNum}_to_${endPageNum}.pdf`; fs.writeFileSync(newPdfTitle, newPdfBytes); async function extractPdfPages( pdfPath, startPageNum, endPageNum, pageNumberOffset, ) { const fullPdfBytes = fs.readFileSync(pdfPath); // Load a PDFDocument from the existing PDF bytes const fullPdfDoc = await PDFDocument.load(fullPdfBytes); const startPageIndex = startPageNum - 1 + pageNumberOffset; const endPageIndex = endPageNum - 1 + pageNumberOffset; console.log({ startPageIndex, endPageIndex }); // get page numbers from start to end const pageIndices = Array.from( { length: endPageIndex - startPageIndex + 1 }, (_, i) => i + startPageIndex, ); const newPdf = await PDFDocument.create(); const copiedPages = await newPdf.copyPages(fullPdfDoc, pageIndices); for (const copiedPage of copiedPages) { newPdf.addPage(copiedPage); } const newPdfBytes = await newPdf.save(); return newPdfBytes; }

Wrapping up

I’ll continue experimenting with this method. Any projects that I make will be in public repositories on my GitHub account.

If this is interesting to you, feel free to reach out. My email address is larryhudson@hey.com