Scraping websites and bot-protection
The levels of bot protection I met while scraping genealogy sites, from a bare curl to shell-launched Chrome, and how to get past each one.

Technologist focusing on cloud-native technologies, DevOps, CI/CD pipelines, and system observability. His focus revolves around creating technical content, delivering talks, and engaging with developer communities to promote the adoption of modern software practices. With a strong background in software, he has worked extensively with the JVM, applying his expertise across various industries. In addition to his technical work, he is the author of several books and regularly shares insights through his blog and open-source contributions.
387 views

Nicolas Frankel
124 viewsThe levels of bot protection I met while scraping genealogy sites, from a bare curl to shell-launched Chrome, and how to get past each one.

Nicolas Frankel
167 viewsA follow-up on AI-assisted genealogy: source quality with GEDCOM QUAY, per-site browsing skills, Playwright sessions, transcriptions, and …

Nicolas Frankel
192 viewsYears ago, a colleague of mine told me about two pockets of foreign land surrounded by Switzerland: Campione d'Italia and Büsingen am …

Nicolas Frankel
59 viewsThis review is about Build an AI Agent (From Scratch) by Jungjun Hur and Younghee Song from Manning. Facts 10 chapters $29.29 (at the time …

Nicolas Frankel
30 viewsMy son recently came to me to brag about using AI to find our ancestors. While the results were correct, I didn't learn anything new, as it …

Nicolas Frankel
35 viewsMy current company has settled on using Gradle. It doesn't make me very happy, but you need to learn to work with constraints. Plus, I must …

Nicolas Frankel
31 viewsI've been a big fan of Renovate for a couple of years already. Renovate scans your repositories, detects outdated package versions, and …

Nicolas Frankel
15 viewsI started as a Java developer, but for some time now, I have broadened my horizons. Recently, I thought about how early languages were …

Nicolas Frankel
32 viewsIn the first post of this series, I focused on the project foundations: what should I do to create a video from photos taken from the same …

Nicolas Frankel
22 viewsBefore working for 2 years on the Apache APISIX API gateway, I was mainly oblivious to API gateways. It's only by working with them that I …