There are times we need to use Puppeteer and appear as a different user type or from a different user device. This value is set in the User-Agent request header. Applications will use this value to determine what browser their users commonly use to determine things like how much they want to focus on certain device types or browser support. Puppeteer is a very powerful tool, using the Chrome headless browser, and it allows the developer to do a lot of powerful tasks. From automated testing to web scraping.
Sometimes we need to “act” as another user type or device type and that’s where we need to modify the User-Agent request header.
A default User-Agent might look something like:
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/85.0.4182.0 Safari/537.36";
Notice the HeadlessChrome portion of the User-Agent string.
If we change the string in Puppeteer, we can “appear” as if we’re a different user or device.
New User-Agent:
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4182.0 Safari/537.36"
The User-Agent is set on the newPage() instance using the setUserAgent() method. To modify the User-Agent your Puppeteer code will look something like this:
const puppeteer = require("puppeteer");
const browser = await puppeteer.launch({
headless: true,
});
const page = await browser.newPage();
page.setUserAgent(
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4182.0 Safari/537.36"
);
Note: If you’re trying to prevent certain browsers, devices, or user types based on the User-Agent header, it’s a losing battle. As you can see it’s easy to modify the User-Agent which circumvents any blocks. I suggest allowing the traffic and monitor the data the various User-Agents are using, it could be an opportunity to make that information available through other methods, like an API.