Puppeteer tricks to write a bot/scrapper

I wrote a bot recently that does some actions after it authenticates with an account using puppeteer and it was incredibly easy.
In case you don't know what puppeteer does, well, it is simply a Node library to control Chromium or Chrome programmatically, now such a library can have many uses but I'll focus on what you need to know when writing a scrapper/bot.

The first thing to know is what kind of website you are dealing with, it is a traditional website or is more like a SPA type of website? And you need to know that because using the waitForNavigation is practically useless on SPA websites.

Most of the information, you'll need can be found in the puppeteer documentation, which is available at https://github.com/GoogleChrome/puppeteer/blob/v1.8.0/docs/api.md.

Now if you will write a bot or a scrapper for a popular website chances are that the website we will use some methods to detect the Chromium browser that comes with puppeteer, one quick solution is to use a plugin called "puppeteer-extra-plugin-stealth", the plugin will make that version of Chromium pass many tests, but if it still fails the newer versions of puppeteer can work with a full-fledge chrome, also if you use the Chromium that's provided with the puppeteer library in case you deal with a website where you need authentication is also great to start the browser with the option for permanent browser user profile.

Here's an example for a quick setup:

const puppeteer = require("puppeteer-extra")
const expect = require('expect-puppeteer')


// add stealth plugin and use defaults (all evasion techniques)
const pluginStealth = require("puppeteer-extra-plugin-stealth")()
puppeteer.use(pluginStealth)

let exampleUrl = "https://yourexample.here";

(async () => {


    const browser = await puppeteer.launch({ headless: false, args: ["--no-sandbox"], userDataDir: './tmp/chrome' });
    const page = await browser.newPage();
    page.setDefaultTimeout(0);



    const context = browser.defaultBrowserContext();
    await context.overridePermissions(exampleUrl, ['notifications']);


    await page.goto(exampleUrl);

    // Do some things 

    await browser.close();


})();

So as you can see there are a few things in this example, first as you can see I included the library "expect-puppeteer" but I never used it in the above example, that's because I wanted to state that this is another small helper library that can help you with writing code for SPA type of website. It can help you transforms actions like click(), type() into functions that try to do that action until they hit a timeout I, believe the default is half of a second, you should check the library on GitHub.