You are currently viewing Web Scraping with Node.js

Web Scraping with Node.js



Learn how to do basic web scraping using Node.js in this tutorial. The request-promise and cheerio libraries are used.

💻 Github: https://github.com/beaucarnes/fcc-project-tutorials/blob/master/node-web-scraping/index.js

🔗 Article on client-side web scraping : https://medium.freecodecamp.org/client-side-web-scraping-with-javascript-using-jquery-and-regex-5b57a271cb86

🐦 Beau Carnes on Twitter: https://twitter.com/carnesbeau


Learn to code for free and get a developer job: https://www.freecodecamp.com

Read hundreds of articles on technology: https://medium.freecodecamp.com

And subscribe for new programming videos every day: https://youtube.com/subscription_center?add_user=freecodecamp

source

This Post Has 28 Comments

  1. AMARNATH CHINTHA

    I am having this error when I am trying to do npm install
    Can anyone know why is this caused or how to solve this.
    Help required

    npm ERR! path /Users/FinalProject/node_modules/jquery
    npm ERR! code ELOOP
    npm ERR! errno -62
    npm ERR! syscall access
    npm ERR! ELOOP: too many symbolic links encountered, access '/Users/FinalProject/node_modules/jquery'

  2. sBastion

    Thanks a bunch for the tutorial! You explained it clearly & helped me in understanding this tutorial 👌

    UPDATE:
    When I tried to run this code, I got an error that says:
    "TypeError: data.directory_items is not iterable"

    How do I fix this?

  3. Jonathan Antoine

    Someone know why he uses a network request link to get all users but go directly on the profile link to get challenges data? Isn't the challenges data on the profile page also dynamic?

  4. Loudkill_

    I make this tuto on February 15, 2020 by chance. Funny, isn't it ?

  5. Payman Khayree

    can we call the function next() at the end of the function getChallenges…() instead of returning it?

  6. PyMoondra

    Really nice job. I like your teaching style. Hope to see more videos from you.

  7. PyMoondra

    Does the link within video used not work anymore?
    Do we need to use some sort of headers?

  8. viko roze

    I have a question. This method will work on the angular and react?

  9. Andre Allwood

    GREAT!!!!!
    Is there any way for you to allow the "index.js" to run when you click a button?

  10. J L

    and then you could run some statistical test to find if there's a correlation between num_hearts and num_challenges_passed

  11. Loizos Vasileiou

    WEBSITE HAVE CHANGED STRUCTURE… Here is how I fixed it (without completed challanges)..

    First change table structure:

    let table = new Table({
    head: ['username', '❤️'],
    colWidths: [15, 15]

    })

    Then change function to:

    function getChallengesCompletedAndPushToUserArray(userData) {

    var i = 0;

    function next() {

    if (i < userData.length) {

    process.stdout.write(`.`);

    table.push([userData[i].name, userData[i].likes_received]);

    ++i;

    return next();

    } else {

    printData();

    }

    }

    return next();

    }

  12. MaxOnMaxxer90

    If you use this in an app, you would get a CROSS-origin read blocking error. How can you resolve this error and use this nice tutorial in a real app? Thanks 👍

  13. Jack Bird

    awesome tutorial, thank you for your video. it is very clear and easy. Also as newby in Shopify eCommerce i am using https://e-scraper.com/shopify/ to scrape shopify stores, all product data from my supplier sites and other sources. It helps me a lot. maybe it helps somebody too.

    Thank you for your input!!!

  14. Alex Zajacek

    wow this is so awsome. thank you for the tutorial. have you done one that shows you how request data from login sessions?

  15. DANK FAM

    request is deprecated. make a new video

  16. Fazal Mehmood

    Scrap data from websites where you have to perform several clicks and then scrap data. This is child play what you are showing : )

  17. Business Opportunity in Ruvol

    I have invented a Board Game [still unpublished and not yet out in the market] that I believe is guaranteed to be as challenging and exciting as CHESS. I called it “RUVOL.”

    It is my hope that one day Ruvol may surpass chess as the “Number One Board Game in the World.”

    The weakness of chess is it always starts in fixed positions that the opening moves become “memorizable.” In fact, not a few have so mastered the moves that they can play against their opponents “blindfolded.” It is for this very reason that the great Bobby Fischer introduced his so-called “Fischer Random Chess,” where the starting position of the pieces is “randomized” to make the memorization of openings impracticable. Fortunately, it is also for this reason that I invented Ruvol where “every game” has been calculated to be a challenging one to play.

    HOW IS RUVOL PLAYED and HOW YOU CAN MONETIZE IT?

    I detailed everything in my YouTube video. Here is the link: https://www.youtube.com/watch?v=jcqth0m3-R0

    BIG MONEY POTENTIAL IN RUVOL!

    It is worthwhile to note that the people who play chess will be the same people who will play Ruvol. In my Google search, I learned there are around 800 million chess players in the world. Even just a small percentage of these 800 million is good enough to earn big money from Ruvol either as an ONLINE GAME BUSINESS or as a PHYSICAL PRODUCT DISTRIBUTOR.

    You may contact me at: rodolfovitangcol@gmail.com.

    Thanks and God bless!

    RODOLFO MARTIN VITANGCOL

    The Ruvol Inventor

  18. Ayoub OMARI

    how we can get all network responses of the website?

  19. ֆօս թճv

    YouTube: I'll recommend this cool stuff after 3 years

  20. Adrian Walker

    I tried, but it got too advanced too quickly. I'm off to find a beginner tutorial on this subject.

Leave a Reply