Learn how to do basic web scraping using Node.js in this tutorial. The request-promise and cheerio libraries are used.
💻 Github: https://github.com/beaucarnes/fcc-project-tutorials/blob/master/node-web-scraping/index.js
🔗 Article on client-side web scraping : https://medium.freecodecamp.org/client-side-web-scraping-with-javascript-using-jquery-and-regex-5b57a271cb86
🐦 Beau Carnes on Twitter: https://twitter.com/carnesbeau
—
Learn to code for free and get a developer job: https://www.freecodecamp.com
Read hundreds of articles on technology: https://medium.freecodecamp.com
And subscribe for new programming videos every day: https://youtube.com/subscription_center?add_user=freecodecamp
source
I am having this error when I am trying to do npm install
Can anyone know why is this caused or how to solve this.
Help required
npm ERR! path /Users/FinalProject/node_modules/jquery
npm ERR! code ELOOP
npm ERR! errno -62
npm ERR! syscall access
npm ERR! ELOOP: too many symbolic links encountered, access '/Users/FinalProject/node_modules/jquery'
Thanks in million.
Thanks a bunch for the tutorial! You explained it clearly & helped me in understanding this tutorial 👌
UPDATE:
When I tried to run this code, I got an error that says:
"TypeError: data.directory_items is not iterable"
How do I fix this?
Someone know why he uses a network request link to get all users but go directly on the profile link to get challenges data? Isn't the challenges data on the profile page also dynamic?
I make this tuto on February 15, 2020 by chance. Funny, isn't it ?
don't work
can we call the function next() at the end of the function getChallenges…() instead of returning it?
Really nice job. I like your teaching style. Hope to see more videos from you.
Does the link within video used not work anymore?
Do we need to use some sort of headers?
I have a question. This method will work on the angular and react?
GREAT!!!!!
Is there any way for you to allow the "index.js" to run when you click a button?
and then you could run some statistical test to find if there's a correlation between num_hearts and num_challenges_passed
WEBSITE HAVE CHANGED STRUCTURE… Here is how I fixed it (without completed challanges)..
First change table structure:
let table = new Table({
head: ['username', '❤️'],
colWidths: [15, 15]
})
Then change function to:
function getChallengesCompletedAndPushToUserArray(userData) {
var i = 0;
function next() {
if (i < userData.length) {
process.stdout.write(`.`);
table.push([userData[i].name, userData[i].likes_received]);
++i;
return next();
} else {
printData();
}
}
return next();
}
If you use this in an app, you would get a CROSS-origin read blocking error. How can you resolve this error and use this nice tutorial in a real app? Thanks 👍
awesome tutorial, thank you for your video. it is very clear and easy. Also as newby in Shopify eCommerce i am using https://e-scraper.com/shopify/ to scrape shopify stores, all product data from my supplier sites and other sources. It helps me a lot. maybe it helps somebody too.
Thank you for your input!!!
Open growing community on web scraping
Join here, guys –
Web Scraping & Automation (Python/Javascript)
https://www.facebook.com/groups/348468202946051
wow this is so awsome. thank you for the tutorial. have you done one that shows you how request data from login sessions?
request is deprecated. make a new video
Scrap data from websites where you have to perform several clicks and then scrap data. This is child play what you are showing : )
Business Opportunity in Ruvol
I have invented a Board Game [still unpublished and not yet out in the market] that I believe is guaranteed to be as challenging and exciting as CHESS. I called it “RUVOL.”
It is my hope that one day Ruvol may surpass chess as the “Number One Board Game in the World.”
The weakness of chess is it always starts in fixed positions that the opening moves become “memorizable.” In fact, not a few have so mastered the moves that they can play against their opponents “blindfolded.” It is for this very reason that the great Bobby Fischer introduced his so-called “Fischer Random Chess,” where the starting position of the pieces is “randomized” to make the memorization of openings impracticable. Fortunately, it is also for this reason that I invented Ruvol where “every game” has been calculated to be a challenging one to play.
HOW IS RUVOL PLAYED and HOW YOU CAN MONETIZE IT?
I detailed everything in my YouTube video. Here is the link: https://www.youtube.com/watch?v=jcqth0m3-R0
BIG MONEY POTENTIAL IN RUVOL!
It is worthwhile to note that the people who play chess will be the same people who will play Ruvol. In my Google search, I learned there are around 800 million chess players in the world. Even just a small percentage of these 800 million is good enough to earn big money from Ruvol either as an ONLINE GAME BUSINESS or as a PHYSICAL PRODUCT DISTRIBUTOR.
You may contact me at: rodolfovitangcol@gmail.com.
Thanks and God bless!
RODOLFO MARTIN VITANGCOL
The Ruvol Inventor
And how can we get data inside shadow root?
Good tutorial but keep running code after each function
how we can get all network responses of the website?
Better idea
Is node js scraping is like python scrapy? which one better performe?
YouTube: I'll recommend this cool stuff after 3 years
I tried, but it got too advanced too quickly. I'm off to find a beginner tutorial on this subject.
Really cool