[ad_1]
@codejediCode_Jedi
Python, Machine Learning, Web-Scraping, Web-Automation and more …
Hi, developer friends
In this tutorial, I will show you how to mine the latest Tesla stock prices using Node.js and Puppeteer.
Let’s get started!
First, you will need to install the puppeteer using
npm i puppeteer. Now if you don’t
npm,
package.json, and
node_modules setup, here is a great tutorial.
After installing puppeteer, create a new javascript file and require puppeteer on the first line:
const puppeteer = require('puppeteer');Then create the async function in which we will write our main code:
const puppeteer = require('puppeteer');
async function start() {
}
start();We are now ready to start scratching.
First, you need to launch a new browser instance, as well as define the url your web-scraper will visit:
const puppeteer = require('puppeteer');
async function start() {
const url = 'https://finance.yahoo.com/quote/TSLA?p=TSLA&.tsrc=fin-srch';
const browser = await puppeteer.launch({
headless: false
});
}Then you need to call the
newPage()function to open a new page in the browser, and go to the url we defined using the
goto()function:
const puppeteer = require('puppeteer');
async function start() {
const url = 'https://finance.yahoo.com/quote/TSLA?p=TSLA&.tsrc=fin-srch';
const browser = await puppeteer.launch({
headless: false
});
const page = await browser.newPage();
await page.goto(url);
}For this next step, you will need to go to https://finance.yahoo.com/quote/TSLA?p=TSLA&.tsrc=fin-srch, right click on the current share price, then click on inspect:
A pop-up will appear to the right of your window, you will need to find the price of action item:
Then you will need to right click on the stock price item and click on ‘copy full xpath’.
This will give us a way to access the share price element:
Once we have the Xpath of the price of action item, we can add these 3 lines of code into our function:
var element = await page.waitForXPath("put the stock price Xpath here")
var price = await page.evaluate(element => element.textContent, element);
console.log(price);The
page.waitForXPath() function will locate the price element of the stock.
Then the
page.evaluateThe function will get the textual content of the stock’s price element which would then be printed by the
console.log()function.
At this point, your code should look like this:
const puppeteer = require('puppeteer');
async function start() {
const url = 'https://finance.yahoo.com/quote/TSLA?p=TSLA&.tsrc=fin-srch';
const browser = await puppeteer.launch({
headless: false
});
const page = await browser.newPage();
await page.goto(url);
var element = await page.waitForXPath("/html/body/div[1]/div/div/div[1]/div/div[2]/div/div/div[5]/div/div/div/div[3]/div[1]/div[1]/span[1]")
var price = await page.evaluate(element => element.textContent, element);
console.log(price);
}
start()If you were to run your current code, you would find that by going to the url you set earlier, a pop-up window will appear:
To work around this problem, plug these 2 lines of code into your function before defining the “element” variable:
var accept = ("#consent-page > div > div > div > form > div.wizard-body > div.actions.couple > button");
await page.click(accept)
This will locate the “Accept All” button and click on it to make the pop-up disappear.
Now you will have a working function that goes to your defined URL, scratches the latest Tesla share price, and prints it to your terminal.
To go further, you can put these lines of code in a for loop:
for(var k = 1; k < 2000; k++){
var element = await page.waitForXPath("/html/body/div[1]/div/div/div[1]/div/div[2]/div/div/div[5]/div/div/div/div[3]/div[1]/div[1]/span[1]")
var price = await page.evaluate(element => element.textContent, element);
console.log(price);
await page.waitForTimeout(1000);
}The
page.waitForTimeout(1000)The function will wait 1000 milliseconds (1 second) before repeating the for loop.
And finally,
add a browser.close()function after the for loop to close the browser and finish executing your code:
const puppeteer = require('puppeteer');
async function start() {
const url = 'https://finance.yahoo.com/quote/TSLA?p=TSLA&.tsrc=fin-srch';
const browser = await puppeteer.launch({
headless: false
});
const page = await browser.newPage();
await page.goto(url);
var accept = ("#consent-page > div > div > div > form > div.wizard-body > div.actions.couple > button");
await page.click(accept);
for(var k = 1; k < 2000; k++){
var element = await page.waitForXPath("/html/body/div[1]/div/div/div[1]/div/div[2]/div/div/div[5]/div/div/div/div[3]/div[1]/div[1]/span[1]");
var price = await page.evaluate(element => element.textContent, element);
console.log(price);
await page.waitForTimeout(1000);
}
browser.close();
}
start();Get the hottest lineup: Tweets, Reddit posts, Github Rests, and memes of the week straight to your inbox every Friday via my newsletter!
Bye
Als published here.
Key words
Create your free account to unlock your personalized reading experience.
[ad_2]
