Bilal K.

Bilal K.

The Complete Beginner’s Guide to Web Scraping

The Complete Beginner’s Guide to Web Scraping

Subscribe to my newsletter and never miss my upcoming articles

Python is so powerful!

Web scraping allows us to extract the public data from website so that we can use it for some purpose.

Usually, we can only view the downloaded web page but what if you want to automate or you want to extract the TSLA stock price every other minute? Yes, it is possible to do it manually but this way is too time-consuming, not scalable, and highly not recommended. That’s where web scraping comes to help us gather and use valuable data that isn’t always available via APIs.

Picture Credits: Real PythonPicture Credits: Real Python

In this post, we’ll learn how to use BeautifulSoup & Requests to perform a quick and effective web-scraping for TSLA Stock Price Alert, an email will be automatically sent to notify the price has been now dropped. Let’s dive in.

1-Import The Necessary Libraries

from bs4 import BeautifulSoup
import requests
from requests_html import HTMLSession
import smtplib
from smtplib import SMTP

2-Loading Webpages with request

The requests module helps to send HTTP requests in Python. So the next step is to inspect the required data classes from the complete source code. We select the card and click on the ‘Inspect Element’ option to get the source code of that particular section. You will get something similar to this:

Inspecting titleInspecting title

Inspecting PriceInspecting Price

Now we need to write a function that could scrap the title and price from the webpage.

  • Get the contents of the following URL using the requests module: **https://finance.yahoo.com/quote/TSLA/**

  • Find the title and price using class the name inspected from the webpage

  • Convert the string price to float

  • Call the function to send an email

def check_price():
    page = requests.get("https://finance.yahoo.com/quote/TSLA/")  #target url to scrap
    soup = BeautifulSoup(page.content, "lxml")
    title = soup.find(class_ = "D(ib) Fz(18px)").get_text()
    price= soup.find(class_ = "Trsdu(0.3s) Fw(b) Fz(36px) Mb(-4px) D(ib)").get_text()
    floatprice=float(price[0:5])

    print ("Title: ",title)
    print("Price:", floatprice)
    if (floatprice < 400):
        print("Hello World")
        sendemail1()

3- Sending Emails in a standard format

  • Establish the SMTP connection at port 587

  • Start TLS based SMTP Session

  • Give the secret credentials along with the to and from email address.

  • Set Email Parameters

  • Terminate the SMTP Session

def sendemail1():
    try:
        server= smtplib.SMTP('smtp.gmail.com:587')
        server.ehlo()
        server.starttls()
        server.ehlo()
        server.login('lalbikhan2014@gmail.com', 'Your App password here') #From email here 
        subject="Tesla Price Alert"
        body="Tesla Stock Price just hit the targeted price. Thanks! "
        msg = f"Subject:{subject}\n\n{body}"
        server.sendmail( 'lalbikhan2014@gmail.com',
        'bilal.khan2014@hotmail.com',
        msg)
    except:
        print ("Error, Email not sent")

    print ('\nEmail sent success\n')
    server.quit()

check_price()

The complete code is available **here**. Feel free to check it out.

So that’s it! I hope you found it helpful, and also I hope that you have a nice day.

Till next time :) Happy Learning.

Bilal Khan.

 
Share this