How to get all links of any website using Python

This code will extract all links form the website and will show to you

Requirements

1)    beautifulsoup4
2)    requests

Steps to install requirements

1) open cmd as admin
2) Type pip install beautifulsoup4
3) wait for installation
4) after completion of installation type pip install requests
5) wait for installation completion

Code

import requests
from bs4 import BeautifulSoup
url = "https://www.python.org"

links = []
website = requests.get(url)
website_text = website.text
soup = BeautifulSoup(website_text)
for link in soup.findAll('a'):
    links.append(link.get('href'))
for link in links:
    if not link:
        pass
    elif "http" in link :
        print(link)
    else:
        print(url+link)
print(len(links))


Comment down queries below if any are there

Comments

Popular posts from this blog

Flutter the upcoming generation software development kit which uses Dart language

Visual Studio Code - the code editor for Python with great features