I need assistance with web scraping.

Question

1.00/5 (1 vote)

See more:

Hello there,

I'm new to data science, and I've been attempting to scrape data from a website using Python and BeautifulSoup, as described in this article, but I keep encountering an issue that I can't seem to fix. I'm using the following code:

import requests
from bs4 import BeautifulSoup

url = 'https://example.com'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')

data = soup.find('div', class_='info').text
print(data)

However, when I execute the code, I see the following error message:

AttributeError: 'NoneType' object has no attribute 'text'

What I have tried:

I double-checked the HTML of the website, and the element with class "info" is clearly present. I have no idea what is producing this problem or how to repair it. Any assistance would be much appreciated!

Posted 25-Jul-23 0:27am

Syed Mubashshir Hasan

Updated 25-Jul-23 2:11am

Add a Solution

Comments

Graeme_Grant 25-Jul-23 6:31am

Page scrapping is not allowed on most websites. Do they have an API?

[no name] 25-Jul-23 6:56am

data var is an array ( you can have 0, 1, lot of values return). you need a loop and indexes to retrieve value by ".text"

1 solution

Add a Solution

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Richard MacCutchan · Answer 1 · 2023-07-25T02:07:00

Python

data = soup.find('div', class_='info').text

The message is telling you that soup.find did not find a div with that class type, so the find result is None. So when you try to access the text attribute it raises the AttributeError exception. And if you go to Example Domain[^] and inspect the page source you can see that there is no such div.

I need assistance with web scraping.

1 solution

Solution 1

Add your solution here

Preview 0

I need assistance with web scraping.

1 solution

Solution 1

Add your solution here

Preview 0

Existing Members

...or Join us