I need assistance with web scraping.

Question

1.00/5 (1 vote)

See more:

Hello there,

I'm new to data science, and I've been attempting to scrape data from a website using Python and BeautifulSoup, as described in this article, but I keep encountering an issue that I can't seem to fix. I'm using the following code:

import requests
from bs4 import BeautifulSoup

url = 'https://example.com'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')

data = soup.find('div', class_='info').text
print(data)

However, when I execute the code, I see the following error message:

AttributeError: 'NoneType' object has no attribute 'text'

What I have tried:

I double-checked the HTML of the website, and the element with class "info" is clearly present. I have no idea what is producing this problem or how to repair it. Any assistance would be much appreciated!

Posted 25-Jul-23 0:27am

Syed Mubashshir Hasan

Updated 25-Jul-23 2:11am

Add a Solution

Comments

Graeme_Grant 25-Jul-23 6:31am

Page scrapping is not allowed on most websites. Do they have an API?

Member 15627495 25-Jul-23 6:56am

data var is an array ( you can have 0, 1, lot of values return). you need a loop and indexes to retrieve value by ".text"

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Richard MacCutchan · Answer 1 · 2023-07-25T02:07:00

Python

data = soup.find('div', class_='info').text

The message is telling you that soup.find did not find a div with that class type, so the find result is None. So when you try to access the text attribute it raises the AttributeError exception. And if you go to Example Domain[^] and inspect the page source you can see that there is no such div.