How do I fix : attributeerror: 'nonetype' object has no attribute 'find_next_sibling'

Question

0.00/5 (No votes)

See more:

Beautifulsoup runs for sometime then.

Traceback (most recent call last):

    c_name = info_box.find('dt', text='Contact Person:').find_next_sibling('dd').text
AttributeError: 'NoneType' object has no attribute 'find_next_sibling'

What I have tried:

Python

import pandas as pd
import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
}

big_list = []
for i in range(1, 50):

    url= f"https://www.usaopps.com/government_contractors/naics-111110-Soybean-Farming.{i}.htm"
    r = requests.get(url, headers=headers)
    soup = BeautifulSoup(r.text, 'html.parser')
    for x in soup.select('div.list-one')[:3]:
        det_url = 'https://www.usaopps.com' + x.select_one('a').get('href')
       # print(det_url)
        req = requests.get(det_url)
        det_soup = BeautifulSoup(req.text, 'html.parser')
        info_box = det_soup.select_one('div.info-gen-box')
        c_name = info_box.find('dt', text='Contact Person:').find_next_sibling('dd').text
        try:
            c_fax = info_box.find('dt', text='Fax:').find_next_sibling('dd').text
        except AttributeError as error:
            print(error)
        else:
            print(c_name, c_fax)
        big_list.append((c_name, c_fax))

df = pd.DataFrame(big_list, columns = ['Contact', 'Fax'])

Posted 20-Aug-22 15:08pm

Life Unexplained

Updated 20-Aug-22 21:23pm

PIEBALDconsult

v2

Add a Solution

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

OriginalGriff · Answer 1 · 2022-08-20T18:58:00

When ever you get a problems that involves a message such as "'nonetype' object has no attribute ..." it means the same thing: you have tried to call a method on something that doesn't exist.

If you cann find and there is no such item, it returns a special value: None
If you try to do anything with that value, you will get this error.

It's a bit like a pocket: you have a pocket in your shirt, which you use to hold a pen. If you reach into the pocket and find there isn't a pen there, you can't sign your name on a piece of paper - and you will get very funny looks if you try! The empty pocket is giving you a null value (no pen here!) so you can't do anything that you would normally do once you retrieved your pen. Why is it empty? That's the question - it may be that you forgot to pick up your pen when you left the house this morning, or possibly you left the pen in the pocket of yesterday's shirt when you took it off last night.

We can't tell, because we weren't there, and even more importantly, we can't even see your shirt, much less what is in the pocket!

Back to computers, and you have done the same thing, somehow - and we can't see your code, much less run it and find out what contains null when it shouldn't.
But you can - and the debugger will help you here. Run your program in the debugger and when it fails, it will show you the line it found the problem on. You can then start looking at the various parts of it to see what value is null and start looking back through your code to find out why. So put a breakpoint at the beginning of the method containing the error line, and run your program from the start again. This time, the debugger will stop before the error, and let you examine what is going on by stepping through the code looking at your values.

But we can't do that - we don't have your code, we don't know how to use it if we did have it, we don't have your data. So try it - and see how much information you can find out!

Richard MacCutchan · Answer 2 · 2022-08-20T21:23:00

Python

c_name = info_box.find('dt', text='Contact Person:').find_next_sibling('dd').text

The message is telling you that info_box.find did not find anythings, so it returned None. And a None object does not have any properties or methods, so you cannot call find_next_sibling on it. When you use a method that may fail you should always check its return value before trying to use it in a pipeline.

How do I fix : attributeerror: 'nonetype' object has no attribute 'find_next_sibling'

2 solutions

Solution 1

Solution 2

Add your solution here

Preview 0