Click here to Skip to main content
15,908,020 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
Beautifulsoup runs for sometime then.

Traceback (most recent call last):

    c_name = info_box.find('dt', text='Contact Person:').find_next_sibling('dd').text
AttributeError: 'NoneType' object has no attribute 'find_next_sibling'


What I have tried:

Python
import pandas as pd
import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
}

big_list = []
for i in range(1, 50):

    url= f"https://www.usaopps.com/government_contractors/naics-111110-Soybean-Farming.{i}.htm"
    r = requests.get(url, headers=headers)
    soup = BeautifulSoup(r.text, 'html.parser')
    for x in soup.select('div.list-one')[:3]:
        det_url = 'https://www.usaopps.com' + x.select_one('a').get('href')
       # print(det_url)
        req = requests.get(det_url)
        det_soup = BeautifulSoup(req.text, 'html.parser')
        info_box = det_soup.select_one('div.info-gen-box')
        c_name = info_box.find('dt', text='Contact Person:').find_next_sibling('dd').text
        try:
            c_fax = info_box.find('dt', text='Fax:').find_next_sibling('dd').text
        except AttributeError as error:
            print(error)
        else:
            print(c_name, c_fax)
        big_list.append((c_name, c_fax))

df = pd.DataFrame(big_list, columns = ['Contact', 'Fax'])
Posted
Updated 20-Aug-22 21:23pm
v2

Python
c_name = info_box.find('dt', text='Contact Person:').find_next_sibling('dd').text

The message is telling you that info_box.find did not find anythings, so it returned None. And a None object does not have any properties or methods, so you cannot call find_next_sibling on it. When you use a method that may fail you should always check its return value before trying to use it in a pipeline.
 
Share this answer
 
When ever you get a problems that involves a message such as "'nonetype' object has no attribute ..." it means the same thing: you have tried to call a method on something that doesn't exist.

If you cann find and there is no such item, it returns a special value: None
If you try to do anything with that value, you will get this error.

It's a bit like a pocket: you have a pocket in your shirt, which you use to hold a pen. If you reach into the pocket and find there isn't a pen there, you can't sign your name on a piece of paper - and you will get very funny looks if you try! The empty pocket is giving you a null value (no pen here!) so you can't do anything that you would normally do once you retrieved your pen. Why is it empty? That's the question - it may be that you forgot to pick up your pen when you left the house this morning, or possibly you left the pen in the pocket of yesterday's shirt when you took it off last night.

We can't tell, because we weren't there, and even more importantly, we can't even see your shirt, much less what is in the pocket!

Back to computers, and you have done the same thing, somehow - and we can't see your code, much less run it and find out what contains null when it shouldn't.
But you can - and the debugger will help you here. Run your program in the debugger and when it fails, it will show you the line it found the problem on. You can then start looking at the various parts of it to see what value is null and start looking back through your code to find out why. So put a breakpoint at the beginning of the method containing the error line, and run your program from the start again. This time, the debugger will stop before the error, and let you examine what is going on by stepping through the code looking at your values.

But we can't do that - we don't have your code, we don't know how to use it if we did have it, we don't have your data. So try it - and see how much information you can find out!
 
Share this answer
 

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900