Attributeerror: 'nonetype' object has no attribute 'get_text' when webscraping

Question

0.00/5 (No votes)

See more:

The aim of this code is o scrape data from Search Jobs | CareerJunction[^] and allow the user to input the job title they are looking for and for it to display the job title with its attributes like salary, date posted,etc.

import requests
from bs4 import BeautifulSoup

def job_details(job_title):
    base_url="https://www.careerjunction.co.za/"
    search_url="https://www.careerjunction.co.za/jobs/results?keywords=&autosuggestEndpoint=%2Fautosuggest&location=0&category=&btnSubmit="
    job_search=f"{search_url}, keywords={job_title.replace(' ','%20')}"
    
    headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"}
    search_url_response=requests.get(search_url, headers=headers)
    if search_url_response.status_code != 200:
        print(f"Unable to fetch page. Status code{search_url_response}")
        return
    
    search_content=search_url_response.text
    soup=BeautifulSoup(search_content,"html.parser")

    jobs_c1=soup.find_all("div", class_="job-result-title")
    for jobs_c1 in jobs_c1:
        title=jobs_c1.find("h2", class_="href").get_text(strip=True) 
        recruiter_name=jobs_c1.find("h3", class_="href").get_text(strip=True) 

    jobs_c2=soup.find_all("div", class_="job-result-overview cjun-serp")
    for jobs_c2 in jobs_c2:
        salary=jobs_c2.find("ul", class_="salary").get_text(strip=True) 
        position=jobs_c2.find("ul", class_="position").get_text(strip=True) 
        location=jobs_c2.find("ul", class_="location").get_text(strip=True)
        date_posted=jobs_c2.find("ul", class_="updated-time").get_text(strip=True)

    print(f"Job Title: {title}")
    print(f"Recruiter name: {recruiter_name}")
    print(f"Salary: {salary}")
    print(f"Position: {position}")
    print(f"Location: {location}")
    print(f"Date Posted: {date_posted}")


job_title=input(" Enter the job you are looking for: ")
job_details(job_title)

What I have tried:

I've tried alterting the get_text to gettext and getText but still the code isnt running and i get an attribute error
The aim of this code is o scrape data from Search Jobs | CareerJunction[^] and allow the user to input the job title they are looking for and for it to display the job title with its attributes like salary, date posted,etc.

Posted 24-Jun-24 4:43am

Lumko Mtengwane

Add a Solution

2 solutions

Add a Solution

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Pete O'Hanlon · Answer 1 · 2024-06-24T06:47:00

The problem isn't that you're using get_text (or whatever function). The problem is that the code has is returning a NoneType (Python's version of null). In other words, you have failed to find a particular element on the web page, and you are attempting to get the text from something that doesn't exist. You should check to see if the value is None before attempting to access the element

Python

titleItem=jobs_c1.find("h2", class_="href")
if title != None
    title = titleItem.get_text(strip = True)

Repeat this through your code as necessary.

Richard MacCutchan · Answer 2 · 2024-06-24T06:49:00

You have not explained where the error occurs, but I assume it is on this line:

Python

title=jobs_c1.find("h2", class_="href").get_text(strip=True)

The error message tells you that the call to find did not return any data, so there is no text to be got. You should always check the response/return codes for method calls, before trying to use what you think may be returned. Do not just assume that such calls always succeed.

Attributeerror: 'nonetype' object has no attribute 'get_text' when webscraping

2 solutions

Solution 1

Solution 2

Add your solution here

Preview 0

Attributeerror: 'nonetype' object has no attribute 'get_text' when webscraping

2 solutions

Solution 1

Solution 2

Add your solution here

Preview 0

Existing Members

...or Join us