Click here to Skip to main content
15,169,185 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
A question about scraping information, with python bs4, from an international e-commerce marketplace.

I’m __scraping__ the site: https://www.thedetoxmarket.com/products/organic-lavender-soap?_pos=2&_sid=fc64927ad&_ss=r

This is a site that changes the prices depending on the country one selects as being in. If I select (as an example) Austria or Venezuela, the prices will be different on the site - but not in the scrape. When I want to get the price information, no matter what country I put as my location, it scrapes the prices for the US location.

Here is the code, that i wrote for the scrape:

Python
price = soup.find('div', class_ = 'apercu-medium-19 pb-3 mt-3 mb-1' ).text.strip() 
print(price)


Here is the code from the site:

HTML
<div class="price d-none d-lg-block">
                        <div class="text-left">
                          <div class="apercu-medium-19 pb-3 mt-3 mb-1" style="visibility: visible;">
                            €‌8.95
                            
                          </div>
                        </div>
                      </div>

How can I scrape in such a way that the prices will reflect the location I selected?

What I have tried:

<pre lang="Python">price = soup.find('div', class_ = 'apercu-medium-19 pb-3 mt-3 mb-1' ).text.strip() 
print(price)
Posted
Comments
Richard MacCutchan 31-Jul-21 11:19am
   
The problem with most of these sites is that the changes are done by Javascript, based on user selections. So your scraper code, will first need to select the country you are interested in to force the page to update itself.
Margarita Usova 31-Jul-21 12:40pm
   
Thank you so much! How can I add a country selection to my code?
Richard MacCutchan 31-Jul-21 12:54pm
   
It depends on how the web page is built. There is no standard way to do this.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)




CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900