(python 3.5)
What I am trying to do is scrape the social club site for public statistics but I can't get the login down.
I have looked for other posts with answers but nothing quite fits the bill. I don't want to use selenium or mechanize or any browser. I am trying to do this with requests and beautifulsoup. This is my first post so please also let me know if I am doing anything incorrectly/wrong.
My code compiles fine no syntax errors or anything like that. It just won't login. I have found all the post information and I get the token but I just can't get it to login. The login always fails. I know this is probably pretty basic stuff so if someone could help me out that would be great! Thanks. If i had to guess I would say it still has something to do with the authentication token in the post.
This is a test account i made so don't worry about me showing the password ect. It's fine
This is the post data when you sign in
login=smittyofwar@mail.com
password=Harambe1
__RequestVerificationToken=5xNBfVT5mZcHNd8e8Bt2pFqzPLxc36QYASAo3koVjDh-wBTJLT6rD5Yaq39mPbq_Q5vDrUcOZ-E0MzAzXrgW7qFRy-Y1
import requests
from bs4 import BeautifulSoup
s = requests.session()
login_url = "http://socialclub.rockstargames.com/profile/signin"
tmp=s.get("http://socialclub.rockstargames.com")
findtoken = BeautifulSoup(tmp.content, "lxml")
token=findtoken.find("__RequestVerificationToken")
username = 'smittyofwar@mail.com'
passwrd='Harambe1'
data = dict(login=username, password=passwrd,__RequestVerificationToken=token, next='/')
s.post(login_url, data=data,headers=dict(Referer="http://socialclub.rockstargames.com"))
r = s.get("http://socialclub.rockstargames.com")
soup = BeautifulSoup(r.content, "lxml")
soup = soup.prettify()
with open('thesauce.txt', 'w', encoding='utf-8') as f:
for line in soup:
f.write(str(line))
What I have tried:
I have looked over countless other questions and researched through google. Also tried different code for hours.