Recently I am using cURL to crawl the recommended question list from stackoverflow.com based on my signedin account.
Currently, in order to register my account info in the HTTP request, I copy and paste the headers+cookies(displayed by my browser) into a file, wrote a script in which curl is used to crawl the data.
However, as I noticed, some keys in the Cookie field, like the __utma and __utmb, sent by the browser regularly changes, thus I have to update my cookie value in the script accordingly, otherwise, I will not be able to get the result based on my account information.
This necessary update on my side is quite cumbersome indeed.
My confusion are:
1, why is the cookie field(sent by the browser) regularly changing? What is the reason behind that?
2, Is there anyway to make CURL behave like the browser so that it can adapt the cookie in the same way as the browser?
Submit an article or tip