Click here to Skip to main content
13,146,365 members (43,971 online)
Click here to Skip to main content
Add your own
alternative version

Tagged as

Stats

19K views
12 bookmarked
Posted 19 Aug 2013

Creating a service for storing your Pocket articles as PDFs

, 19 Aug 2013
Rate this:
Please Sign up or sign in to vote.
A small project for creating a service which downloads web articles saved to Pocket for later reading as PDFs.

Introduction

I'm currently using a system where I handle my news flow by browsing and reading articles in Feedly and the ones I feel I want to keep for future reference, I save to my Pocket account.

Having them saved in Pocket is great for passing on links to friends and so on, but after having built up quite a large reference library of articles in Pocket, it's hard to easily find the articles I'm looking for, and it's also hard to search for something in all my referenced material at once.

To solve this, I've built a Windows service that uses the Pocket API to download a list of all my saved links, and from that list, it downloads all new entries as PDFs and saves them to my local computer. That way, I end up with all the articles as searchable PDFs, which I then can organize and cite using my favorite citation program. 

Using the code

Prerequisites

Before you begin using the code, you need to create your own Pocket Application and create an access_token for your Pocket user. Since this is more of a hack than a completed project, I'll just guide you through how you can do it without coding.

Here are the steps to create your needed keys: 

  1. Create a Pocket application.
  2. This is a simple step. All you need to do is to go to this page and enter the needed information. When that is done, you will get your own application "consumer key", which you will use later to authorize your user for your new application.

  3. Create your access_token.
  4. This takes a bit more hacking. You can save the following text to an HTML-file and follow the steps there to get your access_token. It is documented here on the Pocket developers site, and basically you are doing the steps with a bit of cheating with the callbacks.

<html><head></head><body>
1. Enter your consumer_key and hit "Submit" and you should get a response saying "code=XXXX"
<form action="http://getpocket.com/v3/oauth/request" method="post">Consumer_key<input type="text" 

name="consumer_key" value =""><input type="hidden" name="redirect_uri" 

value="fobar://test"><input type="submit" value="Get code"></form>
2. Go to the following link and accept the application authorization request.
<p>Go to https://getpocket.com/auth/authorize?request_token=&lt;code from step 1&gt;&amp;redirect_uri=fobar://test</p>
3. Enter your consumer_key and the code from step 1&gt; Push submit and you should get a response with "access_token=YYYYY"
<form action="http://getpocket.com/v3/oauth/authorize" method="post">Consumer_key<input 

type="text" name="consumer_key" value =""> Code<input type="text" 

name="code" value =""><input type="submit" value="Authorize"></form>
</body></html>  

Now you've got your access_token and your consumer_key, which are used in the service to download your Pocket information. 

You also need to download the iTextsharp library since it is referenced in the code. 

General logic 

The general idea of this project is to periodically get a list of articles (URLs) saved to a specific Pocket account and compare this list to a stored list of articles already downloaded by the service. If there are any new articles, these are downloaded and converted to PDF format and then saved to a designated directory. After the article has been downloaded, the stored list of articles downloaded by the service is updated with the new entry. 

Connecting to Pocket

If you've created your keys correctly, the task of obtaining a list of all your Pocket articles/URLs is pretty straight forward. Everything you need is documented at this page, and the section in my code is shown below.

using (WebClient test = new WebClient())
{
    System.Collections.Specialized.NameValueCollection reqparm = 
      new System.Collections.Specialized.NameValueCollection();
    reqparm.Add("consumer_key", consumer_key);
    reqparm.Add("access_token", access_token);
    reqparm.Add("state", "all");
    reqparm.Add("detailType", "complete");
    byte[] responsebytes = test.UploadValues("https://getpocket.com/v3/get", "POST", reqparm);
    string responsebody = Encoding.UTF8.GetString(responsebytes);
...
}

This piece of code will retrieve a JSON list of information about the given Pocket account, containing information about each saved URL in Pocket, including tags, dates added and so on.

From this JSON text, I use a regular expression to extract the information that I need, rather than using a more sophisticated JSON parser.

MatchCollection matches = Regex.Matches(responsebody, 
  "\"given_title\":\"(.*?)\".*?\"resolved_title\
  ":\"(.*?)\".*?\"resolved_url\":\"(.*?)\"");  

And then I add each URL to a list for comparison with the saved list of already fetch articles. That list, by the way, is saved in an XML file and serialized/deserialized with an XmlSerializer object. 

Two ways of creating PDFs 

Now, for each URL not present in the articles already downloaded list, a PDF will be created in the designated directory.

I started out using WKHTMLToPDF  to create the PDFs, but I found that the resulting quality was not quite as good as the online supplier PDFCrowd that I had been using before, so I hacked a small routine for using that site as well. Doing the latter though, I'm pretty sure I might be in some violation of the user agreement, but since I really do low volume conversion, I hope it will be all right. 

Using  WKHTMLToPDF

This approach is pretty straight forward. I just start a new process with the command line options I want to use for the application (make sure you've installed the program and that your app.config is pointing to the right path!) and let it do the rest. Afterwards, I just check the output from the program to see if it finished without problems. 

System.Diagnostics.ProcessStartInfo procStartInfo =
    new System.Diagnostics.ProcessStartInfo("\""+ wkhtmlToPDFExePath + "\"");
procStartInfo.Arguments = "--load-error-handling ignore --zoom 1.33 --javascript-delay 5000 \"" + url + 
  "\" \"" + tempFileName + "\"";

Using PDFCrowd 

Using PDFCrowd involves filling in your account username and password in the app.config file, and then simulating a web surfing session to create and download the PDF. There is some details in the code that you can see for yourself on how the session is simulated.

One part that I should mention though is the using of a CookieAwareWebClient class that will use cookies to match the server session of  the client, mainly for the login status. 

Adding PDF meta data 

As a small addition, I use iTextsharp to add some meta data to the PDF so that it is easier to import to document management  systems. I first wanted to add keywords as well to the file - and keep these synced with Pocket tags - but since my DMS does not support that feature, I kind of left it not implemented. I do add the source URL, the web page as Author and the title as title in the PDF meta data though.

private static void AddMetadataToPDF(string filePath, string url, 
        string author, string title, string keywords)
{
    string tempFileName = System.IO.Path.GetTempFileName();
    PdfReader pdfReader = new PdfReader(filePath);
    pdfReader.SelectPages("1-" + pdfReader.NumberOfPages);
    using (PdfStamper stamper = 
      new PdfStamper(pdfReader, new FileStream(tempFileName, FileMode.Create)))
    {
        Dictionary<String, String> info = new Dictionary<string,string>();
        info.Add("Keywords", keywords);
        info.Add("Title", title);
        info.Add("Author", author);
        info.Add("OriginalUrl", url);
        info.Add("CreationDate", "D:" + 
          DateTime.Now.ToString("yyyyMMddHHmmss") + "-01'01'");
        stamper.MoreInfo = info;
        stamper.Close();
    }
    pdfReader.Close();
    File.Delete(filePath);
    File.Move(tempFileName, filePath);
}

Settings 

The app.config file holds configurationFile, logFile, downloadFolder, interval, accessToken, consumerKey, createUsingPDFCrowd, pdfCrowdUserName, pdfCrowdPassword, createUsingWKHTMLToPDF, wkhtmlToPDFExePath. All these are pretty self explanatory, but you need to set them right in order for the program to work.

The configuration.xml file pointed to by the entry configurationFile also needs to be an xml file with the following content to work. This is the file that will hold the list of downloaded URLs by the way. 

<?xml version="1.0" encoding="utf-8"?>
<ArrayOfDownloadedUrl xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 

          xmlns:xsd="http://www.w3.org/2001/XMLSchema">
</ArrayOfDownloadedUrl> 

Error handling 

This is more of a "proof of concept" kind of project than a release candidate, so error handling is bad at best :)

If you want to get it in some kind of production state, you really need to take some steps to make it more error resilient.

Considering that this is a windows service with no user interface and no way of communicating with the user/administrator than by logging of different sorts, you really need to make it fail proof.  

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Martin-Hallonqvist
Software Developer (Senior)
Sweden Sweden
I've been working in the industry since 1998 in many different roles and in many different environments and systems. Most of my work has been done on the Microsoft platform though, and I'm currently working professionally in VB6 (facepalm...) and C# .Net 4.0.

Please see my LinkedIn profile for more information.

You may also be interested in...

Pro
Pro

Comments and Discussions

 
QuestionRegarding project Pin
Member 110592605-Sep-14 9:54
memberMember 110592605-Sep-14 9:54 
GeneralMy vote of 4 Pin
fredatcodeproject19-Aug-13 4:24
memberfredatcodeproject19-Aug-13 4:24 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Terms of Use | Mobile
Web03 | 2.8.170915.1 | Last Updated 19 Aug 2013
Article Copyright 2013 by Martin-Hallonqvist
Everything else Copyright © CodeProject, 1999-2017
Layout: fixed | fluid