|
A couple of things have stalled this
1. We were flat out on other things, but those things are now done so we have time again
2. We've been battling the issue of how we protect the site against API overload and abuse. Lots and lots of ways to handle this, including requiring an API consumer to register and get a key, to throttling API requests, to building a separate cluster specifically to handle API calls.
Would registering your application in order to get an API key be too onerous? Is request throttling per API key a potential issue (assuming that we'd look at each on a case-by-case with a view to helping, not hindering)
cheers
Chris Maunder
|
|
|
|
|
Chris Maunder wrote: Would registering your application in order to get an API key be too onerous?
That would be a perfectly reasonable idea.
Chris Maunder wrote: Is request throttling per API key a potential issue
That would really depend on how popular your app got, but I wouldn't say it would be too burdensome. Potentially, if your app gets that popular, I'm sure that some enlightened chap wouldn't have an issue handing ownership of said app over to CP.
|
|
|
|
|
Whatever works for you should be fine. I'll be one of the first devs to write and publish an app or two if you get an API released 
|
|
|
|
|
The application itself wouldbn't be appropriate because there could be many simultaneous users.
I thought this would be more about reputation points, or article status. Those are the only real categories of data that I think would be of interest to users. Maybe throttling and/or access in general could somehow be tied to the user's reputation points, MVP status, or some other identifiable metric. This would also involve the user registering for access, and having them click thru an agreement not to misuse/abuse the system in the process of using the data.
The rep points consists of nothing more than less than a dozen category name and a points value, and article status would be stuff displayed in the My Articles page (again, not a lot of data for 99.9% of the users here).
It might even be worth setting up a database of user-specific tables for people that actually use the APIs, and run a Sql SSIS package to populate it every night.
Lots of approaches...
As for registering an app, I wouldn't be opposed, but you'd probably have to generate an assembly for us to add to our app that would negotiate the authorization handshake and that doesn't involve a data file being stored on the user's computer.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013
|
|
|
|
|
John Simmons / outlaw programmer wrote: The application itself wouldbn't be appropriate because there could be many simultaneous users.
Apps don't necessarily have user ID info sent and so an app would be the most appropriate registrant to an API, if registration made sense. Alternatively the API client request could include an Application Name that's sent as part of the request, but this obviously allows spoofing: A successful and popular app could be black-banned because someone else spoofed calls by that app to overload our servers. Sending a private key instead is better.
In general it's probably not going to be an issue. However, when something does become an issue, and you didn't put in plans, the issue can become far messier than if you'd just put in mechanisms in the first place.
Sounds like you're fine with whatever we do as long as it's halfway sensible. We're approaching, in our old age, the half way mark in sensible so we should be good.
cheers
Chris Maunder
|
|
|
|
|
I just had another thought - How about this:
0) A user runs an app that gets data from CP
1) The act of requesting that data causes the server to save the date/time of the request, the user's CP id, and the app id in the database.
2) The server creates a XML (or JSON) file for that user with the requesting app(s) data
3) For a period of N days (where N is determined by CP admins), the server automatically updates that file with that info every night at time convenient for the system.
4) The next time the app makes a request, the datetime is updated for that user/app, and the file is returned to him (since it already exists and is updated).
5) If the file expires (older than N days), the service can update it before returning the filename.
6) When the nightly job runs, expired files can be deleted (or marked for deletion).
The only thing that CP would have to do is provide the assembly that handles that negotiation, and the application would have to make all API calls via that assembly.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013
|
|
|
|
|
You're describing a simple caching mechanism.
That's the easy part.
It's DDoS attacks against the API that are the issue.
cheers
Chris Maunder
|
|
|
|
|
Well, the DDoS attack will be aiimed at either the web service or the cached file the app is trying to download. Either way, it's not really affecting access to the database. You can also limit the number of accesses per hour for a requesting IP, and limit the number of simultaneous connections.
Will all the programmers at your immediate disposal that are willing to help, I'm pretty sure it wouldn't take too gawd-awful long to come up with an effective and effecient plan.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013
|
|
|
|
|
One issue is that you can't just hand out a key for an application that is part of an article on the site, because then it's in the open. You have to provide a mechanism (a page on the site?) that allows a user to requesst and receive his own personal API key. At that point, the app in question would need to include a method by which the user can specify his api key so the app can access the API.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013
|
|
|
|
|
An application key would be something that the application developer requests once and then bakes into their app. The app end-user wouldn't know (or need to know) anything about the key. We then ensure all API calls are done over SSL and the key is (mostly) safe from being sniffed and copied by others.
cheers
Chris Maunder
|
|
|
|
|
Chris Maunder wrote: It's DDoS attacks against the API that are the issue
Well, if it's REST based, whatever mechanisms your web server already has against DDOS should protect the API as well, wouldn't it?
|
|
|
|
|
How so?
cheers
Chris Maunder
|
|
|
|
|
They are all GET/POST calls over HTTP. So if you already have DDOS protection that will stop someone from repeatedly GET-ing core pages on your website, wouldn't that same protection prevent them from making repeated hits against API URLs? Add some kind of vendor-key and that's a 2nd layer of protection.
|
|
|
|
|
DDoS protection at a HTTP level is done outside of the application and generally blocks all requests from a given IP. Keys allow us to block an application that is (maliciously or not) sending too many requests without blocking traffic from other users on that IP (big issue for large corporations, universities or certain countries.
cheers
Chris Maunder
|
|
|
|
|
Well, if app-author specific keys will help you safeguard against attacks, that sounds like the best plan. Initially you could expose it to just a few trusted folks, and later expand it to include more app-vendors and authors. Looking forward to more info/updates on this
|
|
|
|
|
IMHO, you have no chance to stand a DoS attack without a kind of API key - best to give per application registration...Who wish not register isn't serious...
I'm not questioning your powers of observation; I'm merely remarking upon the paradox of asking a masked man who he is. (V)
|
|
|
|
|
My latest blog at http://jakemdrew.wordpress.com/ has not been consumed. The blog page says it has been polled, but the new article has not shown up on code project. The blog has been assigned with the CodeProject category, and I have not had had this problem with other articles in the past. 
Thanks,
Jake Drew
www.jakemdrew.com
|
|
|
|
|
Hmmm. Have you tried changing your feed to include the full entry? I can't think of why it wouldn't be a problem before and is now, though. Let's just see what happens.
Thanks,
Sean Ewington
CodeProject
|
|
|
|
|
I apologize. I am not sure what you mean? Is this something I do on wordpress or codeproject?
|
|
|
|
|
From your Wordpress.com menu, under Settings ->, then -> Reading, you will see "For each article in a feed, show" then make sure "Full text" is selected.
Thanks,
Sean Ewington
CodeProject
|
|
|
|
|
I did as you suggested yesterday and updated my wordpress reading settings to "full text". My blog has still not been consumed.
Thanks for all of your help on this!
|
|
|
|
|
Hmmm. Could you please try adding a rel tag to the article as shown here?
Code Project Technical Blog FAQ[^]
Maybe the aggregator is grumpy with wordpress.com right now.
Thanks,
Sean Ewington
CodeProject
|
|
|
|
|
Done! Hopefully, this helps. Thanks!
|
|
|
|
|
I think that didit. Success, my friend
Thanks,
Sean Ewington
CodeProject
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
