DaniWeb API Developer Competition - Page 3

Ok, thats cool. I vote for myself. There's one vote. Am I winning. LOL.

I am going to retire from creating my entry for this competition.

I feel the API was designed mainly with web developers in mind and there are arguments on the net regarding avoiding OAuth, for example, and it also is just nasty for native application development. Where people like Ketsuekiame have enough knowledge to get around its implications on native applications.

I also don't understand the exact use for the API with its web orientation considering we all browse the site using the web anyway so surely the main uses of the API would fall to native applications allowing the user site functionality from their desktop instead of a browser.

That's just my opinion on the current state of play, was looking forward to entering this competition.

I also don't understand the exact use for the API with its web orientation considering we all browse the site using the web anyway so surely the main uses of the API would fall to native applications allowing the user site functionality from their desktop instead of a browser.

You make a fair point here, but we are able to create mashups, e.g. tie in daniweb members with their twitter feed for example. But the API is a bit limited.

Perhaps I am a bit biased, but I really don't find the API very limiting. Then again, I have experience working with other APIs for social media websites, such as Facebook's and Twitter's, which both are OAuth based as well.

OAuth was chosen because it makes the most sense for who we are. We're a social networking website targeting IT professionals and developers. OAuth 2.0 has evolved into somewhat of a standard for the APIs for social networking websites targeting a similar audience to that of DaniWeb, such as Stack Overflow and Github. The latest version of vBulletin also supports OAuth 2.0 for administrative tasks.

I, personally, have no real life experience creating non-web based applications. One of the first things I did before deciding on OAuth was meeting with my friend Matt, who is the one creating an iOS DaniWeb app. I asked him if it was possible to use OAuth to create native applications for desktop/mobile, and he said absolutely and that is what most of his clients actually use, and what they use at his fulltime job (he primarily does mobile development both fulltime and with clients on the side). He showed me a handful of examples in which some used an embedded browser to do the OAuth, and others actually embedded the login natively where you wouldn't even know it used an http protocol. Put it this way: he convinced me.

My opinion was solidified upon upgrading to Windows 8 and finding that Microsoft actually uses the embedded browser method in their integration with Facebook / LinkedIn with their native mail/calendar/people apps. If it's good enough for Microsoft, it's good enough for me! OK, I take that back, but you get the idea.

You should keep in mind that you only need to authenticate the user once. Once you have the code from the authorization process, you should keep it (either as a registry setting or in a local file) and use it to fetch a new access token behind the scenes whenever your existing access token expires.

Therefore, you can write native applications that have an embedded browser to log the person in from within your aplication's setup process or preferences section. Then, the application does not need to ever deal with logging or requesting authorization again in any future uses of the program.

The API works with any language that supports HTTP requests, including C++, Java, C#, etc.

Here are some examples of native applications that can use the DaniWeb API:

• Browser toolbar that gives realtime notification about PMs, watched article updates, etc. regardless of where you are on the web (for those of you who don't want to wait the hour for the email updates)
• Realtime latest articles feed (either standalone or integrated into mail client such as in the form of an Outlook plugin)
• IRC client that keeps track of members logged into DaniWeb.com and matches them up with IRC nicknames logged into our IRC server to show who's currently online
• Greasemonkey script that modifies DaniWeb interface to suit your liking based on information you can fetch from the API
• Stats generator that processes post/article/member history and generates pretty charts of how many people register a day, visit a day, how many new articles/posts are created a day/week/month/etc.
• A moderation tool that scans recent posts for words/phrases that are potentially spam and then flags them

The API is robust enough in that it gives you access to every behind-the-scenes function that the DaniWeb website makes use of. Using the API, you can theoretically rewrite all read functionality of the website as a standalone application, mobile application, or perhaps your own website that presents the information differently.

The purpose of the DaniWeb.com website is to present all information from our database in as concise a way as possible, trying to avoid excessive bloat while striking a balance with a good user experience. However, I understand that my idea of a perfect user experience isn't necessarily your idea of one. Therefore, the API basically gives you all of the backend functionality that was used to create daniweb.com with the option of creating your own website that uses it, or perhaps just expands its featureset in ways that we've never thought of, or don't want to due to not enough widespread interest or just to cut down on too many bells and whistles for simplicity's sake.

Fair response, Apple are the only ones with easy native integration from what I've been told.

Did try the embedded browser authentication myself but ran into difficulties on the second call, when using the browser to do it, to OAuth with the application code and stopped.

Also tried making a POST request for this stage using the RestSharp library for C#, however just got an internal server error (500) when I did so.

IRC client that keeps track of members logged into DaniWeb.com and matches them up with IRC nicknames logged into our IRC server to show who's currently online

I was looking at developing that very thing, but where do we get the logged-in users data? I couldn't see how the api could give that info.

But the API is a bit limited.

Perhaps I am a bit biased, but I really don't find the API very limiting.

Just hit me when I tried to search for particular data. Not having a go - I was just trying to get at data that wasn't possible throught the API interface, e.g. a traditional SQL INNER JOIN :)

I couldn't see how the api could give that info.

IIRC Dani answered this somewhere.

I couldn't see how the api could give that info.

IIRC Dani answered this somewhere.

I think it was get members and sort by last active?

Fair response, Apple are the only ones with easy native integration from what I've been told.

I don't have any mobile developer experience either way to be able to compare. However, OAuth is incredibly popular nowadays, as it powers the APIs for Google, Facebook, Twitter, etc. so I can't imagine that the Android market would shoot themselves in the foot like that.

Did try the embedded browser authentication myself but ran into difficulties on the second call, when using the browser to do it, to OAuth with the application code and stopped.

I only have web development experience, but Matt showed me examples that he wrote using OAuth that both used the embedded browser method and that didn't. Additionally, I've of course seen the embedded browser method in practice many times (i.e. Windows 8 with Facebook and LinkedIn integration) so it's obviously doable. We follow the exact same flow as Facebook.

Just hit me when I tried to search for particular data.

As I've mentioned, the API opens up all of the backend model-layer functionality (MVC model) that the DaniWeb website uses. We are incapable ourselves of performing search queries, which is why we outsource our search functionality to Google. The API allows you to do anything we are able to do.

Not having a go - I was just trying to get at data that wasn't possible throught the API interface, e.g. a traditional SQL INNER JOIN :)

That's the point of the API: to create functionality that doesn't currently exist in core DaniWeb. If native DaniWeb doesn't have a need to do the particular INNER JOIN query, then it's not going to be a native feature of our API either. Otherwise, our native application would be so bloated with every possible, potential combination of query that could ever be thought up for one reason or another. However, that doesn't mean that you can't access our content and perform the query on your end. Scrape our API and reproduce the portions of our database that you will need for your application. Then query your own partially recreated database. It's doable and the fact that you need to do the processing on your end instead of just querying us for an instant response is what would make your application valuable.

The API is NOT designed to give you a means to query our database for various combinations of resultsets. It's designed to give you the building blocks for you to build your own invention on top of.

Also tried making a POST request for this stage using the RestSharp library for C#, however just got an internal server error (500) when I did so.

Unfortunately I don't know C#, but perhaps you can find an OAuth library in C# for Facebook, Stack Overflow, Github, etc., and modify it to use DaniWeb instead.

I was looking at developing that very thing, but where do we get the logged-in users data? I couldn't see how the api could give that info.

Fetch a list of members from http://www.daniweb.com/api/members and they will be sorted based on last logged in date.

That's the point of the API: to create functionality that doesn't currently exist in core DaniWeb. If native DaniWeb doesn't have a need to do the particular INNER JOIN query, then it's not going to be a native feature of our API either. Otherwise, our native application would be so bloated with every possible, potential combination of query that could ever be thought up for one reason or another. However, that doesn't mean that you can't access our content and perform the query on your end. Scrape our API and reproduce the portions of our database that you will need for your application. Then query your own partially recreated database. It's doable and the fact that you need to do the processing on your end instead of just querying us for an instant response is what would make your application valuable.
The API is NOT designed to give you a means to query our database for various combinations of resultsets. It's designed to give you the building blocks for you to build your own invention on top of.

I realise that - and appreciate it. I've been toiling with MongoDB on this very issue - I was looking to enter this into the competition, but found the fiddly nature of setting up a NoSQL a pain as I wanted to provide a downloadable class - just plug and go. So, yes while it's doable, it will take more time than I have currently available. But it may be something I'll come back to.

Please don't take my one line comment as a criticism. 'Limited' was not a pop at the API - as I said - more to do with my attempts to wangle data out of it that it wasn't designed to give (as you mention).

WRT to currently logged in members - I wasn't aware of that so thanks for the heads up. I'll certainly have a go at incorporating it into the VB.NET irc thingy I've been playing with. But it won't be ready for the competition :( UNfortunately, looking at a web-based solution using an embedded irc just won't
cut it. I've been looking at websockets to create a poor version of an irc client too, without luck so far. BUT it's all good learning. Having fun, although I've little to show for it so far. :)

'Limited' was not a pop at the API - as I said - more to do with my attempts to wangle data out of it that it wasn't designed to give (as you mention).

On the contrary, the API was designed to give the flexibility to do nearly everything conceivable to the imagination. However, in order to offer that level of flexibility, we can only provide you with the building blocks and then leave it up to the developer to put them together in an infinite number of different combinations. The API would be much more limiting IMHO if we provided more "polished" data as opposed to just the raw building blocks.

You can do a lot with a bundle of logs, your imagination, and some skills but your choices are a lot more limited when they're already all precut into 2'x4' planks. Sure, it might seem a lot less daunting interlocking some two-by-fours to build something simple, as opposed to staring blankly at a giant heap of lumber, but ultimately the flexibility and infinite potential is no longer there.

http://www.daniweb.com/community-center/geeks-lounge/threads/451027/daniweb-api-developer-competition#post1952715

I would just like to add to this.

OAuth is a great tool to allow website <--> website authentication without having to create handfuls of accounts. I believe this was the original reason it was created. So, as far as being able to log into other websites with Daniweb, that choose to be affiliated with Daniweb, mission accomplished :)

There's an aspect of this process that is assumed: The "man-in-the-middle" is a trusted, neutrally affiliated application. In this case, the browser.

The moment you embed a browser into your application you have broken this trust-chain because it is affiliated with you (the application/developer)

Although the application may be trusted, which we assume so because you're about to give me your username and password, this brings to a null point the reason behind oauth; not giving me your username and password. This is why I don't like OAuth as a mobile developer. The amount of hoops you have to jump through to maintain both a good user experience and the assumed OAuth trust chain, quite frankly, ridiculous.

This brings me on to another point. Whilst I agree that "this is how it's done" nowadays, don't be a lemming :P Just because everyone else embeds the browser and [has the ability to] steal the user credentials doesn't mean we all should.

Finally, and this is more appropriate to Daniweb itself, the OAuth authentication as it's written doesn't follow the correct flow when trying to authenticate natively. Specifically, there is no standard result page, nor JSON/XML result where the redirect_uri is missing or some optional scheme that we can use for mobile/native development. Without this, we have to jump through all the hoops mentioned above.

As in my scenario:

Mobile App [Unauth] -> Kick to Browser -> Log In -> Redirect to App website -> Call App WebService -> Store Token -> Redirect to Daniweb for access_token -> Redirect to App website -> Call App webservice -> Store AuthCode -> Redirect to registered schema [w/ GetTokenParam] -> Mobile App is now in focus -> Call App webservice -> Retrieve token -> Authenticate with Token -> Mobile App [Auth]

Why use all the services in the middle? Well, that's for my own security. If someone decompiles (pretty easy in .net) my application, there goes my secret key! Makes it easy to reverse engineer and allow someone to pretend they are me in an app.

Now, why I don't embed a browser? This brings me on to Mike's point.

In .NET (Including Mobile Development and WindowsRT) the .NET embedded web browser cannot detect or capture 302 REDIRECT. Once you make the call the OAuth, DW redirects you using a 302 header to...well, in a native app this is typically no-where. You would need your own website somewhere that could return a result to your browser that you could capture.

It is possible to hack around this, you need to create your own embedded browser from the base control (COM). But all that work to do a simple OAuth call? You're almost making a markettable product in the process ;)

So why is this possible on iOS? Simply because their control is better. Additionally, the cookies from the standard mobile browser are shared with the embedded one. This simply isn't the case in Windows. I can't speak for Android as I don't know. I also presume that the mobile developer you spoke to use custom schemes in the redirect_url to call straight back into your app. It's this that I'm trying to avoid in order to stay true to the way OAuth was intended to authenticate. I've completed the OAuth part of my application so this is no longer a problem for me. Just thought you might want to know why native developers are getting frustrated.

It's very difficult to do it properly, but not impossible.

This brings me on to another point. Whilst I agree that "this is how it's done" nowadays, don't be a lemming :P Just because everyone else embeds the browser and [has the ability to] steal the user credentials doesn't mean we all should.

While such deficiencies may exist for using OAuth in non-web based mediums, I believe this to be a case where I can't just choose to "not be a lemming". Sometimes you have to do things the way that your audience expects them to be done in order to ensure the desired reach.

Finally, and this is more appropriate to Daniweb itself, the OAuth authentication as it's written doesn't follow the correct flow when trying to authenticate natively. Specifically, there is no standard result page, nor JSON/XML result where the redirect_uri is missing or some optional scheme that we can use for mobile/native development. Without this, we have to jump through all the hoops mentioned above.

Can you please explain what you mean by this? From what I understood from the implementations of Facebook, Github, Stack Overflow, and all of the other implementations I tried my best to imitate (for the convenience of the developer being able to use a single library to connect to all of them), the standard behavior is for an invalid request http error to be passed if a required parameter (such as redirect_uri) is missing.

I also presume that the mobile developer you spoke to use custom schemes in the redirect_url to call straight back into your app. It's this that I'm trying to avoid in order to stay true to the way OAuth was intended to authenticate.

He showed me finished projects for iOS that both used embedded browsers and that didn't. They all looked very well-polished from the end-user standpoint although I have no idea what went on behind the scenes. He said he finally has the navigation working and is now working on the non-OAuth requests before playing around with OAuth, so he cannot comment about DaniWeb's specific implementation yet. I'm looking forward to his feedback since he has so much experience working with OAuth with a wide assortment of APIs.

Also, for those who missed the notice, the code snippet competition is being extended to the end of August for submissions

The facebook API has it's own SDK for C# so that handles its OAuth through their own custom classes, can't say exactly how they do it.

While such deficiencies may exist for using OAuth in non-web based mediums, I believe this to be a case where I can't just choose to "not be a lemming". Sometimes you have to do things the way that your audience expects them to be done in order to ensure the desired reach.

Not actually directed at you, but native app developers.

Can you please explain what you mean by this? From what I understood from the implementations of Facebook, Github, Stack Overflow, and all of the other implementations I tried my best to imitate (for the convenience of the developer being able to use a single library to connect to all of them), the standard behavior is for an invalid request http error to be passed if a required parameter (such as redirect_uri) is missing.

In OAuth 2.0 the redirection URI is an optional parameter as described by the RFC: http://tools.ietf.org/html/draft-ietf-oauth-v2-31#section-4.2.1

In the case where a redirection uri is missing, the server should respond appropriately, but this schema is for you to define and we to follow.

Additionally, even with server side authentication where a client_id/secret is used, the Auth server should still require that the registered redirection uri match the request, where it is not absent.

I completely understand and have no trouble believing that your friend has multiple OAuth projects and has no issue embedding the browser or not doing so (in which case the implementation may vary depending on the schema used) however, embedding the browser is certainly against the intention of OAuth as the entire idea behind it is so that you don't enter your credentials anywhere inside the application/website you want to allow access.

Microsoft Windows is a trusted third party (as it is an operating system) which is actually accounted for in the document (and why they are allowed to embed a browser that uses the username/password credentials)

The resource owner password credentials grant type is suitable in
cases where the resource owner has a trust relationship with the
client, such as the device operating system or a highly privileged
application. The authorization server should take special care when
enabling this grant type, and only allow it when other flows are not
viable.

I'm not really calling for you to change it, just offering reasons as to why the native devs are crying foul.

In OAuth 2.0 the redirection URI is an optional parameter as described by the RFC: http://tools.ietf.org/html/draft-ietf-oauth-v2-31#section-4.2.1

Can you please point me to where in the RFC document it says what the appropriate behavior is for the API if a redirect_uri is not present. I see where it says it's optional, but the server essentially is meant to return a redirect with the values needed (code and access_token) tacked onto the query string. If a redirect_uri is not provided, how should the API respond with the values??

Unfortunately, there's nothing in the spec about it :(

This Page on the wiki lists a few ways native apps can be supported in the absence of it being included in the specification.

Personally, I think the redirection_uri should be a requirement, as you've done it. The absence of the redirection_uri parameter would normally mean you use the registered redirection_uri for the application. Where a redirection_uri is included it needs to be matched with the registered one, which currently doesn't happen (unless it's client-side authentication). This is all to stop redirection hijack :)

I don't particularly want to get into an argument with you; I'm happy enough with what is available that I can make it work fairly easily. My problem is with OAuth itself, not your implementation of it ;)

I don't particularly want to get into an argument with you

I wasn't trying to argue. On the contrary, I trust your advice when you say I'm not implementing the spec perfectly (when I thought I was), and so I was asking for the documentation of how it should be changed.

Also, for those who missed the notice, the code snippet competition is being extended to the end of August for submissions

:(

Why the sad face?

Fair enough; I know I can come over a bit aggressive when I get "factual" so wanted to make sure that you weren't misinterpreting my actions.

Hello and sorry for my bad english,
i want to participate in the contest doing a iPhone native app, has the application be available in the App Store? if not, how you can valorate it?
Thanks

Someone else is already doing an iPhone native app, but it is still heavily in the works. I'm also not sure if he's going to submit to the competition.

Not sure what you mean by "valorate"?? Do you mean evaluate? If so, it's a work in progress and probably won't be released until after the competition is over.

yes, i mean evaluate, sorry, but my question is, how i have to send the app for evalue it?

http://www.idev101.com/code/Distribution/checklist.html

I currently suck at web-based stuff, but I have been wanting a modified interface for awhile, I am talking with a friend of mine who is very into web-based stuff to help me out, but my main concern is that neither of us is experienced enough to get my idea working well fast enough. I am wondering if anybody can describe the process by which I could make a PC program (just an exe file) that could access the data on daniweb, present it differently (IE: I want to highlight any post in which the last post was made by the creator) then edit it differently (IE: a built-in editor that could execute simple code examples to test and ensure that they work, and could allow for custom syntax highlighting). I would know how to make such a program if all the daniweb data was kept on a hard-drive on the users machine, but obviously it isn't. From what I understand an API is designed to allow similar access to the data that is NOT on the hard drive but rather here on daniweb. Unfortunately when it comes to the internet I never know what to do. (I would like if I could access the API functionality in c++/WinAPI, but java or similar would work as well) If it can't be done easily via a c-style language HTML would work too, I could implement my idea as a website without taking a major hit, but again, it would be less than ideal.

I would know how to make such a program if all the daniweb data was kept on a hard-drive on the users machine, but obviously it isn't.

All you need to do is scrape our API and duplicate its contents on your local hdd in a format that you are able to parse and work with. Would that work?

Everything that you're suggesting is definitely 100% doable, but unfortunately I don't have any experience with non-web based languages to be able to give you a hand.