Javascript Graphing

I recently had a requirement to create a proof-of-concept "what-if" calculator for a new product I'm involved in which will launch in the first quarter of next year.

Now, I prefer to write web-based applications because I hate supporting multiple operating systems and client versions (yes, I'm aware that different browsers behave differently, but there are a number of toolkits out there that solve most of those problems). However, this calculator required some pretty graphs (for non-techie types) and I didn't want to go to the effort of writing and deploying a server-based application for a simple calculator.

I went searching and I found there has been some work recently in the realm of browser-based graphics (either SVG or the Canvas tag) but, as with most new things, different browsers handled different things, well, differently. So I searched a little more and found a library called PlotKit which not only wrapped up the different graphics libraries (through a Google's ExplorerCanvas), but it also provides a ready-to-use graphing library.

Web-based development is becoming increasingly more powerful and I think we will continue to see the trend towards web-based applications continuing. Long live the browser! 

JotSpot Backpeddles

Well, it appears that after my little discussion with Ryan this morning JotSpot has decided to update their FAQ (not yet present on their website).

 The new FAQ for anyone looking at using JotSpot's affiliate program is:

Q:  Where can I run your ads?

A:

You can run our ads on websites or emails that you own and control. We recommend that you put our ads on blogs and other websites you own.

You are NOT permitted to purchase advertising that competes with JotSpot in an effort to drive traffic to your affiliate pages. In other words, do not buy ads on Google or Yahoo if JotSpot also bids on those same ads.

If you wish to purchase an ad when the ad does not compete with JotSpot's ad, you may NOT use JotSpot in the ad copy, and you may NOT send traffic directly to our landing page.

Of course, after my experience, I wouldn't suggest using JotSpot until they come up with a real affiliate agreement written by real lawyers with real specifics. 

I thought I liked JotSpot

Every six months or so I usually come across a product that has an affiliate program and decide to have a go at trying to monetize it via online advertising media. Usually, this flutter ends up costing me more money than I ever earn but I end up learning more and more about online marketing and advertising in the process.

This year I came across JotSpot and, with an affiliate program paying $1.00 per lead and the first month's sign-up value (up to $199.95) per sale and after spending some time playing with their products I thought I found a product worth marketing again.

 After reading through their, admittedly, rather limited FAQs (listed below in case they change sometime in the future) I thought I was on a winner:

 Affiliate Program Help

Q:      How do I log into my account?
A:      Just click here.
 
Q:      How much can I earn through the affiliate program?
A:      There's no set limit on what you can earn through the program. We'll pay you $25 for the first person you get to try JotSpot, and after that our standard compensation is to pay $1 per lead. Also, every time you refer someone who becomes a paying customer for JotSpot we pay a commission equal to the monthly rate for the referred subscriber. If your referral pays for the company plan, we pay you $199.95. If your referral pays for the Mini plan, we pay you $9.95.
 
Q:      How often will I be paid?
A:      You can request payment after you have earned $50 through the program. You may request payment again for each $25 earned. We make payments during the first week of each month.
 
Q:      How will I be paid?
A:      At this time the only payment method is Paypal.
 
Q:      Where can I run your ads?
A:      Wherever you like. We recommend that you put our ads on blogs and other websites.
 
Q:      How do I contact someone about the affiliate program?
A:      Contact us via email.

So, after signing up for JotSpot's affiliate program, I went on my merry way creating advertising campaigns on Google, Adbrite and FeedBurner. Out of all the advertising campaigns I've done in the past, this one was definitely shaping up to be the most effective and the amount of clickthroughs sent within 24 hours outranked anything I've done in the past. Until I received the following emails from their adveritsing manager

Hi, Wayne.  I see that you just sent us a number of leads -- how are you
promoting JotSpot?

I just want to make sure everything is legit before awarding credit to your
account.

Thanks,
Ryan

--
Ryan Pollock
Product Manager, Customer Acquisition
JotSpot, Inc.
(650) 323-3225 x 128
email: ryan@jot.com
blog: www.ryanpollock.com

I thought this was nice of Ryan to introduce himself and responsible of JotSpot to ensure that the clickthrough/leads generated are indeed legit (a lot of online affiliate programs suffer from click/signup robots). So I sent this response:

Hi Ryan

I am promoting JotSpot through a series of online advertising campaigns with a number of sites including Google Adwords and some sites on Adbrite (although these ads are yet to start running).

I look forward to a long-lasting relationship with JotSpot.

Kind regards

Wayne Robinson

Within minutes I received the following three emails in quick succession from Ryan:

Exactly what ads are you buying?

We don’t allow you to buy ads that compete with anything that we purchase.

Thanks,
Ryan

-------------------

Are you buying words like “wiki software” and sending them to your affiliate link?

I’m sorry – but this is not permitted and please stop immediately.

Thanks,
Ryan

-------------------

Wayne,

I will approve all leads up til this point, but I can’t approve anymore.

Thanks,
Ryan

Now, I was both worried and a little bit pissed off - especially with how panicy Ryan seems in these emails. After all, the FAQs say you can advertise wherever you want and I just booked a number of advertising spots with other providers (OK, so I decided to have a significant flutter on advertising). This is the first time an affiliate has ever asked me not to spend money marketing on their behalf!

Also too, I'm now concerned about the stats in my affiliate account as I've sent through 425 clicks (from targetted advertising describing the product) and there are only 2 leads showing (free trial signups). Quite frankly  a 0.24% conversion of clicks to free-trial signups is abysmal for untargetted advertising, let alone targetted advertising so I'm starting to wonder how legit JotSpot actually is themselves.

I've corresponded a little more with Ryan but, while he responds almost instantly (and frequently) to me sending clicks to JotSpot, he has chosen to ignore my most important question, "is the stats on the affiliate page an accurate representation of the leads I've sent to JotSpot, or have you put my targetted advertising leads on-hold and for what purpose?"

Normally I would decide to just leave this issue alone at this point. It is their business, they can run it how they like and piss off who they like, but their website didn't tell me I wasn't allowed to do what I am doing and I've already spent over AU$850 on advertising with another US$400 in the pipelines waiting on approvals and availability (did I mention big flutter).

I will keep you all posted with how JotSpot decides to handle this one. 

 Oh, and by the way, there are no other terms and conditions. The FAQ is as specific as it gets.

Just Ask!

Well, it seems that if you wait... just long enough, you may just get what you wished for.

I was just finishing my implementation of a scalable push solution for real-time web-based applications when I was asked to review an implementation for a reader of my blog.

The project isn't officially launched yet, but it is to be called Juggernaut (http://juggernaut.rubyforge.org) and it looks really promising. Juggernaut uses Flash and it's sockets implementation to push data from subscribed channels directly into your application.

I will post the review when I've finished doing the load-testing of the server (as much as possible on my limited hardware) and, of course, get the author's permission to do so.

Argh... sickness

Sometimes I think the world is conspiring against me. Only a day after announcing I would get the second article released on the server-push topic, I come down with the flu.

Now I don't know how the common cold affects you, but when I'm sick, everything arround me seems fuzzy. I have difficulty concentrating on any one task and my ability to form medium-to-long term memory seems to leave me.

 Oh well, I've almost recovered now, so I will make the same committment now as I did last week. I shall have the companion article finished by the end of the week for you all to see. In the meantime, why don't you head on over to Alex's blog where he is putting the finishing touches on a Flash sockets implementation of server-push for when you don't want to push JavaScript to the end of your standards-compliant website or wrap all your data in JavaScript.

I'm not dead yet

Hey everyone. Sorry for not posting in a while (wow, it's been over a month). I promise I haven't abandoned yet another blog, it's just been crazy of late working both on StaffLocation and my day job over at QPFL (nb: I'm not the web designer).

StaffLocation is definitely starting to come together and has inspired quite a long list of articles that I need to blog about. StaffLocation started as a simple In/Out status board but has evolved into areas I never thought possible for a web application. I know, I know... I should just just 'get real' and release a beta for everyone to play with, but there are still some kinks in the interaction with some external services and it's real-time nature that need ironing out before everyone gets to play, so stay tuned (you can register for updates on the home page).

As for QPFL well, it's getting close to the end of the Australian financial year and everyone is racing to get their product applications in by 30 June. It's a real buzz and quite exhausing to be in the office this time of year. Everything needs to be done yesterday (or last month) and everyone's racing around to make sure we get everything done whilst complying with the multitude of Australian financial services regulations.

Anyway, must get back to work. I promise to have part 2 of the push/pull article up by the end of the week.

Efficiency of HTTP Push Vs Pull

I'm working on a new application (StaffLocation.com) that requires the ability to stream live data from a server to the client web application via JavaScript. There are two methods that I could approach this.

The first (and simplest) involves polling a script on the server every x-number of seconds to see if the data has changed. This is the traditional way most web applications retrieve status updates from the server but it has two nasty side affects:

1. Every active client sends a request to the server every x-seconds whether there is changed data or not. Therefore, if there are 1,000 users on the website and you want the data to be updated every second, then those users will generate 1,000 hits to your application back-end (and that's 1,000 times whatever database reads/writes you do per hit).
2. Every hit uses bandwidth whether it contains data or not. A blank request/response will still contain many hundreds of bytes. Therefore, using the example of 1,000 active users polling every second at a 200 byte payload we would be using 200KB/s (1.2mbps - almost 500GB per month) of bandwidth.

Of course this situation could be improved by decreasing the frequency of polls, and some empirical testing with end-users of our prototypes show that anything below 4 seconds for an update feels instantaneous. However, bandwidth for this type of solution is still using 300kbps (125GB per month) for 1,000 users. Also, advanced caching techniques will be required as there is no way MySQL can handle the authorization & status checks required for much more than 1,000 hits per second. Maybe it's time to investigate other options.

The other type of client/server communication is via server-push. So, instead of the client requesting the server for a new status with 90% of the responses being, "NOT YET!!" The client connects once to the server and the server sends the clients updates on what the client is listening for. At this point I hear you all telling me that HTTP and the web just doesn't work this way. It is a PULL-only based technology. Well, some of you might be surprised to know that server-PUSH functionality has existed since Netscape 1.1 in the form of the content-type multipart/x-mixed-replace. The trouble is, Internet Explorer doesn't seem to support this content-type anymore (it did in 3.0) and it is very hard to use with JavaScript. However, there are ways in which JavaScript may be pushed to the client using modern browsers.

Now, the trouble with most web servers is (especially when utilizing dynamic content creation) that they don't really like pushing data to the browser without caching it first (large static files are the exception). Luckily for me, it is relatively easy today to roll your own web server. For this proof of concept I shall be using Ruby and WEBrick.

Of course, WEBrick by default follows the same pattern of, "lets cache all the data before sending it to the client." Luckily, due to the object-oriented nature of Ruby, that assumption is very easy to override. As a proof of concept, I've created a web server that dynamically adds lines to the page every 3 seconds with the use of JavaScript. One of the tricky things to remember is that the browser is expecting regular data. If it doesn't receive it within it's timeout window (usually 60 seconds), then it will think the connection has been closed. This example sends a single space character every second as a KeepAlive packet. The code is a little long, so you can download it here (push_server.rb).

Now, of course, this doesn't solve the problem of working out when there are new statuses to update the client with, but it does provide an example of a highly efficient server that can easily handle tens of thousands of connections (with the correct file-descriptor permissions on your server) with very little load when the data-set isn't changing. Also, it reduces the bandwidth usage down to the KeepAlive packets (1 byte per 10 seconds for 1,000 users is 12.5kbps or 250MB per month).

To start the server just run ruby push_server.rb. The server will start on port 2000 and you can access the example page via http://localhost:2000/hold.

The next article will be about putting a scalable observer/listener layer on top of this HTTP push server connected to our back-end status database.