2013 Will Rock More

If you are into JavaScript Engineering / Front End Development (and chances are very high that you “are” if you are reading this) you must have realized that every year things evolve exponentially faster than the former one.

In the year 2012, so many awesome articles, tools and resources were released; and so much passion, time and energy was invested in establishing standards and finalizing new APIs.

Also the developer tools have become even better (take recent changes in Firebug, for example — or the “Framed Mode” in Chrome Developer tools.

So for those who think HTML5 won’t be ready until, say 2050… I say “Bah, humbug!”.

Actually that phrase takes me back to the year 2006, and it belongs to a thought leader, whom you (have to) know — sometimes I think that I’ve a photographic memory of some sort

Anyways, W3C has officially announced that HTML5 is done and HTML5.1 is on the agenda. And browser vendors are working aggressively hard to implement as many of the HTML5 features as they can.

I don’t care if some “Sugar Mountain” utters that HTML5 is not good enough to be used in a mobile application — Bah, humbug!

“JavaScript Interview Questions” in the New Year

The rapid evolution of the industry was also a core part of what I’ve e-mailed to the readers of JavaScript Interview Questions today.

Bottom line of my message was that acing your job interview like a Samurai was only part of the equation. Your aim should be to be “great” at what you do (not to be a great “virtual realitiy” game (i.e. interviews) player, instead).

And it’s not only a message for the book’s audience, but also a reminder for everyone who want to be ahead of the competition. So I have an urge for repeating myself here:

To be a great JavaScript Engineer, you have to keep track of everything and update yourself with a continuous stream of knowledge, technologies, and tools. I don’t know you, and I really “love” being a part of this (r)evolution!

Yes everything is evolving rapidly. And during all this hassle, my aim is to come up with a “time-independent” book, that won’t get “rusted“. And to reach this aim, I…

• Try to give be as much platform/framework agnostic as possible (giving plain old JavaScript examples);
• Try to provide enough supporting material to establish firm grounds to stand when an interviewer asks you a tricky question;
• Try to “incept” a thirst to read an learn more in the reader (and serve this thirst by providing further reading material which is really hard to gather with a naive google search – have I told you that I have a good memory ?).

I’m not sure whether I’m successful in these goals. Especially the last one is the hardest and trickiest one — And I’d love to hear your comments and suggestions that can make “JavaScript Interview Questions” book rock.

Yet I have been (and I will be) thoroughly enjoying writing the book. To be honest, I even use the material for my personal reference. And it’s “awesome” to refer to something you’ve written in your daily work.

I hope the readers of the book are enjoying their read too.

And more importantly, this blog post wouldn’t have existed if it wasn’t you who have supported me through your motivational comments and emails. And even knowing that you’re reading this and I’m making a net positive difference in your perception on not only JavaScript, but also on the interview “virtual reality” as a whole, is an excellent motivation to sit on my arse and continue writing the book.

So I owe a big thank you to you, who help me make an impact.

Currently I’m working on the “Patterns” section of the book, and I believe that section will be ready by the end of January, 2013.

Until then, have a good one and enjoy the festive season!

How to Nail That Job Interview

It has been almost a year since I wrote about interviews. In that time frame I’ve learned a lot. Furthermore, I’ve learned most of the things the hard way.

I also realized that most of the interviewing books, articles and blog posts are completely and utterly misdirections, to say the least.

From personal experience I can say that if you waste your time memorizing brain teasers, you won’t be able to get a decent JavaScript Engineering job in the Silicon Valley.

Any book that claims this, says virtually nothing useful about interviews. Moreover it does not reflect the facts. Here’s one true fact about interviews:

Interviews are virtual realities specially designed to keep you out of the company. Since it’s a virtual environment, everyone will be extremely kind and understanding to you, and you will feel like a “sir”.

Nonetheless, the default answer in any interview is “Well… NO!”
and it’s up to your “persuasive ability” to turn that answer to a “Hell YES!”.

Most of the time you’ll think that you literally aced that interview, while the reality is that you totally sucked at either what you say, or how you say it, or your facial expressions, or your body language.

JavaScript Interview Questions

Having seen all these, I decided to write a book that will change the perception of people on JavaScript and Front End Engineering interviews as we know it.

Your Resumé Stinks

Every job search begins with a resumé (that’s also wrong by the way; there is something way more important than your resumé). And most of the resumés I see around stink so much that it’s a miracle to even receive a call for an interview using them.

I’ll be honest here, my resumé used to stink too; and I learned the hard way what interviewers are paying attention to in a resumé.

Take the Red Pill

Due to circumstances out of my control, in the last couple of years I have been to at least a hundred technical interviews on JavaScript Engineering, Front-End Development, and related positions. That’s why I’m so confident in telling you that this book will be a game-changer. Here’s why:

Rather than giving you some HR Manager’s perspective on what interviews should look like, I’m going to hand over you the red pill and tell what the JavaScript Engineering interviews really are, and what you will need to know to get the job you want.

Which means that this book is for “doers”; not spectators or readers. This book is for “believers”, for those who instinctively say “yes that makes sense” and dive in right away. This book is also for “skeptics”, who may doubt the efficacy and value of the material, but are open-minded enough to begin and give things an honest chance.

This Book is a Game-Changer

Contrary to most of the programming interview books on the market, this book will teach you how to think, instead of forcing you memorize a set of “how do you balance an unbalanced binary tree?” kind of questions. Rather, this book will show you all the most important features of JavaScript and how they fit together.

Being a Rock Star is Only a Part of the Equation

It’s part of interviewer’s job to establish a virtual reality to make you feel comfortable.

Assume that you are excellent in what you do (of course you are, by the way), and you are a perfect match for the job. However, for some reason, you totally mess up in the interview. If the interviewer is good at what she’s doing, she will see the hidden gem in you and extend you an offer. But not all interviewers are like that.

The more likely scenario is that the interviewer does nothing but listen to what you tell them and take notes.

Worse, they don’t even “get” what you’re telling them. Rather, they will believe in what they think you have told them. So being technically prepared is necessary and not sufficient. Unless you manage to deliver your answer in the expected way, you are tossed.

I know that it’s unfair.
And insisting that it’s unfair will not change the rules of the game .
Welcome to the adult world of consequences.

And even worse, even you completely mess up in the interview, you won’t understand that you’ve messed up. The interviewer will kindly thank you for your time and say that it was a great discussion.

I’ve learned the hard way not to believe the sweet things I’ve been told and fight until the bitter end (i.e. until I get an offer).

If you forget everything you’ve read so far remember this:

“Until you’ve got somethin’; you’ve got nothin’.”

It’s not the Questions, it’s “You” that Matters

Another mistake people do in studying for interviews is to memorize answers to snowcloned interview questions.

Blindly memorizing questions will not take you anywhere.

What you know is nothing when compared to how you convey that knowledge to the interviewer.

It’s not 2005 anymore, and interviews are not about memorizing brain teasers, or about studying CS 101 material. In deed there’s no spoon. That is to say, what you are being asked is not “that” important.

Interviews are all about your persuasive ability.

Don’t get me wrong though. I’m not talking about any of those NLP bullshit you’ll see around.

The persuasive ability you’ll gain to ace any interview comes out of brutally hard technical preparation, and strategically sharpening your knowledge, skills, and abilities.

Memorizing Things Will Work Against You

Interviewers hate developers parroting out some memorized jargon bubble talk. I know, because in the recent two years I have grown close connections with recruiters and decision-makers, as a side effect of being into a lot of interviews. And that’s what they’ve been telling me all the time. A skilled recruiter will understand that you are on “auto pilot” from miles apart.

Are You Ready for Hard Work?

It’s f*cking hard work! If it were easy, then everybody would be working in their dream jobs already. I never say it will be easy; and I don’t say it will be quick, either. Yet, if you’re persistent, stubborn, and ambitious enough, sooner than later you will reap the benefits of the book.

If you want to work in a rock star company, be a rock star: Do your homework, don’t just skim, but digest the book: Browse any links cited; read through any supporting materials given.

If you investigate every single one of the tips, tricks and ideas presented in the book, all the places that you’d admire to work at will be racing against each other to have you.

Still Looking for a List of Questiong to Memorize?
Don’t Buy This Book!

This book is not for the faint-hearted. It is for those who want to improve themselves by working their asses out.

If you merely want stuff to memorize there is a gazillion “interview question and their answers” available on the Internet.

My humble advice is to approach these techniques you’ll find over the Internet with a grain of salt. Personal experience has shown me that they are just misdirections and ill advice given by people living inside their ivory tower.

As you’ll see in the book, it’s not the question that’s important, it’s the road that leads to it.

Rather than focusing the needles on individual pine trees, try to see the forest. Then you’ll realize the relatively few topic areas that these questions converge. That way, you will be able to handle anything thrown at you.

I want to re-iterate things up front. The book demands hard work. The more you work on yourself, the better your overall understanding would be.

Bonus: This Book Builds a Career Network Around it

If you decide to pre-order the book, as an early bird you will have exclusive lifetime access to o2.js Career Network (only valid while the book is on pre-order, so you’d better hurry up ).

I believe if you’re interested enough to buy a book to advance your professionalism, then you deserve exclusive professional offers too .

Have I told you, this would be the best career investment you will make, ever?

Who Else Want to Gimme a Hand?

I’ve left everything, and I’m writing this book in all my free time. You’ll never know how writing a book is such a hard work, until you get into it. I’m putting an unimaginable time and effort to it; and I assure you it’ll be worth every penny you spend on it:

It’ll be both an excellent technical reference material, and a great game changer in terms of your behavioral interviewing skill set.

Each preorder further motivates me to finish the book as soon as possible.

And there are other ways you can help me out too:

• You can help me about editing, copywriting, and proof-reading of the book — I will thank you for your efforts by explicitly mentioning you in the acknowledgments section;
• You can share the pre-order link on Facebook and twitter;
• You can write a blog post, after having read the book — I’ll really appreciate that;
• You can send emails to you friends whom you think might benefit from this book.

In short, I’d be grateful if you help me spread the word.

Aside:
Spending all my available time on the book also means that I won’t be updating this blog for a while. However, the book will come out to be a great reference material, and who knows, the drafts that I don’t put into the book may give birth to further blog posts too.

As always, you are more than welcome to share your comments and suggestions.

Now back to writing a game-changer, so that you can go ace that interview!
( 34,546 words… for the time being, and increasing ).

Until then, may the odds be ever in your favor!

refactoring the codebase into logically-related modules

In the former article of the series

• We rendered the user interface of our widget inside an anchor element in publisher’s website;
• We asynchronously loaded widget styles;
• We implemented a very naive authentication mechanism.
• We also defined a job queue _wdq (similar to google analytics’ _gaq) to be able to push async commands to our widget;
• And we defined a _wdAsyncInit callback (similar to Facebook’s fbAsyncInit) to be able to notify the publisher when our widget is ready and responsive.

In the following articles, we will improve these mechanisms.
However, in this tutorial our subject matter is a little different:

This tutorial is not directly related to JavaScript widget development specifics;
and it’s more about code reorganization.

Logically Splitting Widget API File

If you remember from the former article we used to have a single monolithic api file (api.v.0.1.js) to manage the entire widget initialization flow, event registration, and other API behavior. In a real production application, it’s highly possible that our api code will grow over a few hunderds of thousand lines of code. We will have hard time improving, developing and debugging our codebase.

My experience shows that chaos is the inevitable fate of any software project that lives for several years, and gets developed by different teams of developers along the way. Although we cannot avoid it, at least we can minimize the consequences of it, by organizing chaos in an orderly manner as much as we can.

Refactoring to the Rescue

One way to manage our codebase, is to keep our files in logically-coherent modules; and to segregate those modules further into tiers (we’ll come to that later).

By splitting our code into logically-coherent modules;

• We will have less merge conflicts when working with teams,
• And it will be easier to find the exact point of failure, when a functionality breaks.

A clever refactoring is not a waste of time. Per contra, it will gain you a lot of time, and help you easily find and fix bugs in the long run.

We’ve seen a refactoring case study in the BPC architecture article.

In this tutorial we will follow a similar approach for layering our widget codebase.

BPC Architecture in Review

Here’s an overview of the BPC architecture for a recap:

Four Tiers of Separation

In the figure above, the dashed arrows stand for DOM Event callbacks, and asynchronous server-side callbacks (AJAX/JSONP etc).

The moral of the story is:

• All callbacks go to the delegation tier.
• Delegation tier cannot call the communication tier or the presentation tier directly, it should call the behavior tier first.
• Presentation tier is the “sink“, it cannot call anyone else.
• The only exception is the persistence tier: everyone can read from it, but only the behavior tier can write to it.

Similar to the model above, in this article we will organize our codebase into well-defined behavior, communication, delegation, and persistence tiers; where each tier will contain one or more modules.

Improve BPC Architecture Further

And to improve the structure further, we will decouple our modules as much as we can:

Instead of having modules call public methods of one another, we will be raising custom events, for inter-module communication.

This idiom is widely known as the Publish-Subscribe Pattern.

Using the pubsub pattern, we will decouple our modules further:

Any given module will not have direct knowledge of any other module talking to it.

This loose coupling;

• Will give us the flexibility to easily move code pieces around;
• And, since the modules only know about themselves and no one else, it will be easier to unit test those modules individually in their sandbox.

Application Directory Structure

Enough introductory talk; now let’s see how we apply all this theory to our widget API.

To begin with, here’s the directory struture of our application:

Widget API Directory Tree

In the lib folder we have helper libraries (such as o2.js) and widget modules (i.e. the /lib/wd/ folder).

The widget modules raise events for inter-module communication. The modules also directly use helper libraries for utility functions (such as JSONP calls, and DOM event delegation).

We will come to the implementation of these very soon.

As seen in the image above we have 5 tiers (represented by folders): behavior, comunication, communication, delegation, persistence and presentation.

Each tier, further down, contains modules.

And each module is nothing but a static object under window._wd.protecteds namespace.

For example the Init module under the behavior tier, resolves to window._wd.protecteds.Init

The “protecteds” namespace is an informal indicator that these modules are meant to be used by the internal functionality of our widget.

Protected modules are not meant to be accessed publicly by the publisher.

Establishing Inter Module Communication

In order to send messages to other modules, each module either publishes events:

    // protecteds/behavior/init.js

    /*
     * Load initial widget state data from the server.
     */
    function loadState(config) {
        log('Init.loadState(');
        log(config);
        log(')');

        // Behavior -> Communication
        p.pub('SEND_GET_PARAMS', [config]);
    }

…or subscribes to events:

    // protecteds/communication/proxy.js

    /**
     * @function {static} Proxy.subscribe
     *
     * Subscribes to relevant events.
     */
    me.subscribe = function() {
        log('Proxy.subscribe()');

        ...

        // Get widget parameters from the server.
        sub('SEND_GET_PARAMS', function(params) {
            log('event<SEND_GET_PARAMS');
            log(params);
            log('>');

            get(
                concat(url.API_ROOT, path.PARAMS),
                params,
                callback.sendGetParams_complete
            );
        });

        ...
    };

One caveat though:

Implementing a pubsub pattern is a bit like using a hand grenade.

In the hands of thoughtful experts, you can expect to have a sky scraper demolished safely and in a reasonable time frame.

However letting amateurs mess with the pubsub pattern is akin to giving a box of hand grenades to a gang of monkeys:

You’ll end up with a very noisy and messy scene when you least expect it.

And every once in a while, even experts act like monkeys .

Split Main API file into #regions

This is a habit I grew back in my .net/C# development years.

In Visual Studio, you can separate logically distinct parts of the code with #region pragmas. Those #regions will be rendered distinctly to indicate that they are logicially-related groups of code.

Here’s the new api.v.0.1.js split into regions
(I’m replacing parts of the code with … to save some screen real estate):

...
(function(window, document, isDebugMode) {
    'use strict';

    /* #region version */

        /*
         * Should match beacon version timestamp.
         */
        var versionTimestamp = '20120720135547909116';

    /* #endregion */

    /* #region widget state */

        /*
         * Resources to be loaded asynchronously.
         */
        var scriptQueue = [];

        /*
         * This will be set after resource initialization.
         */
        var o2 = null;

    /* #endregion */

    /* #region module common constants */

        /*
         * Query Formation
         */
        var kAnd    = '&';
        var kEmpty  = '';
        var kEquals = '=';
        var kQuery  = '?';

        /*
         * Regular Expression
         */
        var kCompleteRegExp = /loaded|complete/;

        /*
         * Tags
         */
        var kHead   = 'head';
        var kScript = 'script';

        /*
         * Mime Types
         */
        var kScriptType = 'text/javascript';

        /*
         * Globals
         */
        var kO2Alias     = '_wd_o2';
        var kWidgetAlias = '_wd';

        /*
         * Common Widget Keys
         */
        var kReadyState = 'readyState';

    /* #endregion */

    /* #region exported configuration */

        /*
         * Parameter Names
         */
        var param = {
            GUID     : 'guid',
            RANDOM   : 'r',
            VERSION  : 'v',
            USERNAME : 'u',
            PASSWORD : 'p',
            ACTION   : 'action',
            PAYLOAD  : 'payload'
        };

        /*
         * Element IDs
         */
        var elm = {
            LOGIN_BUTTON : 'wd_btnLogin'
        };

        /*
         * Widget Ready States
         */
        var readyState = {
            LOADED               : 1,
            LOADING_DEPENDENCIES : 2,
            LOADED_DEPENDENCIES  : 3,
            BEGIN_PROCESS_QUEUE  : 4,
            BEGIN_RENDER         : 5,
            COMPLETE             : 6
        };

        /*
         * URL
         */
        var url = {
            API_ROOT        : 'http://api.widget.www/',
            O2_ROOT         : 'http://api.widget.www/lib/o2.js/',
            WIDGET_LIB_ROOT : 'http://api.widget.www/lib/wd/v.0.1/',
            LIB_ROOT        : 'http://api.widget.www/lib/'
        };

        /*
         * Path
         */
        var path = {
            BEACON : 'api/v.0.1/beacon',
            CSS    : 'css/v.0.1/widget.css',
            LOGIN  : 'api/v.0.1/login',
            PARAMS : 'api/v.0.1/params'
        };

        /*
         * Custom Events (used for inter-module messaging)
         */
        var event = {
            BEGIN_RENDER     : 'wd-begin-render',
            CSS_LOADED       : 'wd-css-loaded',
            DELEGATE_EVENTS  : 'wd-delegate-events',
            FIRE_ASYNC_INIT  : 'wd-fire-async-init',
            INSERT_BEACON    : 'wd-insert-beacon',
            LOAD_STATE       : 'wd-load-state',
            OVERRIDE_QUEUE   : 'wd-override-queue',
            PROCESS_QUEUE    : 'wd-process-queue',
            RENDER_DOM       : 'wd-render-dom',
            RENDER_LOGGED_IN : 'wd-render-logged-in',
            RENDER_WIDGET    : 'wd-render-widget',
            SEND_GET_PARAMS  : 'wd-send-get-params',
            SEND_LOAD_CSS    : 'wd-send-load-css',
            SEND_USER_LOGIN  : 'wd-send-user-login',
            USER_LOGGED_IN   : 'wd-user-logged-in',
            USER_LOGIN       : 'wd-user-login'
        };

    /* #endregion */

    /* #region helper methods (exported) */

        /*
         * Does nothing, and that's the point.
         */
        function noop() {}

        /*
         * Logs to console for debug mode.
         * Does nothing in release mode.
         */
        var log = function(stuff) {
            ...
        };

        /*
         * Sets the internal ready state.
         */
        function setReadyState(state) {
            window[kWidgetAlias][kReadyState] = readyState[state];
        }

    /* #endregion */

    /* #region sanitization */

        // Publisher has forgotten to provide initialization data.
        if (!window[kWidgetAlias]) {
            log('Widget namespace cannot be found; exiting.');

            return;
        }

        // To avoid re-defining everything if the bootloader is included in
        // more than one place in the publisher's website.
        if (window[kWidgetAlias][kReadyState]) {
            log('Widget has already been loaded; exiting.');

            return;
        }

    /* #endregion */

    /* #region protecteds namespace (export root) */

        /*
         * The "protected" methods are shared across modules, but they
         * are not intended for public use.
         */
        window[kWidgetAlias].protecteds = {};

    /* #endregion */

    /* #region widget initialization */

        /*
         * Asynchronously inserts a script element to the head
         * of the document.
         */
        function insertScript(root, src) {
            ...
        }

        /*
         * Revalidates cache for this bootloader script, if there's a newer
         * version available. The changes will take effect only AFTER the user
         * refreshes the page.
         */
        function checkForUpdates() {
            ...
        }

        /*
         * Exports protected methods for intra-module use.
         */
        function exportProtecteds() {
            ...
        }

        /*
         * Loads the next resource after the former one
         * has loaded successfully.
         */
        function loadNext(root, loader, callback) {
            ...
        }

        /*
         * Loads the given script.
         * <strong>callback</strong> is the function to be executed after
         * there's no resource left to be loeded next.
         */
        var loadScript = function(root, src, callback) {
             ...
        };

        /*
         * Loads an array of scripts one after another.
         */
        function loadScripts(root, ar, callback) {
            ...
        }

        /*
         * First loads necessary o2.js components in noConflict mode.
         * Then loads protected modules.
         */
        function loadDependencies(callback) {
            log('o->loadDependencies(');
            log(callback);
            log(')');

            setReadyState('LOADING_DEPENDENCIES');

            loadScripts(url.LIB_ROOT, [
                'o2.js/o2.meta.js',
                'o2.js/o2.core.js',
                'o2.js/o2.string.core.js',
                'o2.js/o2.jsonp.core.js',
                'o2.js/o2.dom.constants.js',
                'o2.js/o2.dom.core.js',
                'o2.js/o2.dom.load.js',
                'o2.js/o2.event.constants.js',
                'o2.js/o2.validation.core.js',
                'o2.js/o2.event.core.js',
                'o2.js/o2.event.custom.js',
                'o2.js/o2.method.core.js',
                'o2.js/o2.collection.core.js',

                'wd/v.0.1/protecteds/behavior/init.js',
                'wd/v.0.1/protecteds/behavior/queue.js',
                'wd/v.0.1/protecteds/behavior/widget.js',

                'wd/v.0.1/protecteds/communication/proxy.js',

                'wd/v.0.1/protecteds/delegation/callback.js',
                'wd/v.0.1/protecteds/delegation/event.js',

                'wd/v.0.1/protecteds/persistence/config.js',

                'wd/v.0.1/protecteds/presentation/dom.js',
                'wd/v.0.1/protecteds/presentation/rendering.js'
            ], callback);
        }

    /* #endregion */

    /* #region widget initialization flow */

        // At the end of the initialization flow, readyState will be finally
        // set to COMPLETE. When the readyState is COMPLETE, it means that
        // the widget UI has been rendered, the events have been bound,
        // widget job queue has been processed, and the widget is completely
        // ready and responsive.
        setReadyState('LOADED');

        checkForUpdates(versionTimestamp);
        loadDependencies(initialize);

    /* #endregion */
}(this, this.document, true));

Adding Modules as Dependencies

One thing we add to our widget’s initialization flow is the URL‘s of our protected modules.

We first load them as dependencies, and let each module subscribe to custom API events once they are loaded.

We’ll come to the module subcription soon.

Here’s the set of files we load as dependencies:

        /*
         * First loads necessary o2.js components in noConflict mode.
         * Then loads protected modules.
         */
        function loadDependencies(callback) {
            log('o->loadDependencies(');
            log(callback);
            log(')');

            setReadyState('LOADING_DEPENDENCIES');

            loadScripts(url.LIB_ROOT, [
                'o2.js/o2.meta.js',
                'o2.js/o2.core.js',
                'o2.js/o2.string.core.js',
                'o2.js/o2.jsonp.core.js',
                'o2.js/o2.dom.constants.js',
                'o2.js/o2.dom.core.js',
                'o2.js/o2.dom.load.js',
                'o2.js/o2.event.constants.js',
                'o2.js/o2.validation.core.js',
                'o2.js/o2.event.core.js',
                'o2.js/o2.event.custom.js',
                'o2.js/o2.method.core.js',
                'o2.js/o2.collection.core.js',

                'wd/v.0.1/protecteds/behavior/init.js',
                'wd/v.0.1/protecteds/behavior/queue.js',
                'wd/v.0.1/protecteds/behavior/widget.js',

                'wd/v.0.1/protecteds/communication/proxy.js',

                'wd/v.0.1/protecteds/delegation/callback.js',
                'wd/v.0.1/protecteds/delegation/event.js',

                'wd/v.0.1/protecteds/persistence/config.js',

                'wd/v.0.1/protecteds/presentation/dom.js',
                'wd/v.0.1/protecteds/presentation/rendering.js'
            ], callback);
        }

Widget Initialization

We also added a couple of extra lines to the widget initialization code:

        /*
         * Initialize after loading prerequisites.
         */
        function initialize() {
            ...

            exportProtecteds();

            var config = wp.Config.get();

            config[param.GUID] = o2.String.generateGuid();

            subscribe();

            wp.pub('LOAD_STATE', [config]);
        }

Let’s expand the above code, line by line.

We first export a set of methods that will be commonly used in all modules:

        /*
         * Exports protected methods for intra-module use.
         */
        function exportProtecteds() {
            log('o->exportProtecteds()');

            var wp = window[kWidgetAlias].protecteds;

            wp.sub = function(name, callback) {
                var nom = wp.event[name];

                if (!nom) {
                    log(['wp.sub: No such event for "', name, '"'].join(kEmpty));

                    return;
                }

                o2.Event.subscribe(nom, callback);
            };

            wp.pub = function(name, payload) {
                var nom = wp.event[name];

                if (!nom) {
                    log(['wp.pub: No such event for "', name, '"'].join(kEmpty));

                    return;
                }

                o2.Event.publish(nom, payload);
            };

            wp.event         = event;
            wp.log           = log;
            wp.noop          = noop;
            wp.o2            = o2;
            wp.path          = path;
            wp.setReadyState = setReadyState;
            wp.url           = url;
            wp.param         = param;
            wp.elm           = elm;
        }

wp is an alias for window._wd.protecteds namespace. We bind a bunch of helper functions and configuration objects to that namespace to access them from other modules.

Then we call subscribe methods of related modules (which makes the modules subscribe to related event, as the name implies):

    /*
     * Trigger modules to subscribe to events.
     */
    function subscribe() {
        var wp = window[kWidgetAlias].protecteds;

        wp.Init.subscribe();
        wp.Queue.subscribe();
        wp.Event.subscribe();
        wp.Widget.subscribe();
        wp.Proxy.subscribe();
        wp.Rendering.subscribe();
    }

o2.js PubSub Internals

The internals of the pubsub mechanism is managed by o2.event.custom.js module, which is an extension to the o2.Event class.

The o2.event.custom module exports three methods: o2.Event.publish, o2.Event.subscribe, and o2.Event.unsubscribe as follows:

(function(framework) {
    'use strict';

    var _         = framework.protecteds;
    var attr      = _.getAttr;
    var create    = attr(_, 'create');
    var def       = attr(_, 'define');
    var require   = attr(_, 'require');

    var exports = {};

    var kModuleName = 'Event';

    var me = create(kModuleName);

    /*
     * Aliases
     */

    var isArray = require('Validation', 'isArray');

    var cache = {};

    exports.publish = def(me, 'publish', function(name, argv) {
        if (!name) {
            return;
        }

        var delegates = cache[name];

        var args = argv || [];

        if (!isArray(args)) {
            args = [args];
        }

        var i         = 0;
        var len       = 0;

        if (!delegates) {
            return;
        }

        for (i = 0, len = delegates.length; i < len; i++) {
            try {
                delegates[i].apply(null, args);
            } catch (ignore) {}
        }
    });

    exports.subscribe = def(me, 'subscribe', function(name, callback) {
        if (!name) {
            return;
        }

        if (!cache[name]) {
            cache[name] = [];
        }

        cache[name].push(callback);

        var handle = {
            name     : name,
            callback : callback
        };

        return handle;
    });

    exports.unsubscribe = def(me, 'unsubscribe', function(handle) {
        var name = handle.name;
        var callback = handle.callback;

        var delegates = cache[name];
        var i         = 0;
        var len       = 0;
        var delegate  = null;

        if (!delegates) {
            return;
        }

        for (i = 0, len = delegates.length; i < len; i++) {
            delegate = delegates[i];

            if (delegate === callback) {
                delegates.splice(i, 1);
                len = delegates.length;
                i--;
            }
        }
    });
}(this.o2));

Where window._wd.pub and window._wd.sub methods wrap around o2.Event.publish, and o2.Event.subscribe methods respectively.

Read the Source Luke

You can view the source code from this github history snapshot.

Conclusion

That concludes our quick tour on widget refactoring .

In this article;

• We divided our widget into logical modules, and grouped those modules into different tiers;
• We also implemented a custom publisher subscriber messaging system for inter-module communication;
• As a corollary to the above; almost all of the modules never exposed public methods, and instead modules sent and received messages using the pubsub mechanism above, rather than calling methods on each other.
• This pubsub mechanism allowed “loose coupling” between modules; which, in turn, will make it possible to unit test each module independently.

Next Up?

In the following article, we will convert our application with something more meaningful, use CORS for cross domain POST requests, detect if CORS is supported, and discuss alternatives if it’s not. We will also discuss minification, compression and deployment strategies; and ways to secure our widget and prevent it from certain XSS and CSRF attacks.

In short we will try wrap things up in the next article .

Until then, feel free to share your comments and suggestions.

jstanbul was a great JavaScript conference; and it was great to be a contributor!

After my conference on jstanbul 2012,

I updated the presentation adding some bullet points for the questions and feedback from the community.

You can view and download the presentation from SlideShare.

And I’ll continue this blog’s series on JavaScript Widget Development Best Practices
from where I left.

Until then, feel free to add your comments .

getting our hands dirty

I’m giving a talk on JavaScript Widget development best practices tomorrow at jstanbul 2012.

At the conference, I will have a 30-minute timeframe to express as much as I can with respect to external JavaScript widget development best practices.

Since 30 minutes is not enough for this, my aim is to give a quickly overview, and send the audience to this blog for further investigation.

This means that this blog article (that I’m currently writing) must be as complete and as coherent as possible, before my talk tomorrow.

So I’m going to cram as much things into here as possible. I will intentionally leave certain parts incomplete, and I will definitely leave (some obvious, some not so obvious) security loopholes. I will cover them in the following articles of the series.

The Result

At the end of this tutorial we will be creating something similar to this:

widget login user interface

widget login form

widget (logged in)

widget console

Note that the login form above is dynamically injected from the widget provider’s domain (http://api.widget.www). And the widget lives in a totally different publisher domain (http://publisher.www).

That’s one of the many things that makes external JavaScript widget development complicated.

For a broad overview of other challenges involved in External JavaScript Widget Development, you can view my #jstanbul 2012 slides on the topic

So let’s start and get our hands dirty; shall we?

Last Part in Review

Here’s a recap of what we’ve done so far: In the former article, we have seen different ways to send initialization parameters to our widget, and we have also listed several techniques on how to establish cross-domain communication between the publisher’s site and the widget API server. We choosed JSONP as a cross-browser proxy, while mentioning the possible drawbacks of our choice.

Let’s Render a User Interface

When it comes to external JavaScript widgets, there are various ways to render. The most common two are:

• Rendering the widget’s UI by appending HTML before the widget’s script tag.
• And rendering the widget’s UI by explicitly specifying a target location (i.e. a DOM element with a specific attribute) on the publisher’s website.

The former technique involves traversing the publisher’s DOM, locating the instance or instances of widget bootloader scripts, and injecting HTML before each script.

While in the latter, we explicity specify placeholder HTML tags as an anchor positions to render the widget. We’ll see an example of this technique down below.

We’ll continue our tutorial with the second implementation, since…

• It gives more flexibility;
• It is less prone to errors;
• It does not create race conditions (when we have more than one scripts on the page);
• Requires less lines of code;
• And is a better demonstration of separation of concerns.

Create an Anchor on the Publisher’s Website

Let’s start by adding a div with a special data-wd-anchor attribute.

Our bootloader script will detect this attribute and render our widget’s user interface inside that div.

-//publisher.www/index.jade
    ...
    body(lang='en')
        section#Main
            h2 Welcome to Publisher Website

            p(style='float:right;width:300px') Lorem ipsum dolor sit amet.

            p(style='float:left;width:300px') Vestibulum dui erat.

            div(data-wd-anchor, style='float:left;width:300px;margin-top:20px;margin-left:10px;')

            p(style='float:right;width:300px') Mauris ac dolor ante.

And here is the piece of code on the widget API script that does the rendering:

// api.widget.www/api.v.0.1.js

    /*
     * Renders the widget
     */
    function render(state) {
        var div = getWidgetAnchor();

        if (!div) {
            return;
        }

        o2.Dom.loadCss(
            o2.String.concat(kApiRoot, kCssPath),
            function() {
                renderWidget(div, state.data);

                processPostRenderActions();
            }
        );
    }

We will come to o2.Dom.loadCss later.
For now lets focus our attention to getWidgetAnchor method:

    /*
     * Find a place to append the widget UI.
     */
    function getWidgetAnchor() {
        var div = null;

        // divs is a "live" node list
        var divs = document.getElementsByTagName(kDiv);
        var len  = divs.length;
        var i    = 0;

        for (i = 0; i < len; i++) {
            div = divs[i];

            if (div.hasAttribute(kWidgetAnchor)) {
                return div;
            }
        }

        return null;
    }

The method simply finds the first div element with the data-wd-anchor attribute and returns it.

Styling the Widget

To give the desired look and feel and functionality to our widget, we have to add some CSS styling to our widget.

There are several ways of doing this:

• We can use inline styles (as in [div style="background:#aaccff;border:1px solid">...[/div]);
• We can load our CSS file as a dependency at the initialization phase of the bootloader;
• We can lazy load the CSS just before rendering the widget;
• We can embed the CSS declerations directly into JavaScript and append them to a style node that we dynamically create.

Each of these techniques have their pros and cons.
In this tutorial we will continue with the “lazy loading” approach.

Adding More Modules to the Dependency Loader

But first we’ll need to add a couple of more JavaScript files to our dependency loader:

    /*
     * Load necessary o2.js components in noConflict mode.
     */
    function loadDependencies(callback) {
        log('o->loadDependencies(');
        log(callback);
        log(')');

        setReadyState(kLoadingDependencies);

        loadScripts(kO2Root, [
            'o2.meta.js',
            'o2.core.js',
            'o2.string.core.js',
            'o2.jsonp.core.js',
            'o2.dom.constants.js',
            'o2.dom.core.js',
            'o2.dom.load.js',
            'o2.event.constants.js',
            'o2.event.core.js',
            'o2.validation.core.js',
            'o2.method.core.js',
            'o2.collection.core.js'
        ], callback);
    }

Note that we are adding more and more files to the initializer array. Since every item in this array corresponds to one more HTTP request, in a production application we will have to merge, minify and gzip those scripts into a single file to improve speed and performance (we’ll come to that in followup articles).

Asynchronously Load Widget CSS

Let’s see our render code once more:

    /*
     * Renders the widget
     */
    function render(state) {
        ...

        o2.Dom.loadCss(
            o2.String.concat(kApiRoot, kCssPath),
            function() {
                renderWidget(div, state.data);

                processPostRenderActions();
            }
        );
    }

The above code appends http://api.widget.www/css/v.0.1/widget.css file to the head of the document, and then executes the callback closure.

Just for a sec, assume that the CSS that we load contains the following directive

body {background: red !important;}

After loading the CSS, we’ll see that all of a sudden the publisher’s page background will become red.
Remember? We don’t own publisher’s DOM, so we have to constraint our styling to our widget only.

Styling a widget requires a lot of testing with a combination of different demo publisher pages and user agents; since unless you’re injecting your widget with an iframe (and there are good reasons to do so; one of which is preserving the look and feel), the publisher’s styles can always override yours.

Moreover you have to decide the following yourself?

• Shall the widget look exactly the same in every single one of the publishers?
• Or shall the widget inherit some of the styling of the publisher, to give a more streamlined look and feel?
• Shall we, as the widget authors, provide a way to easily modify the styling of our widget depending on their taste?
• And if so, to what extent?

There is no silver bullet to correctly answer these questions, and you have to judge for yourself regarding how rigid or how flexible your approach may be.

Techniques to Ensure Style Consistency

To avoid our styles conflict with the publisher’s we will need to prefix them with our namespace, for example:

.wd-box p {
    margin : 5px !important;
    padding: 0 !important;
}

.wd-box.wd-login {
    width : 332px;
}

.wd-boxBody {
    background   : #fefefe;
    border-top   : 1px solid #dde0e8;
    border-bottom: 1px solid #dde0e8;
    border-left  : 0;
    border-right : 0;
    padding      : 10px 20px;
}

Since .box is a really generic class name, it’s possible that our publisher may have a .box class defined already, and it may interfere with ours. So instead of using class names like .box and .login, we prefix them with our widget namespace wd- as in .wd-box, and .wd-login.

As a rule of thumb, prefix anything that’s publicly reachable and modifiable (like css classes, global variables, DOM attributes…)

However due to the rules of CSS specificity, simply prefixing our classes may not provide enough protection: Our rules may be overridden by a more specific ruleset of the publisher’s.

If that’s the case we can over-specify our selectors like:

body#wd-mastercontainer #wd-wrap .wd-box.wd-login {
    width : 332px;
    margin: 10px;
}
[/sourcecodde]

Using two id selectors makes our selector <strong>highly unlikely</strong> to be unintentionally overridden by the publisher's styles. And if the publisher "intentionally" overrides the styles, she should know what she's doing anyway.

Another thing we can do is to <strong>abuse !important</strong> declarations:

[sourcecode language="css"]
.wd-boxBody {
    background   : #fefefe !important;
    border-top   : 1px solid #dde0e8 !important;
    border-bottom: 1px solid #dde0e8 !important;
    border-left  : 0 !important;
    border-right : 0 !important;
    padding      : 10px 20px !important;
}

Keep in mind that overusing !important declarations also takes away the ability give the publisher freedom to override our styles, if they want: !important declerations will always take precedence. Use them sparingly.

Render the Initial UI

After loading the CSS, we render the widget UI inside the callback. That’s as easy as assigning an innerHTML.

    /*
     * Does the actual rendering.
     */
    function renderWidget(container, html) {
        if (!container) {
            return;
        }

        container.innerHTML = html;
    }

Post-Render Processing

And after rendering the widget UI, we do further processing (in processPostRenderActions):

    /*
     * Things done after the initial view is rendered.
     */
    function processPostRenderActions() {
        delegateEvents();

        processQueue();

        window[kWidgetQueueAlias] = queue;

        setReadyState(kComplete);

        fireAsyncInit();
    }

Here’s what the above code does:

• We bind event handlers;
• Then we process the job queue that the publisher has provided us;
• Then we overload the queue implementation with an Array-like object;
• Then we set ready state to complete to indicate that our initialization has been completed;
• Then we call _wdAsyncInit to tell the publisher that we are done with the widget initialization.

Let’s see all those codes one by one:

Event Delegation

    /*
     * Use event delegation to bind widget events.
     */
    function delegateEvents() {
        log('o->delegateEvents()');

        o2.Event.addEventListener(document, kClick, document_click);
    }

We simply add a global event listener to document’s click event.

Process Job Queue

The job queue is simply an array (window._wdq) of directives that are delated to be executed until the widget is ready. Since the directives are executed async, it does not affect speed of the widget’s initialization. It also does not degrade the publisher site’s performance.

Here’s how we process the job queue:

    /*
     * Processes the job queue item by item.
     */
    var processQueue = function() {
        log('o->processQueue()');

        setReadyState(kBeginProcessQueue);

        var q = null;

        if (window[kWidgetQueueAlias] &&
                    o2.Validation.isArray(window[kWidgetQueueAlias])) {
            q = window[kWidgetQueueAlias];

            while (q.length) {
                execute(q.pop());
            }
        }

        processQueue = noop;
    };

Where the execute function is as follows:

    /*
     * Executes the job queue asyncronously.
     */
    function execute(item) {
        log('o->execute()');

        var action = item[kAction];

        switch (action) {
            case kEcho:
                log('ECHO: ');
                log(item[kPayload]);

                break;
            default:
                log('ERROR: no mapping for action "' + action + '"');
        }
    }

The method simply executes API methods depending on the action parameter.

So the following code in the publisher’s page:

window._wdq.push({action : 'echo', payload : {'hello' : 'world'}});

Will print

ECHO:
{"hello" : "world"};

on the console.

Overload Queue Implementation

And after that we overload the queue implementation:

window[kWidgetQueueAlias] = queue;

Where the queue implementation is:

    /*
     * An overridden version of the async job queue.
     */
    var queue = {
        items : [],

        push : function(item) {
            log('o->queue.push()');

            execute(item);
        }
    };

We just overload the push method so that the publisher can call window._wdq.push even after we initialize our widget.

Call Async Init Function

Facebook uses exactly the same approach, when initializing FB JS SDK.

At the end of the flow we simply call window._wdAsyncInit to give publisher the freedom execute their post-initialization logic, if necessary.

Click Event Delegation

Now let’s observe the global document click event handler:

    /*
     * Global event handler on document's click event.
     */
    function document_click(evt) {
        log('document_click()');

        var target = o2.Event.getTarget(evt);

        var id = target.id;

        if (!id) {
            return;
        }

        // Just for demonstration.
        var params = {};
        params[kUsername] = 'dummy';
        params[kPassword] = 'dummy';

        if (id.indexOf(kLoginButtonId) > -1) {
            o2.Jsonp.get(
                o2.String.concat(kApiRoot, kLoginPath),
                params,
                processUserLogin
            );
        }
    }

To understand what this function does let’s flashback to initialize method:

    /*
     * Initialize after loading prerequisites.
     */
    function initialize() {
        log('o->initialize()');

        setReadyState(kLoadedDependencies);

        if (!window.o2) {return;}

        window.o2.noConflict(kO2Alias);

        o2 = window[kO2Alias];

        var config = getConfiguration();

        config[kGuid] = o2.String.generateGuid();

        loadInitialState(config, processPostInitialization);
    }

We send a guid parameter (via o2.String.generateGuid()) to the server. This way, the server’s the rendered HTML will
have this guid prefix appended to ids of its elements.

Here’s a sample view for our widget’s rendered HTML:

<form class="wd-box wd-login" id="wd_h5b9s072n8l">
    <fieldset class="wd-boxBody">
        <label for="wd_usernameh5b9s072n8l">Username</label>
        <input type="text" required="required" placeholder="type in your &quot;Cool Publisher&quot; 
        username." tabindex="1" id="wd_usernameh5b9s072n8l">
        <label for="wd_passwordh5b9s072n8l">
        <a class="wd-rLink" tabindex="5" href="/">Forgot password?</a><span>Password</span></label>
        <input type="password" required="required" tabindex="2" id="wd_passwordh5b9s072n8l">
    </fieldset>
    <footer>
        <label for="wd_chk_rememberh5b9s072n8l">
            <input type="checkbox" tabindex="3" id="wd_chk_rememberh5b9s072n8l">
            <span>Keep me logged in</span>
        </label>
        <input type="button" class="wd-btnLogin" tabindex="4" 
            value="Login" id="wd_btnLoginh5b9s072n8l">
    </footer>
</form>

So on the global document_click event handler, if the user clicks on the login button, we detect it an do a JSONP API call:

...
        if (id.indexOf(kLoginButtonId) > -1) {
            o2.Jsonp.get(
                o2.String.concat(kApiRoot, kLoginPath),
                params,
                processUserLogin
            );
        }
...

When the call succeeds, processUserLogin is executed, which simply refreshes the widget view:

    /*
     * User login JSONP callback.
     */
    function processUserLogin(response) {
        var div = getWidgetAnchor();
        div.innerHTML = response.data;
    }

Session Persistence

And here’s what happens on the server when the user sends a login request:

// api.widget.wwwroot/index.js
...
        /**
         * Authenticates user.
         */
        app.get(v_0_1(route).LOGIN, function(req, res) {
            var username = req.param(parameter.USERNAME);
            var password = req.param(parameter.PASSWORD);
            var callback = req.param(parameter.CALLBACK);

            if (!username) {
                res.send(statusCode.NO_DATA);
            }

            if (!password) {
                res.send(statusCode.NO_DATA);
            }

            if (!callback) {
                res.send(statusCode.NO_DATA);
            }

            var session = req.session;

            session[parameter.USERNAME] = username;

            var params = {};

            sendJsonp(req, res, params, callback);
        });

Where sendJsonp is:

    /*
     *
     */
    function send(res, callback, result) {
        res.send(
            callback + '(' + JSON.stringify(result) + ');'
        );
    }

    /*
     *
     */
    function sendJsonp(req, res, params, callback) {
        var result = {
            data :
                isUserLoggedIn(req) ?
                createDashboard(params) :
                createLoginForm(params)
        };

        send(res, callback, result);
    }

And isUserLoggedIn is:

    /*
     *
     */
    function isUserLoggedIn(req) {
        return !!req.session[parameter.USERNAME];
    }

Without any verification, we add user’s username to the session, and send a dashboard html to the publisher to refresh its widget’s view.

Note that this is just for demonstration purposes. Normally we should have validated username/password of the user against our database and send an error in the JSONP response if they don’t match. We’ll implement this in followup articles.

Since we store the username in the session; when we refresh publisher’s page our widget will immediately render the dashboard instead of the login form. We will not see the login form until the user’s session expires.

Third Party Issues

There’s one caveat though. To persist session, we are making use of third party cookies.

In some of the browsers (like Safari) third party cookies are disabled by default. And in all of the browsers users can explicitly disable third party cookies. We cannot persist user date in the Session object, if third party cookies are disabled.

There are ways to overcome these, which we will explore in followup articles too. For instance Facebook uses a dedicated login window in a popup in conjunction with an approach similar to the klein bottle method, to be able to read and write third party cookies.

Size Does Matter

If you have followed all the tutorials this far, you migh have seen that the size of our bootloader is growing as we add more and more feature.

Currently this size is manageable, however after a certain limit, you may want to submodules for each logical group, and either lazy-load them or load it while loading the initial dependencies.

I’m trying to bring this tutorial series to a somewhat complete state before my seminar in jstanbul 2012, so I won’t have time for this refactoring right now.

However we’ll be doing this refactoring, along with certain security enhancements, deplyoyment scripts, compression and minification in the upcoming tutorials.

Next Up?

In the upcoming tutorials we will cover the following topics, and more:

• Refactoring our API to further split into logically related modules;
• Minification and deployment;
• Authorization and authentication;
• Persisting and retrieving data using CORS;
• Strengthening the security of our application, preventing against CSRF attacks.

Read the Source Luke

You can get the source code accompanying this tutorial at this github history snapshot.

Conclusion

If this tutorial looks like a few articles loosely coupled together out of hassle, it probably is

In the followup articles we will elaborate on many of the issues we’ve just touched the surface of here.

Also for the unlikely case that you are in Istanbul (since 90% of my blog readers are in the Bay Area), see you in jstanbul 2012!

And, as always, feel free to send your comments and suggestions.

outside in, inside out, that’s how things work around

In the former part of the series we’ve seen how to revalidate the cache and load our widget code using a self updating bootloader script.

Now it’s time to pass initialization parameters to our widget, and request some state data from the widget API server.

Let’s recap:

Our lovely publisher is including a async loading snippet to their pages. Once the script loads, any additional dependencies are loaded too. And then our widget’s initialization flow starts.

Configuration Parameters

A thing almost all of the widgets have in common is a set of initialization parameters, which give some flexibility over the widget’s initial construction and which may also be used to identify the publisher.

For the sake of our example, let’s assume that we need to pass around a publisher ID to our api bootloader.

We, as widget provider, can use this publisher ID to send different data to different publishers, track API usage of various publishers, throttle/limit/block the API responses depending on the type of the publisher. All of these and more can be possible by simply passing a publisher ID to the server, during widget’s initialization.

Note that a simple ID, or a simple API token is not enough to identify a publisher, since anyone can copy the initialization script and inject it to their site.

You can use this ID in conjunction with Referrer header to indicate that your request is coming from the intended publisher. There are other metrics you can use like setting third party token cookies for additional verification, but from experience I can say that creating an application that just works is far more important than working on a problem that you don’t actually have.

We will be working on session, persistence, authentication, and authorization in the upcoming articles too. Stay tuned

There are different ways of sending parameters to our API layer. Let’s analyze a few:

Passing Widget Configuration Through the Querystring

We simply send the publisher id through the query string in the src attribute of the bootloader script.

Once the bootloader script loads, we parse that URL inside the script to get the publisher ID and then send it to the server to get initial state data:

<script>
(function() {
    var script = document.createElement('script');
        script.type  = 'text/javascript';
        script.async = true;
        script.src   = 'http://api.widget.www/api.v.0.1.js?pubId=123456'
        var node = document.getElementsByTagName('script')[0];
        node.parentNode.insertBefore(script, node);
    }());
</script>

Here’s a better way to implement this:

<script>
(function() {
        var pubId = '123456';

        var script = document.createElement('script');
        script.type  = 'text/javascript';
        script.async = true;
        script.src   = 'http://api.widget.www/api.v.0.1.js?pubId=' + pubId;
        var node = document.getElementsByTagName('script')[0];
        node.parentNode.insertBefore(script, node);
    }());
</script>

If we use this approach, we will be able to see the publisher id on the server’s HTTP logs, since it’s transferred as a part of the URL.

Passing Widget Configuration Through the Hash Fragment

Another option is to pass the parameters through the hash fragment:

<script>
(function() {
    var pubId = '123456';

    var script   = document.createElement('script');
    script.type  = 'text/javascript';
    script.async = true;
    script.src   = '//api.widget.www/api.v.0.1.js?#pubId=' + pubId;
    var node     = document.getElementsByTagName('script')[0];
    node.parentNode.insertBefore(script, node);
}());
</script>

Please note that we are using protocol relative URLs too. This will help us avoid browser’s warning when the publisher’s pages are served over HTTPS.

Keeping all your assets within the same protocol prevents the dreaded “This Page Contains Both Secure and Non-Secure Items” warning message in IE.

Other than that, we just send the publisher ID through the hash fragment, instead of the query string. When we do that, we won’t see the parameters in the server logs, since the hash fragments are not transmitted to the server during page requests.

Passing Widget Configuration Through HTM5 Data Attributes

We can also use the new kid on the block HTML5 data- attributes to pass our initialization parameters to the widget script:

<script data-wd-pub-id="123456">
(function() {
    var script   = document.createElement('script');
    script.type  = 'text/javascript';
    script.async = true;
    script.src   = '//api.widget.www/api.v.0.1.js'
    var node = document.getElementsByTagName('script')[0];
    node.parentNode.insertBefore(script, node);
}());
</script>

We then simply parse DOM to get a script element with data-wd-pub-id, and use the value of this attribute in widget initialization.

Note that we prefix our attibute with -wd and use data-wd-pub-id instead of data-pub-id.

Always keep in mind that you may not be the only widget provider that the publisher uses.

You don’t own the publisher’s website.

Prefix everything that’s publicly accessible (like global variables, attributes, CSS class names…) with your widget namespace, so that it does not gets overridden, or it does not conflict with the publisher’s or another widget provider’s configuration or styling data on the page.

Passing Widget Configuration Through Global Variables

Another option is to pass the configuration data as global variables. Note that we are also prefixing our variable with _wd (our widget’s namespace) to avoid conflicts.

<script>
var _wd_pubId = 1234;

(function() {
    var script   = document.createElement('script');
    script.type  = 'text/javascript';
    script.async = true;
    script.src   = '//api.widget.www/api.v.0.1.js'
    var node     = document.getElementsByTagName('script')[0];
    node.parentNode.insertBefore(script, node);
}());
</script>

Or it’s even better to use a common _wd namespace, to enable group a bunch of initialization parameters together:

<script>
if (!window._wd) {
    window._wd = {
        pubId : '123456'
    };

    (function() {
        var script = document.createElement('script');
        script.type  = 'text/javascript';
        script.async = true;
        script.src   = '//api.widget.www/api.v.0.1.js'
        var node = document.getElementsByTagName('script')[0];
        node.parentNode.insertBefore(script, node);
    }());
}
</script>

In the above code there is another checks for additional safety:

First, we check for window._wd namespace and do not do anything if it already exists. If such a namespace exists, then it means that our widget bootloader has been included before, so there’s no need to enter the same initialization flow once again.

If there’s no such window._wd namespace, then we create one and asynchronously append our bootloader script as we’ve done before.

We also have slightly changed our bootloader, adding two guard clauses:

// api.v.0.1.js

    ...

    /*
     * Ready States
     */
    var kLoaded              = 1;
    var kLoadingDependencies = 2;
    var kLoadedDependencies  = 3;
    var kBeginProcessQueue   = 4;
    var kBeginRender         = 5;
    var kComplete            = 6;

    ...

    // Publisher has forgotten to provide initialization data.
    if (!window[kWidgetAlias]) {
        log('Widget namespace cannot be found; exiting.');

        return;
    }

    // To avoid re-defining everything if the bootloader is included in
    // more than one place in the publisher's website.
    if (window[kWidgetAlias][kReadyState]) {
        log('Widget has already been loaded; exiting.');

        return;
    }

The first “if” checks for the existence of window._wd namespace, if it does not exist, then the publisher has forgotten to include their pubId parameter, so the initialization cannot continue.

The second one checks whether the widget has been loaded. So even the publisher does not include the if (!window._wd) condition on their script, the initialization flow will not proceeed, because another script will have already done the initialization.

Getting the Widget Configuration

The next thing is to get the widget configuration. That’s relatively easy, since it’s a globally accessible variable, rather than being a data attribute, hash fragment, or query string. The latter three will require traversing the DOM and doing regular expression matches. It’s also a relatively easy task, but providing a global configuration data is faster and cleaner:

    ...

    /*
     * Parameter Names
     */
    var kPublisherId = 'pubId';
    var kRandom      = 'r';
    var kVersion     = 'v';

    ...

    /*
     * Globals
     */
    var kO2Alias          = '_wd_o2';
    var kWidgetAlias      = '_wd';
    var kWidgetQueueAlias = '_wdq'

    ...

    /*
     * Get widget configuration from DOM.
     */
    function getConfiguration() {
        log('o->getConfiguration()');

        var result = {};

        result[kPublisherId] = window[kWidgetAlias][kPublisherId];

        return result;
    }

Send Configuration Data to Server

Here’s where we left last time, on the widget’s initialization flow:

    /*
     * Initialize after loading prerequisites.
     */
    function initialize() {
        log('o->initialize()');

        setReadyState(kLoadedDependencies);

        if (!window.o2) {return;}

        window.o2.noConflict(kO2Alias);

        o2 = window[kO2Alias];

        var config = getConfiguration();

        loadInitialState(config, processPostInitialization);
    }

The initialize method is called once the additional dependencies are loaded by our bootloader.

Let’s see what those dependencies are:

    ...

    /*
     * Load necessary o2.js components in noConflict mode.
     */
    function loadDependencies(callback) {
        log('o->loadDependencies(');
        log(callback);
        log(')');

        setReadyState(kLoadingDependencies);

        loadScripts(kO2Root, [
            'o2.meta.js',
            'o2.core.js',
            'o2.string.core.js',
            'o2.jsonp.core.js'
        ], callback);
    }

    checkForUpdates(versionTimestamp);
    loadDependencies(initialize);
}(this, this.document, true));

So we simply load a couple of required o2.js modules, and then call initialize function when they are all done.

Then we exit the initialization flow if we fail to load the o2.js framework for some reason:

        ...

        if (!window.o2) {return;}

        ...

Since we don’t own the publisher’s DOM, we don’t also know whether the publisher has included a different version of o2.js framework on his web page. If there is such a case, our o2.js methods may conflict with that of the publisher’s.

To avoid this we load o2.js onto window._wd_o2 namespace, in noConflict mode.

This essentially:

• Loads the current framework into a new namespace;
• Overwrites window.o2, from the cached version (the version that the publisher is using);
• And returns a reference to the new namespace.

Here’s the actual code from the o2.js source:

// o2.core.js

    exports.noConflict = def(me, 'noConflict', function(newName) {
        var name = newName || [myName, ((new Date()).getTime() +
            Math.random() * (1 << kGuidShift)).toString(kGuidRadix
            ).replace(kDecimalPoint, kEmpty)].join(kEmpty);

        window[name] = myself;

        window[myName] = window._o2_cached;

        return window[name];
    });

Then we cache the new namespace in a local o2 alias for ease of reference:

   ...
        o2 = window[kO2Alias];
   ...

Then we get the configuration and load initial state from the API server:

    ...
        var config = getConfiguration();

        loadInitialState(config, processPostInitialization);

Communicating with the Server

Since our api server is on a different domain than the publisher’s website, we are back into the cross domain boundary passing problem.

There are different approaches to create a cross domain channel from the publisher to the widget API server.

These can be listed as:

• Cross Origin Resource Sharing
• Using flash as a proxy
• Using iframe hash fragments
• Using window.name as a communication proxy
• Using HTML5 Cross Frame Messaging API
• Klein Bottle Technique
• Using a Web Proxy for Cross Domain Communication
• JSON With Padding (JSONP)

I’m sure there are even crazier techniques, that don’t come to my mind right now. We, web hackers, are really creative in finding loopholes and bending security limitations .

Each of the above techniques have their own pros and cons.

And once you manage to pass the cross domain boundary, there are other beasts waiting for your exploration (which we will talk about in the followup articles) .

If you are targeting relatively newer versions of browsers (IE10+, Firefox 3.5+, Chrome 3+, Safari4+, Opera12+), or if you are targeting mobile devices specifically, then you can safely use cross origin resource sharing as a modern way to do cross-domain communication. Otherwise you will need to provide a couple of fallback mechanisms.

Cross-Domain Communication is a hard problem to tackle with. There are also several libraries that choose an aproproate communication channel depending on the current user agent’s capabilities. The most popular one is EasyXDM. It’s a well-established library which big guys like LinkedIn, twitter, and disqus use for their cross-domain communication needs.

For the sake of our argument, we will implement our solution using JSONP, which can run in almost any user agent without needing further modifications.

Drawbacks of JSONP

However there are several limitations of JSONP. To name a few:

• The size of the data you can send is limited (around 2000 characters);
• JSONP only works with GET requests, you cannot do POST, PUT, or DELETE requests with it;
• There’s no error handling in JSONP, either you request succeeeds, or you do not get a response back. The only thing
  you can do is wait for a long enough timeout (like 10 seconds) and raise an error if there’s still no response from the server;
• JSONP requests are vulnerable to CRSF attacks, so you should take additional security precautions if you are using JSONP as a cross-domain proxy.

Having said all these, let’s see how we communicate with our server using JSONP:

    /*
     * Loads initial widget state from the server.
     */
    function loadInitialState(config, callback) {
        log('o->loadInitialState(');
        log(config);
        log(callback);
        log(')');

        o2.Jsonp.get(
            o2.String.concat(kApiRoot, kParamsPath),
            config,
            callback
        );
    }

We simply do a JSONP request to http://api.widget.www/api/v.0.1/params.

Since we’ve included o2.js JSONP module as a dependency, doing a JSONP request can be done in just a single line of code.

And here’s what happens on the server:

    /**
     *
     */
    app.get(v_0_1(route).PARAMS, function(req, res) {
        var callback    = req.param(parameter.CALLBACK);
        var publisherId = req.param(parameter.PUBLISHER_ID);

            // very primitive access control.
        if (publisherId !== '123456') {
            res.send(statusCode.NO_DATA);

            return;
        }

        var result = {
            data : 'Hello World. Hello Stars. Hello Universe!'
        };

        res.send(
            callback + '(' + JSON.stringify(result) + ');'
        );
    });

We just send a dummy JSON data to the client.

We will be doing more meaningful things, like rendering a login form, in the following article.

Read the Source Luke

If you want to play with what we have done so far, you can see the code at this github history snapshot.

Conclusion

In this article we’ve observed different ways to provide initialization parameters to our widget.

We’ve also learned varios approaches on cross-domain communication; and we implemented a JSONP tunnel as our cross domain proxy.

We mentioned various drawbacks of using a JSONP proxy, pointing to more modern alternatives such as CORS.

In the next article of the series, we’ll finally start creating something we can interact with. We’ll create an initial login screen, and work with authentication and authorization.

Until then, feel free to share your comments and suggestions.

setting a far future expires header and cache revalidation makes your widget load faster

In the former post we have outlined a broad brushstrokes initialization flow of our external JavaScript Widget:

What we did was to basically create a bootloader script that first loaded required resources asynchronously and then continued its flow with the initialization and rendering of the widget.

Currently, those initialization and rendering functions are empty placeholder functions that are yet to be implemented. We will be dealing with them in the upcoming articles.

For the time being, let’s focus our attention to a more important problem:

Versioning and Caching

For any code that you constantly upgrade an develop versioning and caching are two important things to keep in mind and to get working from day zero, if your end result will be consumed through the Internet.

For our external JavaScript Widget to be responsive, and to load faster it’s a good practice to serve the widget script with a far future expires header.

“For static components: implement “Never expire” policy by setting far future Expires header

A first-time visitor to your page may have to make several HTTP requests, but by using the Expires header you make those components cacheable. This avoids unnecessary HTTP requests on subsequent page views.

Expires headers are most often used with images, but they should be used on all components including scripts, stylesheets, and Flash components.

Browsers (and proxies) use a cache to reduce the number and size of HTTP requests, making web pages load faster.”

It seems, however, that the big guys have not done their homeworks well:

• google‘s ga.js has a 2 hour cache time;
• Facebook‘s all.js has a 15 minute cache time;
• And twitter‘s widgets.js has a 30 minute cache time.

These have negative impact on these scripts’ loading performance.

Ideally they should have a far future expiration date, say… like 10 years;
but heck, even 1 week will be a good starting point .

And There’s a Cache…

However there’s one complication with setting a far future expires header:

If you change your widget code, the users may still be using older versions of your widget.

One approach to solve this is to give your script a version number, and change the version number of the script every time you make a code update and ask the publishers (hint: not a good idea ) to update their widget loader script to reflect the change, like in the following example.

// Widget Include Code
(function() {
    var script = document.createElement('script');
    script.type  = 'text/javascript';
    script.async = true;

    // Ask the publishers to update this version number
    // every time you make a change.
    // This is not the most ideal approach since most
    // of the publishers will rarely want to change a
    // working code snippet that does not belong to them.
    script.src = 'http://api.widget.www/api.v.0.1.js'

    var node = document.getElementsByTagName('script')[0];
    node.parentNode.insertBefore(script, node);
}());

Moreover, assume that what you did is a critical security update. You cannot take it for granted that all the publishers will increment the API version in their initialization scripts.

From former experience, I can say that some most of the publishers will not bother changing that version number and will continue using the older version of the widget.

Cache Revalidation to the Rescue

We will approach the problem as follows:

• We will create a cache revalidation mechanism to push hotfixes, bugfixes, improvements and non-breaking changes to the current version of the API.
• And we will copy our script and create a new version (like api.v.0.2.js) for major improvements, and breaking changes.

Doing the latter is as easy as copying the current working file, and renaming it to give a new version number. However, implementing a cache revalidation mechanism is a bit tricky. Let’s try to work on it step by step:

Add a Far Future Expires Header

First start with setting a very long expires header for our API bootloader:

//api.widget.wwwroot/index.js

var express = require('express');
var app     = express.createServer();

var kOneYear = 31536000000;

/*
 * Make sure that the static assets have a far future expiration date.
 */
app.use(
    express['static'](
       path.STATIC_FILE,
       config.farFutureExpiration
    )
)

//where config is:
var config = {
    farFutureExpiration : {maxAge : kOneYear},

    api_v_0_1 : {
        VERSION_TIMESTAMP : '20120720135547909116'
    }
};

Cache Me, If You Can…

We will come to what VERSION_TIMESTAMP is in a minute .

When we request http://api.widget.www/api.v.0.1.js, the HTTP response headers we get are as follows:

Accept-Ranges   bytes
Cache-Control   public, max-age=31536000
Connection      keep-alive
Content-Length  6672
Content-Type    application/javascript
Date            Fri, 20 Jul 2012 20:41:55 GMT
Etag            "6672-1342801807000"
Last-Modified   Fri, 20 Jul 2012 16:30:07 GMT
X-Powered-By    Express

Where

Cache-Control   public, max-age=31536000

indicates that our bootloader script (i.e. api.v.0.1.js) will be cached for 1 year.

If we refresh the webpage and analyze the browser’s cache, we’ll see that it’s in deed true:

Cache:
Data Size       6672
Device          disk
Expires         Sat Jul 20 2013 23:43:10 GMT+0300 (EEST)
Fetch Count     4
Last Fetched    Fri Jul 20 2012 23:43:11 GMT+0300 (EEST)
Last Modified   Fri Jul 20 2012 23:43:10 GMT+0300 (EEST)

Upon that request we will also get a HTTP 304 Not Modified response from the server as expected.

Caching is an indispensable reality of the web which not only widget providers, but every web developer should know. You can read this caching tutorial for caching best practices, and RFC-2616 for HTTP status codes and header definitions.

Self-Updating Bootloader

Now it’s time to implement a self updating bootloader script:

To begin, let us add a version timestamp to the API first:

//api.v.0.1.js

(function(window, document, isDebugMode) {
    ...

    /*
     * Should match beacon version timestamp.
     */
    var versionTimestamp = '20120720135547909116';

    ...

And then pass along that version timestamp to load a cache validator beacon:

    /*
     * Revalidates cache for this bootloader script, if there's a newer
     * version available. The changes will take effect only AFTER the user
     * refreshes the page.
     */
    function checkForUpdates() {
        log('o->checkForUpdates()');

        insertScript(kApiRoot, [kBeacon, kQuery,
            kVersion,  kEquals, versionTimestamp , kAnd,
            kRandom,   kEquals, (new Date()).getTime()
        ].join(kEmpty), noop);
    }

    //Where the constants above (and some more) are defined as follows:

    var kAnd            = '&';
    var kApiRoot        = 'http://api.widget.www/';
    var kBeacon         = 'api/v.0.1/beacon';
    var kCompleteRegExp = /loaded|complete/;
    var kEmpty          = '';
    var kEquals         = '=';
    var kHead           = 'head';
    var kO2Root         = 'http://api.widget.www/lib/o2.js/';
    var kQuery          = '?';
    var kRandom         = 'r';
    var kScript         = 'script';
    var kScriptType     = 'text/javascript';
    var kVersion        = 'v';

In the above code, we basically call:

    insertScript(
        'http://api.widget.www/api/v.0.1/beacon?v=20120720135547909116&r={#}'
    );

Where insertScript simply does what its name implies:

    /*
     * Asynchronously inserts a script element to the head
     * of the document.
     */
    function insertScript(root, src) {
        var s = document.createElement(kScript);
        var x = document.getElementsByTagName(kScript)[0] ||
            document.getElementsByTagName(kHead)[0];

        s.type  = kScriptType;
        s.async = true;
        s.src   = [root, src].join(kEmpty);

        x.parentNode.insertBefore(s, x);

        return s;
    }

Thus, what we do is to request a beacon script at each widget load.

The Cache Validator Beacon

Now let us look at what the beacon script does:

//api.widget.wwwroot/index.js

    ...

    /**
     * Beacon to validate the cache of the JS API.
     */
    app.get(v_0_1(route).BEACON, function(req, res) {
        res.header('Content-Type', 'text/javascript');

        setShortExpiresHeader(res);

        var versionTimestamp = v_0_1(config).VERSION_TIMESTAMP;

        var requestedVersion = req.param(
            parameter.VERSION,
            versionTimestamp
        );

        if (requestedVersion !== versionTimestamp) {
            res.render(template.UPDATE_SCRIPT, {
                iframeUrl : v_0_1(url).UPDATE_IFRAME
            });

            return;
        }

        res.send(204);
    });

If the requested version timestamp is identical to the what we sent, then it means that we are using the most up-to-date bootloader script, we don’t need to update the cache; and the beacon simply sends a HTTP 204 (no content) response.

Otherwise (i.e. if the versions don’t match) then we conclude that the user’s cache is stale and we inject a simple JavaScript in the beacon as follows:

(function() {
    var update = function() {
        if (!document.body) {
            setTimeout(update, 500);
            return;
        }

        var loader = document.createElement('iframe');

        loader.style.display = 'none';
        loader.src = '#{iframeUrl}';

        document.body.appendChild(loader);
    };

     update();
})();

Where we append a hidden IFRAME element to the document.

And the contents of that IFRAME will be (in Jade syntax):

    !!! 5
    html(lang='en')
        head
            script(src= apiBootstrapUrl)
        body(lang='en')
            script
                if (!window.location.hash) {
                    window.location.hash = 'checked';
                    window.location.reload(true);
                }

So the hidden iframe simply reloads the api bootloader script once more, where the true parameter forces the newest script to be retrieved from the server. Which will update the browser’s cache.

Note that the changes will be effective upon the next retrieval of the script:

The most up-to-date version of the script will be served to the end user once she reloads the publisher’s website, or navigates to another page.

This concludes our cache revalidation flow.

Widget Sanity Check

Another issue that you may coincide as a widget developer is that the publisher may unintentionally include your widget bootloader script more than once, or worse she can include several different versions of your widget bootloader script in different parts of the page.

To avoid redefining of resources a sanity check on the top of the widget is required.

//api.v.0.1.js

...

// To avoid re-defining everything if the bootloader is included in
// more than one place in the publisher's website.
if (window._wd) {
    return;
}

window._wd = {};

As always, we do our best to keep the global namespace clean. Note that window._wd is the only accessible object outside the module.

Our assumption here is that, nobody other than the widget provider uses the window._wd namespace, and window._wd namespace solely belongs to the widget provider.

So if a publisher redefines window._wd at their own peril, they should know that our widget will not work properly.

Later, in the upcoming tutorials, it will be our widget namespace that we will export (or expose) public methods of the module to the outside world.

Load Prerequisites

Another thing we would do is to load required assets (such as helper libraries) before continuing with the widget’s initialization flow.

We do this in loadPrerequisites function as follows:

    /*
     * Load necessary o2.js components in noConflict mode.
     */
    function loadPrerequisites(callback) {
        loadScripts(kO2Root, [
            'o2.meta.js',
            'o2.core.js'
        ], callback);
    }

Where loadScripts is:

    /*
     * Loads an array of scripts one after another.
     */
    function loadScripts(root, ar, callback) {
        // Populate global script queue.
        scriptQueue = ar;

        loadScript(root, scriptQueue.shift(), callback);
    }

And loadScript is:

    /*
     * Loads the given script.
     * <strong>callback</strong> is the function to be executed after
     * there's no resource left to be loeded next.
     */
    var loadScript = function(root, src, callback) {
        var s = insertScript(root, src);

        function processNext() {
            loadNext(root, loadScript, callback);
        }

        s.onreadystatechange = function() {
            if(kCompleteRegExp.test(s.readyState)) {
                processNext();
            }
        };

        s.onload = function() {
            processNext();
        };
    };

So we inject an array of script one after another until all of the scripts are loaded and then execute a callback.

This callback continues our widget initialization flow, which is the subject of the next article in this series.

Read the Source Luke

You can download the source code of what we’ve done so far from this github history snapshot.

Conclusion

In this article we’ve built upon what we’ve learned last time.

• We’ve implemented a cache validator beacon, that refreshes the cache whenever there’s a new version of the script;
• We’ve implemented a way to prevent multiple widget initialization if the script is included more than once on the page;
• We’ve also implemented a mechanism to preload required assets before continuing with the widget initialization flow.

In the upcoming article, we will continue our journey with initializing and rendering our widget.

Until then, feel free to share your comments an suggestions.

initial setup

In the former part of the series we had a brief introduction on challenges in developing JavaScript widgets.

In this article, we’re going to continue from where we left: We will start by installing necessary instruments to develop and test our code.

First Things First

Before we start actual coding, we need to install node.js.

You can download and install the most recent version of node.js from http://nodejs.org/. The installation process is pretty straightforward.

Note that this article is not by no means a node.js tutorial. I assume you know enough node.js to follow through. If not, there are excellent tutorials around to get you started on node.js.

Then we’ll install express.js to have a higher-level abstraction on top of node.js and to be easily manage cookies, session state, templates… and the like.

To install express.js:

$ cd /path/to/your/projects/folder/
$ npm install express

While we’re at there let’s install jade template engine too:

$ cd /path/to/your/projects/folder/
$ npm install jade

Create Development Servers

The next thing we are going to do is to create development environments.
We’ll have two server instances:

• http://api.widget.www/ will be our widget provider;
• And http://publisher.www:8080/ will be our publisher website.

We’ll simply add these lines to our /etc/hosts file.

127.0.0.1 api.widget.www
127.0.0.1 publisher.www

Project Folder Structures

Initially, there’s nothing fancy:

Here’s the initial folder structure of our api widget provider wwwroot:

Widget API Website Root

And here’s the initial folder structure of our demo publisher website:

Demo Publisher Website

You can view this initial version of the sites at this github history snapshot. We will be adding more files and changing the folder structure as proceed to the next articles of this series.

Widget Server

The node.js server of the widget API website is quite simple:

var express = require('express');

var app = express.createServer();

app.use(express.static(__dirname + '/static'));

app.get('/api/v.0.1/login', function(req, res) {
    res.send('hello authentication');
});

app.listen(80);

We simply server everything under the api.widget.wwwroot/static folder as static content.
Let’s test that it works.

:api.widget.wwwroot$ sudo node index.js

And then when we browse to http://api.widget.www/api.v.0.1.js we’ll see that our Widget API JavaScript is served without problem.

Publisher Server

And here’s the publisher‘s node.js server index file:

var express = require('express');

var app = express.createServer();

app.use(express.static(__dirname + '/static'));

// Set path to the views (template) directory
app.set('views', './views');

app.get('/', function(req, res) {
    res.render('index.jade');
});

app.listen(8080);

And we have a very simple Jade template to be rendered when requesting the site root (“/”):

# views/index.jade
!!! 5
html(lang='en')
    head
        meta(charset='utf-8')
        title= 'Publisher Website'
        link(rel='stylesheet', media='all', href='/css/main.css');

        script
            (function() {
                var script = document.createElement('script');
                script.type  = 'text/javascript';
                script.async = true;
                script.src   = 'http://api.widget.www/api.v.0.1.js'
                var node = document.getElementsByTagName('script')[0];
                node.parentNode.insertBefore(script, node);
            }());

    body(lang='en')
        section#Main
            h2 Welcome to Publisher Website

If this were a real publisher site, the template folder structure would have been more complicated possibly with a handful of partial templates. Though this basic page is good enough for our initial setup.

Let’s run

:publisher.wwwroot $ sudo node index.js

on publisher.wwwroot.

When we browse http://publisher.www:8080/ we’ll get a very basic welcome page.

Note that we are listening to port 8080 for the publisher website because our api server is already listening to port 80. In a production environment we won’t need ts.his too.

Asynchronous Loading of the Bootsrapper

On the publisher page we include our widget initialization script as follows:

(function() {
    var script   = document.createElement('script');
    script.type  = 'text/javascript';
    script.async = true;
    script.src   = 'http://api.widget.www/api.v.0.1.js'
    var node     = document.getElementsByTagName('script')[0];
    node.parentNode.insertBefore(script, node);
}());

This structure is known as async loading pattern.

This usage is better than simply including the script with

<script src="http://api.widget.www/api.v.0.1.js"></script>

because the latter will block downloading of the resources and rendering of the page until it’s fully downloaded and executed.

If you remember from the introduction article, as widget developers we don’t own the publisher’s website. Per contra, we are like thieves wandering in their home. So the less noise we make, and the less trace we leave, the better.

That’s why our widget should initiate itself with minimal impact to the publisher’s performance: If you want to have faster and responsive website, loading third-party resources asynchronously is a best practice to follow.

However keep in mind that the publisher should have the liberty to load your script inline (as in the second example) if she likes. If she knows the consequences of it, and if she has a reason to do so, we shall not prevent her from doing it. As before, we do not own publisher’s DOM. The publisher can do anything (imaginable) and we should be flexible enough to support every possible scenario.

Widget Code

Here comes the fun part . Let’s see our widget code:

(function(window, isDebugMode) {
    'use strict';

    var version = 'v.0.1';

    function isArray(item) {
        return window._wd_o2.isArray(item);
    }

    function checkForUpdates() {
    }

    function render() {
    }

    function execute() {
    }

    var queue = {
        items : [],

        push : function(item) {
            log('o->queue.push()');

            execute(item);
        }
    };

    var processQueue = function() {
        var q = null;

        if (window._wdq && isArray(window._wdq)) {
            q = window._wdq;

            while (q.length) {
                execute(q.pop());
            }
        }

        processQueue = window._wd_o2.nill;
    };

    function initialState_ready(state) {
        render(state);

        processQueue();

        window._wdq = queue;
    }

    function loadInitialState(config, callback) {
        config = null;

        callback({});
    }

    function getConfiguration() {
        return {};
    }

    function initialize() {
        var config = getConfiguration();

        loadInitialState(config, initialState_ready);
    }

    function getPrerequisites(callback) {
        callback();
    }

    checkForUpdates(version);
    getPrerequisites(initialize);
}(this, true));

I’ve removed comments and console.log calls to make the code look cleaner.

The first thing we see is we’re utilizing the module pattern to hide our widget’s private static context and functions from the publisher. We should not pollute global namespace as widget development best practice.

Widget Initialization Flow

Here’s a broad overview of the initialization flow of the widget:

Widget Initialization Flow

Most of the methods are blank, left to be implemented in the followup articles. However, the initialization flow will more or less remain the same.

In the diagram above, the gray boxes represent async operations (such as loading a beacon, or making a resource request to the server).

Let’s observe each of the blocks one by one:

BootLoader Cache Validation

• We start by running two requests in parallel:
  • We check for new versions of our bootloader script and update our browser cache if we find out that our version is obsolete (so that the user will load the new version from cache the next time she visits the publisher site).

    This will help us solve caching issues, and enable us serve a faster-loading widget.

  • And, in parallel, we start our regular widget initialization flow.

Widget Initialization

In our widget initialization flow:

• We get additional resources (such as helper libraries) in the getPrerequsities function.
• After we retrieve all required resources we get widget configuration data from DOM and request initial state data of our widget from the server (this is basically the “Model” data that’ll be used to render the “View” of the widget)
• We render our widget’s initial view with the state data we acquired.
• We process any jobs in the job queue that has been registered by the publisher to be run as soon as widget is available (this job queue works similar to the _gaq of google analytics; you can register jobs, and they will be run in order whenever the widget is able to respond).
• We override the job queue object with an Array-like implementation.

Fear not, if these flows seem a bit complicated. Most of these methods are currently abstract anyway, and we’ll be implementing them all in the following articles. And everything will be crystal-clear. Stay tuned .

Conclusion

In this article we’ve learned how to set up our environment for JavaScript widget development and testing.

We’ve also had an abstract overview of our widget’s cache-revalidation and initialization flow logics.

In the next sections, we will be filling out some of these empty functions with real code.

Until then, feel free to share your comments and suggestions.

a familiy of widgets of different sizes

This is the beginning of a new article series where we will see best practices, common pitfalls, and “how to”s on creating JavaScript Widgets for external sites.

Throughout the examples we will be using:

• node.js for the server;
• On top of node, expres.js framework as a higher-level abstraction;
• mongodb for the persistence layer;
• And borrow some code from o2.js for client-side DOM interaction and event delegation.

But before we start let’s agree on the definitions:

What is a JavaScript Widget?

A JavaScript Widget can be defined as any type of functionality that is loaded from an external site (widget provider), to different sites (publishers) that include the widget code via a simple JavaScript include.

A JavaScript Widget may or may not include user interface components. An exmaple of a JavaScript Widget that does not contain a GUI counterpart is google analytics code that most of us is familiar with. Examples for widgets that include GUI components are the “vote“, “share“, “like“, and “add comment” widgets of social networks such as twitter and Facebook.

During the article series we will use the terms publisher and widget provider frequently. So it’s important to understand the distinction:

A widget provider is the source website where anyone can get a widget to their sites by adding a few lines of JavaScript that the provider provides (hence the name “provider“).

A publisher is a website that publishes these widgets (hence the name “publisher“) to enhance its functionality. The publish operation is nothing but simply adding the JavaScript snippets that the provider implements.

Stateful Widgets vs Stateless Widgets

Just like a regular website a JavaScript Widget can have state information (i.e. persist user-specific data in a session object on the widget provider servers); or it can be stateless. A stateless widget does not store data in widget provider‘s session.

To have a stateful widget, you will need third-party cookie support; which is one of the many challenges listed below.

Challenges in JavaScript Widget Development

There are many challenges one can face when developing a JavaScript Widget. The list below only covers the tip of the iceberg:

Versioning can be a Hassle

You need to define a consistent way to distribute different versions of your code to publishers. So that when you add a new functionality to your existing widget API, it does not break the working code on the publisher sites.

You have to enable different versions of your widget, so that you can introduce major changes in the newer versions of the API, and let publishers gradually upgrade their widgets at their convenience.

You Don’t Own the Execution Environment

When you don’t own the DOM, really careful you should be.
– Yoda

The environment that the widget will run is totally unknown to the widget provider. Worse, the widget provider has almost zero control over that environment.

The execution environment can be a mobile device, a tablet, a power pc, a developer machine, an emulator or simulator, a really old browser… Our widget script has to run equally well in a wide spectrum of doctypes, platforms, and configurations (where some of these configurations can even be overridden by the user).

Moreover the publisher may include your script inside head…/head, or at the middle of body, or just before /body. And the publisher may include the widget script multiple times either intentionally, or by mistake.

Execution Environment is Shared

Developing a JavaScript Widget is like doing business in a public dorm, where the kitchen, bedroom and the bathroom is shared: “Anything” can happen.

Not only you don’t own the execution environment, but you also share that environment with guys whom you have no idea where they come from.

Our beloved widget has to happily coexist with other widgets, user-defined scripts, and copy/pasted JavaScript snippets (most of which are a pile of “junk”). Some of these script may even override standard JavaScript objects and methods that you rely on.

Therefore our widget has to do its best to:

• Keep the global namespace clean and be a self-contained piece of code.
• And try to protect itself from badly-written scripts that coexist with it. If not taken proper precautions, these scripts may adversley affect the behavior and performance of our widget.

Passing the Cross Domain Boundary

Since our widget is hosted on an external site, the good old cross domain restrictions apply.

We need to find ways to create a two-way communication channel between the publisher (www.publisher.com) site and the widget provider site (api.mywidget.com). We’ll explore different techniques to accomplish this in the followup articles.

Hack It Like A Thief: Act, but Don’t be Seen

That’s a corollary to the “you don’t own the context” item above.

Since you are a guest, behave like one.

The less our widget changes how provider site works, the better.

And there are several things we can do to decrease the effect of our widget script on the publisher site:

• Use a single namespace;
• Prefix common classes, ids, and attributes;
• Utilize async script includes;
• use lazy loading patterns to delay execution of code that will not be needed immediately.

We’ll come to each and every single one of those items in the followup articles of this series.

Problems With Persisting and Retieving State

In a good old web application, the application state is persisted on the server in the Session object; and genrally “cookies” are used to pass a session ID around to share the session state while browsing different pages.

Since our widget code lives in a different domain than the publisher site, the client may impose restictions on saving cookies as well. By the time of this writing, Safari browser, for instance, disables third party cookies by default.

For our widget to share and persist state information with the widget api server we need to use third party cookies. And since we have no control over the environment our widget script executes (see above), our widget script may be executing in an environment where third-party cookie reading or writing may not be allowed.

There are ways to bypass this restriction too, which we’ll cover in the upcoming articles.

What’s Next?

That’s was a brief introduction on the things to watch out while architecting a JavaScript Widget.

We will explore everything mentioned in this post (and more) in the followup articles.

In the next of the series, we will be preparing our development environment. The interested may get their feet wet by starting to install node.js.

…

Do you develop external widgets to be distributed to a variety of publisher websites?

I’d love to hear your experiences.

Feel free to share your ideas and suggestions as comments.

o2.js + node.js + npm

Unless you have been living in a cave for the recent two years, you should known that node.js is a highly efficient platform built on Chrome’s JavaScript runtime for easily building fast, scalable network applications, by using an event-driven, non-blocking I/O model.

node.js is lightweight and efficient; perfect for data-intensive real-time applications that run across distributed devices.

And if you are a regular reader of this blog, then you know about the #JFDI movement, and its successor project sarmal (on github).

sarmal is an MIT-licensed web project that uses Microsoft IIS and node.js. It’s doing it the Jedi way in the evil empire

Why Export o2.js as a node.js module?

o2.js has helpful functional programming, string manipulation, and collection manipulation methods, and a unit testing library. Thus, exporting it to as an npm module may be useful to the node.js community.

And since the sarmal uses node.js, I thought it’d be cool to export o2.js as a node.js module. Then I could be able to use it on the server-side components of sarmal.

Let’s step by step analyze this node.js module creation process; how to modify o2.js to be exported as a node.js module, and how to publish it to the npm registry:

Install npm

npm stands for node package manager. It is the de-facto package manager for node.js modules.

The first thing we’ll need is to install npm. The easiest way to do it is to install node.js, because npm comes automatically with node.js.

Add A User

After installing npm, we’ll need to add a user with npm adduser command.

Volkans-MacBook-Pro:o2.js volkanozcelik$ npm adduser
Username: (volkan) 
Email: (volkan@o2js.com) 

Create a Package Descriptor (package.json)

Then to publish our package to the npm registry, we’ll need to describe the package meta data in package.json:

The package.json is the meta descriptor of our node.js module.

Here’s the package.json we’ll be using for o2.js:

{
    "name"         : "o2",
    "version"      : "0.25.8",
    "author"       : "Volkan Özçelik <volkan@o2js.com>",
    "description"  : "Node.js module export for o2.js JavaScript Framework",
    "preferGlobal" : "true",
    "main"         : "./o2.js/index.js",
    "keywords" : [
        "o2.js",
        "collection",
        "array",
        "ajax",
        "functional",
        "string",
        "unit test",
        "validation",
        "helper"
    ],
    "repository" : {
        "type" : "git",
        "url"  : "https://github.com/v0lkan/o2.js.git"
    },
    "license" : "MIT",
    "engines" : { "node": "*" }
}

You can see the most up-to-date version of o2.js package.json at github.

One important thing give special care is the version number. Ideally, any change in the minor version (i.e. 0.25.8 to 0.25.9) should be non-breaking; and any change in the major version (i.e. 0.25.8 to 0.26.0) will be breaking.

The package.json can have many other fields, which are annotated in package.json.jit.su.

Merge Script

The next thing is to write a simple shell script to merge o2.js modules into a single file to be exported (which will be the ./o2.js/index.js we see in the package descriptor).

# batch/node_export.sh
#
# o2.js NodeJS Export Script.
# Exports entire o2.js to o2.js/index.js, as a NodeJS Module, by properly
# ordering and merging the files.

echo 'Starting node.js export...'

cat \
../o2.js/o2.nodejs.header.js \
../o2.js/o2.meta.js \
../o2.js/o2.core.js \
../o2.js/o2.string.core.js \
../o2.js/o2.string.encode.js \
../o2.js/o2.string.strip.js \
../o2.js/o2.string.transform.js \
../o2.js/o2.event.constants.js \
../o2.js/o2.event.core.js \
../o2.js/o2.event.extend.js \
../o2.js/o2.ajax.core.js \
../o2.js/o2.ajax.extend.js \
../o2.js/o2.ajaxstate.core.js \
../o2.js/o2.ajaxcontroller.core.js \
../o2.js/o2.validation.core.js \
../o2.js/o2.validation.regexp.js \
../o2.js/o2.method.core.js \
../o2.js/o2.method.event.js \
../o2.js/o2.method.inherit.js \
../o2.js/o2.method.repeat.js \
../o2.js/o2.method.timer.js \
../o2.js/o2.method.transpose.js \
../o2.js/o2.collection.core.js \
../o2.js/o2.convert.core.js \
../o2.js/o2.cookie.core.js \
../o2.js/o2.date.core.js \
../o2.js/o2.debugger.core.js \
../o2.js/o2.dom.class.js \
../o2.js/o2.dom.collide.js \
../o2.js/o2.dom.constants.js \
../o2.js/o2.dom.coordinate.js \
../o2.js/o2.dom.core.js \
../o2.js/o2.dom.style.js \
../o2.js/o2.dom.dimension.js \
../o2.js/o2.dom.form.js \
../o2.js/o2.dom.load.js \
../o2.js/o2.dom.modify.js \
../o2.js/o2.dom.ready.js \
../o2.js/o2.dom.scroll.js \
../o2.js/o2.dom.traverse.js \
../o2.js/o2.effect.core.js \
../o2.js/o2.object.core.js \
../o2.js/o2.jsonpstate.core.js \
../o2.js/o2.jsonpcontroller.core.js \
../o2.js/o2.jsonp.core.js \
../o2.js/o2.querystring.core.js \
../o2.js/o2.sortdelegate.core.js \
../o2.js/o2.supports.core.js \
../o2.js/o2.template.core.js \
../o2.js/o2.timer.core.js \
../o2.js/o2.try.core.js \
../o2.js/o2.unit.core.js \
../o2.js/o2.nodejs.footer.js \
> ../o2.js/index.js

echo 'Completed node.js export.'

There are two files that draw immediate attention o2.node.js.header.js and o2.nodejs.footer.js. These files are not a part of o2.js Framework. They are helper files to help export o2.js classes to the node.js runtime.

o2.node.header.js

Since node.js runtime does not have DOM objects and methods, we create a very basic DOM mockup in o2.node.header.js to make the framework function properly:

if (!document) {
    var document = {
        getElementsByName      : function() {},
        createElement          : function() {},
        createTextNode         : function() {},
        createDocumentFragment : function() {},
        getElementsByTagName   : function() {},
        styleSheets            : []
    };

    this.document      = document;
    this.setTimeout    = setTimeout;
    this.setInterval   = setInterval;
    this.clearTimeout  = clearTimeout;
    this.clearInterval = clearInterval;
    this.escape        = escape;
    this.navigator     = {
        userAgent : 'o2.js DOM Mockup for Node.JS'
    }
    this.self          = {};
    this.Image         = {};
    this.scrollTo      = function() {};
    this.location      = {};
}

When we are using o2.js in a browser environment the global this reference will refer to the window object; and document will be an alias to window.document. However inside a node.js module there’s neither window, nor document. We’ll need to fake them, so that o2.js self-consistency checking mechanism does not to throw errors.

This may even be the starting point of a node.js DOM mockup module.

o2.nodejs.footer.js

And the o2.nodejs.footer.js simply consists of one line that exports the entire o2 namespace:

exports.o2 = this.o2;

Without this exports statement, o2 will not be available to the consumers of this module.

After running batch/node_export.sh our module will be ready to be published to the npm registry.

Publish the Module to npm

To publish our package to the npm registry we simply execute npm publish from the project root (where we the package.json file is located as well).

A typical output will be as follows:

Volkans-MacBook-Pro:o2.js volkanozcelik$ npm publish
npm http 200 https://registry.npmjs.org/o2/-/o2-0.25.8.tgz/-rev/16-6c390d4126b7a1898094568951e899ab
- o2@0.25.8
npm http PUT https://registry.npmjs.org/o2
npm http 409 https://registry.npmjs.org/o2
npm http GET https://registry.npmjs.org/o2
npm http 200 https://registry.npmjs.org/o2
npm http PUT https://registry.npmjs.org/o2/0.25.8/-tag/latest
npm http 201 https://registry.npmjs.org/o2/0.25.8/-tag/latest
npm http GET https://registry.npmjs.org/o2
npm http 200 https://registry.npmjs.org/o2
npm http PUT https://registry.npmjs.org/o2/-/o2-0.25.8.tgz/-rev/18-e4db0577a9ff287373344943a58c2587
npm http 201 https://registry.npmjs.org/o2/-/o2-0.25.8.tgz/-rev/18-e4db0577a9ff287373344943a58c2587
+ o2@0.25.8

This will make o2.js available on the npm registry

o2.js on npm registry

You can also search npm registry from the command line:

Volkans-MacBook-Pro:o2.js volkanozcelik$ npm search o2.js
NAME DESCRIPTION                                           AUTHOR   DATE              KEYWORDS
o2  Node.js module export for o2.js JavaScript Framework  =volkan  2012-06-27 22:42  o2.js collection array ajax functional string u
npm http GET https://registry.npmjs.org/-/all/since?stale=update_after&startkey=1340838514000

Use o2.js as a node.js Module

To use o2.js inside our node.js application we’ll need to install it first:

Volkans-MacBook-Pro:~ volkanozcelik$ npm install o2
npm http GET https://registry.npmjs.org/o2
npm http 200 https://registry.npmjs.org/o2
npm WARN prefer global o2@0.25.8 should be installed with -g
o2@0.25.8 ./node_modules/o2

And to use o2.js in our node.js application, we’ll need to require it first.
Here’s a very basic example that does String formatting:

var o2 = require('o2').o2;

var stuff = o2.String.format('Hello {0}!', 'Universe');

// stuff will be 'Hello Universe!'

Summary

In this short tutorial we’ve seen that it’s really easy to create a node.js module out of a JavaScript framework;
We’ve seen how to install npm;
We’ve also seen that it’s really easy to publish our newly created node.js module to the npm registry;
We’ve also briefly looked at how to search the nmp registry either from search.nmpjs.org or from the command line;
And then how to install and use a node.js module from the nmp registry.

Although this tutorial was mainly focused on o2.js, you can use it as a starting point to export your own libraries as node.js modules.

Was it helpful?

Feel free to share your ideas and suggestions.