CommonJS Reference Framework (version 0.5, this post is still under edit!)

Creative Commons License
CommonJS Reference Framework by Sander Tolsma et al. is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.

During the last 4 months I started to get involved in the world of standards for JavaScript Modules and Packages. Something I think is very important if JavaScript wants to rule the world in the next 5 years ;-) After some investigation I found out that there is a group of developers that is interested in the same thing and they already work together in a sort of Development Group called CommonJS. The only problem was that is was very difficult to get a clear overview of the direction and interaction of the current 'standards' of the group. That's why I started this post: a reference framework describing the CommonJS standards.

What is CommonJS

JavaScript is a powerful object oriented language with some of the fastest dynamic language interpreters around. The official JavaScript specification defines APIs for some objects that are useful for building browser-based applications. However, the spec does not define a standard library that is useful for building a broader range of applications.

So, what is holding JavaScript back from world domination?

  • JavaScript has no module system. To compose JavaScript scripts, they must be either managed in HTML, concatenated, injected, or manually fetched and evaluated. There is no native facility for scope isolation or dependency management.
  • JavaScript has no standard library. It has a browser API, dates, and math, but no file system API, much less an IO stream API or primordial types for binary data.
  • JavaScript has no standard interfaces for things like Web servers or databases.
  • JavaScript has no package management system that manages dependencies and automatically installs them, except JSAN (not to be confused with JSON), which falls short for scope isolation.

The CommonJS API will fill that gap by defining APIs that handle many common application needs, ultimately providing a standard library as rich as those of Python, Ruby and Java. The intention is that an application developer will be able to write an application using the CommonJS APIs and then run that application across different JavaScript interpreters and host environments. With CommonJS-compliant systems, you can use JavaScript to write:

  • Server-side JavaScript applications
  • Command line tools
  • Desktop GUI-based applications
  • Hybrid applications (Titanium, Adobe AIR)
  • Browser based Applications (User Agents)

To get order in the chaos the CommonJS Reference Framework is defined which describes the interaction between the basic CommonJS layers:


In this picture three main blocks can be depicted:

  • The Core CommonJS System with the core implementation of Modules/Packages and Context,
  • the (extra) Module Environment where the module code every day developers are writing is running
  • and the, with plugins extensible, asynchonous loader environment with the Specific Loader System and the Transport system to transport information/code from (remote) servers.

In the next chapters the high level use of those layers will be described.

Core CommonJS System Layer

This is the basic CommonJS layer with stores for Module, Package and Context definitions and functions for manipulating those stores and the host environment. The total CommonJS Environment consists of an ECMAScript execution environment embedded in an application such as a web browser or scripting host, known as the host environment. In an addition to the ECMAScript interpreter, the CommonJS environment provides a facility for loading and executing ECMAScript known as the CommonJS module and package system, and bootstrapping facilities that allow us to initialize and execute CommonJS scripts.

In this layer several specific subsystems can be defined, depending on the required level of implementation of CommonJS core elements (Module, Package, Context). Those subsystems work together as a sort of Matryoshka doll. The lowest level of implementation of the CommonJS System Layer is the implementation of the Module System. The second level is the use of Packages to group modules in 'package environments' for easy deployability and shareability. The last layer is the use of Contexts to be able to implement separate module/package environments in one host

The notion of Context is not yet defined or even discussed in the CommonJS group!! Just defined to show how easy it can be to add new functionality to the Core System Layer.

The relation between the host environment and the core CommonJS elements are depicted in the next image:

Add distinction that in the same context environment module exports of the same module in the same package are identical. This is not the case between module exports of the same package in another context environment!! So between context environments only the Factory Function is reused not the created exports!!

If we look at implementing those core elements in the Core Systems Layer we get the following picture:

The Context System

In the host at least one (secure) contexts will exist. A context has one (and only one) context environment. A context environment can be seen as a configurable package and its package UID is 'commonjs.org'. In the host initialization phase at least one context will be defined. The standard configuration of that context environment will be defined in the 'Context' standards. The CommonJS Context System performs x main tasks:

  1. Creates separate module/package environments called context environments
  2. Makes it possible to switch between contexts
  3. Can create new context environments
  4. Can delete a context environment
  5. Can ??

The context environment (with UID = 'commonjs.org') has a defined module namespace root. This is the location of which the module identifiers are resolved. This location can be a root in a database, a directory or folder in a file system or a URI to a remote or local location with context environment modules. See the newest Context/x.x specification for more information about the CommonJS Context System.

The Package System

The CommonJS package system performs x main tasks:

  1. Maps modules within packages to defined paths
  2. Stores Package definition files in a local store
  3. ??

Package Definition Store

In the Package Definition Store the static and dynamically obtained Package Descriptors are stored for later use by the other Systems. The data in the store can be used to get the Mappings field of a Package Descriptor or other package meta information.

Registry Store

In the Registry Store static and dynamic learned Package UID to Package URI translations are stored for later use by the Generic Loader System. The data in the store can be used to get a location for a package only referenced by a Package UID in for example the Mappings field of a loaded Package Configuration or a call to the mappings function of the Module API.

I've put the Registry store in this system but it could be argued to implement a seperate Registry System like Context/Package/Module Systems

The Module Store System

The CommonJS module system performs x main tasks:

  1. Provides module Factory Functions and dependency id's from the Module Stores to the core system on request
  2. ??





The Core System

The CommonJS Core system performs x main tasks:

  1. Provide modules to the Module and Host Environments and memoize their exports
  2. Resolve dependencies between modules and if needed dynamically loads those dependant modules by using generic loader functions
  3. Allow host scripts to access properties exported from modules which have been provided to the environment

The Generic Module System

The Generic Module System performs x main tasks:

  1. Provides translation functions between local module environment and the Core CommonJS System Layer (relative id to system wide canonical id translation)
  2. ??

The Generic Loader System

<Add generic description>
The Generic Loader System performs x main tasks:

  1. Calls Module Store System to store module definition information that is send by plugins from the Specific Loader System through the Loader API (memoize function).
  2. By request from the Generic Module System, Package System or Module Store System dynamically loads modules, package definitions or registry information by using the Loader API towards the Specific Loader System plugins (moduleLoad, packageLoad, registryLoad functions).
  3. After a {module/package/registry}Load ready callback call, loads dependent modules that are not loaded (or are not loading yet) defined in the added module definition information calls in between the {module/package/registry}Load call and the {module/package/registry}Load ready Callback by using the Loader API towards the Specific Loader System plugins. See the GenericLoader specifications for more information.

Loader API Description:

  • <something>.id(id) : Accepts a module identifier, returning the corresponding canonical module.id
  • <something>.uri(id) : Accepts a module identifier, returning a URI as a string which canonically identifies the resource
  • <something>.memoize(id, dep array, module factory function) : Accepts a canonical module id, dependency array and a module factory function and stores that information in the dynamic module store.
  • <something>.isMemoized(id) : Accepts a canonical module id, returning true if that module is available in one of the Module Store System Stores
  • <something>

The module.registerPlugin function accepts an object with the following possible properties referring to an function:

  • loadModule(id, readyCallback) :
  • loadPackageDescriptor(packageUri, readyCallback):
  • QueryRegistry(packageUID, readyCallBack):

The registerPlugin function returns an object describing access to the Loader API functions.

Module Environment Layer

To date, client side JavaScript has generally been able to get away with something as simple as the <script> tag and no standard way to do namespaces. On the server, it's a bit different because you're more likely to use more libraries and you can potentially load up a lot of code. Having a basic system for loading code and encouraging the use of namespaces to avoid unintentional interference will underlie everything else.

Minimal requirements:

  • Private module code scope
  • Possibility to export module API (i.e exports) to other modules
  • Facility to load module API’s (i.e. exports) from other modules
  • Easy straightforward module referencing nomenclature
  • Declaring dependencies on other modules by referencing them in export definition state

A CommonJS module code section (Module Environment) is the group of expressions contained within a module factory function. Any series of ECMAScript expressions that can be parsed as an ECMAScript FunctionBody may be used as a module code section.
CommonJS modules may be stored as any type of resource, including local files, remote URLs, database entries, and so on. The storage mechanism of a module is relevant only to the environment’s built-in module provider and not to the defined module itself.

The module environment has at a minimum access to three APIs

  • The require API - The API to the module, package and context providers
  • The exports API - The API to the modules exports
  • The module API - The API to the generic module and package/module loader functions

Those three APIs are injected as free variables in the Module Environment via arguments in the Module Factory Function (browser hosts) or via other kind of implementations (scripting hosts).

Specific Loader System Layer

Todo!!







How are Specific Loader System Plugins with specific loader functions coupled with the Core CommonJS System (Loader API/Generic Loader System). Think of plugins being coupled from within modules or plugins coupled from the extra module environment i.e. basic global script code

Transport System Layer

This layer defines methods/protocols for transmitting module source code, package definitions, registry information or repository information from a (web) server to a scripting host (like a browser page).

Currently defined specifications are for example: Packages/AsynchronousDefinition or Packages/Transport for transporting package metadata from a remote server to a host.





Describe some basic Transport Protocols in a high level fashion. Details must be specified in specific Transport specification documents.

Flow Examples

High level Examples of flow cases. For example Module dynamically requests another module but that module is not available in exports and Module Store System so it needs to be loaded through the loader etc etc.

Glossary

Here all definitions need to be inserted. Please help with definitions!!!

References



General Article To do:

What needs to be investigated and added to this article:

  • Description of every block as depicted in the overview diagram and a reference to the CommonJS standards describing that block.
  • What to do with Module and Package context ??
  • Package Manager Functions etc etc