Overview of desire, a software knowledge and distribution system

You can use desire in the same manner as ASDF-INSTALL, but in a somewhat more friendly manner:

• no non-essential packages need to be installed, because desire has a concept of central system of a module³;
• you can ask it about available modules in an apropos-like fashion, because of the availability of an explicitly declared nomenclature;

Additionally, you can use it to track non-lisp upstreams – without dependency handling, for obvious reasons.

There are two modes of operation, a participating one and a non-participating one.

The latter requires almost no involvement on upstream's part – not even use of desire, whereas the former assumes control over the definition of the corresponding part of the global upstream namespace.

In either case you get:

• your modules and their dependencies are globally declared and easily available to end-users;
• buildbot-style testing on http://feelingofgreen.ru/desire-waterfall ⁴.

You know you're in this category when you want to do things like:

• programmatically generate/maintain/test changes/branches across a set of modules, like the automated XCVBifier does, perhaps for facilitating coordinated release making;
• programmatically populate the definition namespace (perhaps for a web-sphere where people could register and manipulate their projects);
• extend the buildbot infrastructure, perhaps adding things like automated package generation phase;

I won't go into details at this point, but I believe that desire provides enough foothold to make all that feasible, if not easy-ish.

Like with any underspecified area, desire employs a degree of heuristics here and there -- and that's where we need specifications to make things predictable and reliable.

Absolute requirements at this point are a POSIX shell, git and SBCL.

Besides the obvious download and compilation/load steps, desire requires an initialisation procedure parametrised with a source code storage area, in order to become operational.

All these steps can be carried out:

• manually;
• in a semi-automated manner, with assist of ASDF-INSTALL;
• in a completely automated manner, using the bootstrap script.

Before we proceed, a crucial concept of the storage area must be discussed.

The storage area is a writable directory whose writable subdirectories fall into one of the three categories:

• module repositories maintained by desire through git;
• foreign VCS subroots containing transitory repositories in foreign formats;
• metadata repositories used for desire's internode communication and information tracking.

For the time being, the system-wide policy for maintenance of these storage roots is not devised yet, so they are effectively per-user.

Unless you go the fully-automated bootstrap way, the first time you run desire you will need to prepare an empty writeable directory.

The INIT procedure ensures that repositories constituting your desire node are in working order.

Depending on whether you run a well-known desire node (that is, a wishmaster) you need to provide the :AS keyword to INIT:

(init "/path/to/root/")                            ; for non-well-known mode

or:

(init "/path/to/root/" :as 'your-node-domain-name) ; for wishmaster mode

The specified root directory will contain all VCS-specific master localities, as well as anything module's post-install scripts choose to deliver. This pathname is available in *SELF*, using the ROOT reader.

Unless you already have a '.meta' module, an initial seed version will be downloaded for you. Currently the wishmaster chosen for this is git.feelingofgreen.ru.

This procedure also determines the available VCS tools, as well as conversion tools, and determines the set of accessible remotes.

Further, it scans the git locality for known modules, and makes their systems registered in the ASDF registry.

Unless you happen to have some conversion tools, the set of modules available to your node is restricted to those available via git remotes.

The DESIRE function serves to either initially download or update a set of defined modules.

APROPOS-DESR and LIST-MODULES provide convenient knowledge base query facilities. For a wider set of functions, please see section 3.

From the wishmaster point of view the INIT function also does:

• checks that the locally available set of modules covers every module that is claimed to be "well known" to be published by our distributor⁵, otherwise signalling an error
• publishes the informations about non-released, converted modules in the gate remote's DEFINITIONS file

ADD-MODULE and the accompanying reader macro #@"u://r.l/" is a one-stop point useful for manual extension of the set of known entities. The URI type of the URL must name to the VCS used at the given distribution point, that is one of 'git', 'http' (which actually means git+http), 'darcs', 'cvs' or 'svn'.

The required super-entities are either found among current definitions, or created on the spot.

SAVE-DEFINITIONS writes out changes into <value-of-(ROOT *SELF*)>/git/.meta/DEFINITIONS

This is the term you will often see mentioned, as it is central to the process of distribution of information about the domain.

For the moment it would suffice to say that this is a text file containing forms (which are READable using desire-provided readers) which carry information about terms covered in this section.

The largest unit of software granularity is a distributor, which corresponds to an internet domain name⁷. Currently, they don't carry much information beyond just that.

Distributors contain one or more remotes, which correspond to points of distribution of sets of similarly-accessible modules.

Location is a general term for objects storing module repositories, and are either local, in which case they are localities, or non-local, in which case they are, unsurprisingly, /remotes/⁹.

Modules represent versioned, atomic units of software, as released by the distributor, and, from the point of desire, carry the additional information necessary to complete the information provided by the less granular concepts to obtain the module from its containing remote.

Modules can be provided by several different remotes of different distributors. When the end-user requests retrieval of a module, gate remotes are preferred above others.

Locally, all incoming modules end up in the local gate, which always exists, nevermind the dominant operation mode of the desire node. Once in the local gate, the module becomes /locally available/¹⁰, and is made loadable through the preferred local system definition facility.

Locally available modules can be classified into one of the four categories, the first two of which are only applicable to well-known wishmasters:

• released, for modules advertised through DEFINITIONS to be released through the distributor corresponding to *SELF*;
• converted, for module originating elsewhere, but advertised in DEFINITIONS as being converted in the gate of *SELF*;
• unpublished, modules accessible through the gate, yet unadvertised in DEFINITIONS;
• hidden, modules physically present in the local gate, but made unavailable to anonymous remote clients¹¹.

Descending further down we meet systems. Systems are objects only meant to be relevant in the domain of Common Lisp software, and more precisely – to backend system definition facilities, such as ASDF, XCVB, Mudballs or others¹².

The concept of system introduces inter-system dependencies, which cross module boundaries, producing inter-module dependencies.

Evidently, there can be several systems per module, and also those can be obscured from the end-user, either intentionally or by unfortunate accident¹³.

Desire handles all these complications and operates on the full inter-module dependency graph. It also doesn't store that graph anywhere, recomputing it, instead, every time a request for a module is performed.

It should be noted, that there is no requirement for modules to have systems, which enables end-users to manage (and provide) gittified non-Lisp software for local (and not-so-local) needs.

Applications are simple extensions of systems, providing some very preliminary support for launching programs. They are intended to simplify end-user experience by making requests such as "run climacs" expressible and actionable.

init path &key as (default-wishmasters (list desr:*default-wishmaster*)) => <no values>

Initialise desire with PATH chosen as directory for storage of all VCS-specific locations.

When AS is non-NIL an attempt is made to establish an identity to a defined distributor named by the AS keyword.

This is performed by checking that the locally available set of modules covers every module that is claimed to be to be published by our distributor⁵, according to the local DEFINITIONS. When this check fails an error is signalled.

*self* => distributor

The local distributor set up during INIT, be it well-known or not.

root local-distributor => pathname

The root directory containing all VCS-specific locations of LOCAL-DISTRIBUTOR, chosen during INIT-time.

distributor name &key (if-does-not-exist :error) => distributor

remote name &key (if-does-not-exist :error) => remote

module name &key (if-does-not-exist :error) => module

system name &key (if-does-not-exist :error) => system

app name &key (if-does-not-exist :error) => app

locality name &key (if-does-not-exist :error) => locality

Find objects by name.

name object => symbol

Yield object's name.

url remote-designator &optional module-specifier => string

Compute the URL of a module designated by MODULE-SPECIFIER contained a remote designated by REMOTE-DESIGNATOR.

apropos-desr string-designator &optional set-designator => <no values>

Like APROPOS, but finds objects from the domain of desire.

apropos-desr-list string-designator &optional set-designator => desirables

Like APROPOS-LIST, but finds objects from the domain of desire.

list-modules => <no values>

List all known modules, with some additional information.

module-hidden-p module-designator &optional (locality (gate *self*)) => boolean

See whether whether module designated by MODULE-DESIGNATOR is unavailable to anonymous remote clients.

module-present-p module-designator &optional (locality (gate *self*)) check-when-present-p (check-when-missing-p t) => boolean

Determine whether module designated by MODULE-DESIGNATOR is present in LOCALITY, which defaults to the local gate locality.

local-summary &optional (stream *standard-output*) => <no values>

Print a summary about modules within the local gate to STREAM.

module-best-remote module-designator &key (if-does-not-exist :error) => remote

Produce the remote, if any, which will be chosen to satisfy desires for module designated by MODULE-DESIGNATOR.

module-best-distributor module-designator &key (if-does-not-exist :error) => remote

Produce the distributor, if any, whose remote will be chosen to satisfy desires for module designated by MODULE-DESIGNATOR.

module-fetch-url module &key allow-self => string

Return the URL which is to be used while fetching MODULE, that is the location of MODULE in the preferred remote. When ALLOW-SELF is specified, and non-NIL, remotes within *SELF* are not discarded from consideration.

touch-module module => boolean, string

Try 'access' MODULE via its preferred remote and return whether the attempt was successful as the primary value, and the output of the toucher executable as the secondary value.

system-loadable-p system-designator &optional (locality (gate *self*)) => generalised-boolean

Determine whether system designated by SYSTEM-DESIGNATOR is loadable in LOCALITY, which defaults to the local gate locality.

desire module-name-or-names => boolean

Make modules with MODULE-NAME-OR-NAMES locally available.

add-module url &key module-name systemlessp (system-type desr:*default-system-type*) obtain => module

Define a new module, with download location specified by URL, and the module's name either deduced from the URL, or provided via MODULE-NAME.

When OBTAIN is non-NIL, the module is fetched after its definition is internalised.

update module-designator &optional (locality (gate self)) => <no values>

Update a module specified by MODULE-DESIGNATOR, possibly specifying the target LOCALITY.

make-module-unpublished module-designator &optional (locality (gate *self*))

Stop advertising MODULE in DEFINITIONS, without completely hiding it. If it is hidden, unhide it.

hide-module module-designator &optional (locality (gate *self*))

Stop advertising MODULE in DEFINITIONS, as well as make it inaccessible to general public through LOCALITY.

*fetch-errors-serious* => boolean

Whether to raise an error when external executables fail to fetch modules during DESIRE or UPDATE. Defaults to NIL.

*follow-upstream* => boolean

Whether tracking upstream should update HEAD. Defaults to T.

*dirty-repository-behaviour* => keyword

Whenever a dirty repository comes up in a situation which requires a clean one to proceed, do accordingly to the value of this variable:

• :RESET - reset the dirty repository, losing unsaved changes,
• :STASH - reset the dirty repository, stashing unsaved changes,
• :ERROR - raise an error.

Defaults to :RESET.

system-definition system repository-path &key (if-does-not-exist :error) => pathname

Return the pathname of the SYSTEM's definition.

clear-definitions => <no values>

Forget everything. A subsequent READ-DEFINITIONS will be instrumental to continue any productive use.

remove-remote remote-designator &key keep-modules => nil

Forget everything associated with a remote specified by REMOTE-DESIGNATOR, optionally, when KEEP-MODULES is non-NIL, keeping modules referred by it.

remove-module module-designator &key keep-localities => nil

Forget everything associated with module specified by MODULE-DESIGNATOR, including its systems and applications.

remove-system system-designator => nil

Forget everything associated with the system specified by SYSTEM-DESIGNATOR.

save-definitions &key seal => <no values>

Write out the current idea about the desire's domain into DEFINITIONS, optionally committing changes, when SEAL is non-NIL.

read-definitions &key (source self) (force-source (eq source self)) (metastore (meta-path)) => <no values>

Append definitions currently available in METASTORE to the current idea about desire's domain.