ASIS, a project to manage and to distribute application software in the HEP community

Paper: 157
Session: F (talk)
Speaker: Defert, Philippe, CERN, Geneva
Keywords: desktop computing, development environments, software tools, system management


ASIS, a project to manage and to distribute application software in
the HEP community.
********************************************************************************
Ph. Defert, E. Fernandez D., M. Goossens, O. Le Moigne,
A. Peyrat, I. Reguero

1. Introduction.
------------
ASIS (the Application Software Installation Server) provides an
environment that offers a large number of freely available UNIX
applications which are useful for day-to-day work. It frees end-users
from having to get, configure, generate, install and
manage the software. It also offers tools to system maintainers to
generate, install and manage those packages.

The ASIS repository contains all applications in an executable format.
They cover a large variety of domains: High Energy Physics (HEP) data
analysis software, most GNU packages, the latest TeX/LaTeX setup, MIT
X-windows contributions, TCL/TK based software and many other tools
written by the UNIX community. Presently, the system allows users
access to more than 400 products for eight different platforms on SUN,
DEC, IBM, SGI, HP and PC computers. Soon ten platforms will be supported.
A data base describing all packages with version control information
is maintained in the repository.

The end-user is given access to software on ASIS from any computer of
the site. The only change in the system setup is to schedule the
execution of a single script, each time the repository is updated.

A tools suite helps product maintainers keep their packages
up-to-date. The provided tools automate the generation and
installation process with quality and reliability receiving maximum
attention and helps the version control of the applications.

Other utilities are provided to the remote site managers in order to
replicate the ASIS repository on their own facilities.
2. The Organisation of the Repository
----------------------------------
Within the context of a heterogeneous environment i.e. a combination
of more than one architecture/operating system (O.S.), it is useful to
partition software into two parts:
- a specific part, corresponding to each O.S., containing
binary files and libraries;
- a sharable part, usable on all platforms, containing
scripts, include files, fonts, startup files, man pages, etc.
Each specific part contains references to the shared part.

In the repository, multiple versions of a package can coexist, but
generally, only one is "InProduction", i.e. it is formally supported.

A version of a product which is not "InProduction" is stored in a
specific directory that contains all specific and shared
data such as in the NIST Depot [depot-4].

The "InProduction" products for a platform are all stored in the
same directory tree structure (see Figure fig:insttree). The
correspondence between files and products is kept in the ASIS data
base.

3. Users Access to ASIS
--------------------
The tool ASISinfo and its TK based incarnation ASISbrowse
give users a short description of each package. They also detail
differences between versions and indicate the current state of the
products in the repository.

ASISlwmgr, the ASIS Local Workstation Manager is a perl script which
builds on the local computer the directory(ies) where applications
should reside (in most cases /usr/local) in such a way that all
packages have the correct execution environment. It creates links
from the directory /usr/local to a distributed file system, for
instance at CERN: /afs/cern.ch/asis (AFS), /nfs/cern.ch/asis (CERN
NFS) or /:/asis (DFS).

A local system administrator can customise the behaviour of
ASISwsm to ignore some packages, to get a local copy of others or to
access versions of a product that are only Certified and not
InProduction. A TK based configuration editor is also provided.
ASISwsm optimises the number of links and therefore the space used.4. The Product Maintainer's Environment.
-------------------------------------

4.1 The Software Processing Model
- - - - - - - - - - - - - - -
Observing the various steps that packages go through, from the moment
that the sources are released until when the programs are delivered to
the users. we propose a ``Software Processing Model'' whose
state/transition diagram is shown in Figure fig:software-states.

The model defines the following states:
- Unknown: the package is not present in the data base.
- RemotePackedSources: the system knows from which remote site
it can get the source and where to store it.
- LocalPackedSources: the sources as distributed by the
author(s) are stored in the repository.
- ExpandedSources: ASCII sources are available in the repository.
- ConfiguredSources: sources are ready to be compiled, generally
after Makefile(s) and configuration file(s) were generated.
- Executable: the executable files were built.
- Tested: the tests provided with the sources have been run successfully.
- PreInstalled: the executable files, their execution environment
and the full available documentation have been transferred into
the pre-installation area and gathered within the ASIS repository format.
- Installed: the PreInstalled files are copied into the
repository and are ready to be used by the ``testers''.
- Certified: the software is available to all users after having
been validated by the ``testers''.
- InProduction, : the software is included in the default
environment of the users and officially supported.
- Deleted: the product is removed from the repository.

The states ConfiguredSources to InProduction, are
split in as many sub-states as there are supported platforms.

Possible state transitions are also shown in Figure fig:software-states.

All transitions from and to Installed, Certified, InProduction or
Deleted alter the repository. Any other transition only affects a
particular version of a product leaving unchanged the other products
data and the repository.
4.2 Repository Changes as Transactions
- - - - - - - - - - - - - - - - -
Even though the Software Processing Model describes the actual
operations performed on a package satisfactorily, the modelling of
repository changes needs the introduction of a new concept. When a
product maintainer has to change the version of a product that is
InProduction, two state transitions must take place: (1) a first
version is RemovedFromProduction, (2) a second one is
IntroducedInProduction. Moreover, both operations should be
executed in the right order and in an atomic way. Therefore, a
transaction system was introduced in ASIS. A transaction is a sequence
of operations (transitions and/or specific actions) that are all
performed completely or at all. For a transaction to be committed,
many checks have to be performed about user authorisation, file name
conflicts, file system quotas, available temporary space, etc.


4.3 The Automated Environment
- - - - - - - - - - - - -
A tool called ASIShappi where happi stands for ``Heterogeneous
Automated Product Processor and Installer'' performs all operations in
the ASIS ``Software Processing Model'', happi can execute a single
transition, bring a product from its current state into another or
display the current state of a product and its associated internal
information. It does all necessary operations and dispatches the
generation tasks to dedicated computers, each associated with a
supported platform.

As mentioned earlier there are two types of transitions depending if
they modify or not the repository. Therefore, two user interface are
provided with happi:

- tkhappi is a TK based interface. It increases the usability of
ASIShappi and the readability of its output (see Figure
fig:happi:tk). It covers all transitions that do not interfere with
the repository i.e. from LocalPackedSources to
PreInstalled. Generally, ASIShappi is automatically started as soon as
a new version is detected in the LocalPackedSources directory. tkhappi
is used to view the logs or to repair run-time errors. With
regards to this generation part, the actual work of the ``product
maintainer'' is to provide happi with a ``configuration file'',
written in perl, where all transitions are described. Figure
fig:happi:tk shows part of the one for GNU/emacs. In most cases, this
file is very simple, since many packages are well designed and easy to
build, but also because happi offers a library of functions
corresponding to most frequent operations.- tktes is used by the product maintainer to edit and submit ASIS
transactions to ASISctm (the ASIS Central Transaction
Manager). These transactions change the availability of products and
versions. They are built as a sequence of transitions or special
operations and submitted to the agreement of the ASISctm. When all
checks are positive, the transaction is committed and the product
maintainer's task is over.

The actual modifications of the repository are performed by the
ASISrcm (the ASIS Reference Copy Manager) after the transactions are
committed. The ASISrcm is a specific version of a more general tool,
the ASISlrmgr which is described below.

4.4 Replication.
- - - - - -
The different replicas are updated from a master which is not
necessarily the "Reference Copy". The ASISlcm (the ASIS Local
Copy Manager) reads on the master's ASIS copy the transactions
which were executed since last update and performs them on its own
repository. ASISlcm is scheduled and customised by the system
administrator of the replicating site.




5. The Status of the Project.
-------------------------
ASISwsm is now part of the standard CERN installation
procedure of all UNIX systems

ASIShappi and tkhappi are used for maintaining the majority of
the public domain packages. tktes is not yet in production but will
be by the time of CHEP'97 (tkhappi is used instead).

ASISlcm is being tested with some remote sites and will be put in
production as soon as they are validated.

6. Future Developments.
-------------------
Software interdependencies are at present not handled. We are studying
possible solutions as proposed in Fermi-lab's UPS/UPD [ups], in
Depot-Lite [depot-lite], or in some "Linux distributions" like Debian
or RedHat. They could be used as a basis for ASIS product dependency
management
URL's
*****
http://wwwcn.cern.ch/dci/asis

References
**********

[1] Paul Anderson. Managing program binaries in a heterogeneous
unix network. In LISA V Proceedings, pages 1--9. Usenix, 1991.
[2] William Bliss, Jonathan Streets, Lourdu Udumula, and Margaret
Votava. UNIX Product Distribution, User's Guide. Fermilab, FNAL,
1992.
[3] Michel Dagenais, Stephane Boucher, Robert Gerin-Lajoie, Pierre
Laplante, and Pierre Mailhot. Lude, a distributed software library.
In LISA VII Proceedings, pages 25--32. Usenix, 1993.
[4] Philippe Defert, Alain Peyrat, and Eusebio Fernandez Dominguez.
ASIS: Product Maintainer's Guide. CERN, the European Laboratory for
Particle Physics, 1.00 version, 1995.
[5] Philippe Defert, Alain Peyrat, and Ignacio Reguero. ASIS: Users
and Reference Guide. CERN, the European Laboratory for Particle
Physics, 3.00 version, 1994.
[6] Don Libes. Exploring Expect. O'Reilly and Associates, Inc.,
1995.
[7] Kenneth Manheimer, Barry Warsaw, Stephen Clark, and Walter Rowe.
The depot: A framework for sharing software installation across
organizational and unix platform boundaries. In LISA IV Proceedings.,
pages 37--46. Usenix, 1990.
[8] John K. Ousterhout. Tcl and the Tk Toolkit. Addison Wesley,
1994.
[9] John P. Rouillard and Richsrd B. Martin. Depot-lite: A
mechanism for managing software. In LISA VIII Proceedings, pages
83--91, 1994.
[10] John Sellens. Software maintenance in a campus environment: The
xhier approach. In LISA V Proceedings, pages 21--44. Usenix, .
1991.
[11] Stephen Shafer and Mary Thompson. The sup software upgrade
protocol. Technical report, Carnegie Mellon University, School of
Computer Science, 1988.
[12] Rainer Toebbicke and Philippe Defert. Recommended standard for
unix workstation environment setup. Technical report, CERN, the
European Laboratory for Particle Physics, 1990.
[13] Larry Wall and Randal Schwartz. Programming perl. O'Reilly and
Associates, Inc., 1991.
[14] Walter C Wong. Local disk depot - customizing the software
environment. In LISA VII Proceedings, pages 51--56. Usenix, 1993.