Linking PCI based processor platforms
Paper: 439
Session: B (poster)
Presenter: Weymann, Martin, CES, Petit-Lancy
Keywords: ATM, control systems, data acquisition systems, event building, off-the-shelf products
Linking PCI based processor platforms
Martin Weymann
Creative Electronic Systems
70, route du Pont-Butin
CH-1213 Petit-Lancy
Tel: +41 22 792 57 45 / Fax: +41 22 792 57 48
Email: martin@lancy.ces.ch
Abstract
The data acquisition system of a HEP experiment typically contains a
large number of processor nodes working in parallel in order to
collect, format and analyse event data. Processors may be embedded,
such as VME processor boards, or they may be desktop workstations. For
high end processors in both domains PCI bus is becoming the predominant
choice for the local bus. PCI bus, while providing a fairly high
bandwith locally (128 MByte/s at 32-bit,32 MHz), is confined to the
spatial dimensions of a processor board or PC-motherboard. The
following paper will describe the PVIC concept developed by CES. The
PVIC allows PCI based processor boards, such as the CES PowerPC VME
processors RTPC 8067 or RIO2 806x, to be connected to each other and
to PCI based desktop workstations on three different distance scales:
below 1 meter (e.g. within a VME crate), up to 20 meters (via a
differential copper connection), and up to 200 meters (via fiber).
A single system can contain up to 15 nodes connected by a mix of all
three types of connections. The PVIC allows both blocktransfers at the
full PCI bandwith and directly mapped single cycle access remote PCI
addresses. In addition to minimizing the latency, direct mapping
drastically reduces the software overhead - after an initialization
phase the remote PCI registers are directly mapped into the local PCI
address space and can be transparently accessed by the local processor.
Mechanisms for inter-processor synchronisation are also provided: A
reflective memory (RFM) allows a coherent data set to be maintained
amongst all nodes in the system. In addition asynchronous notification
of a processor is possible if data in the RFM have been updated by
another node. Mailbox signalling and interrupt dispatching is provided
as well. All nodes contain an integrated DMA controller that can
dispatch data throughout the system without loading the host CPU. The
DMA hardware is able to process chains, linked list of indivdyual DMA
transfers, and may be triggerd on external events (e.g.PCI interrupts).
Broadcasts allow data or control information to be updated
simultaneaously in the PCI memory of several nodes. The destination
addresses of these broadcasts can be locally re-mapped with a
granularity of 4 KByte pages, allowing also the selective acceptance of
broadcast operations (multicast). DMA and multicast features render
the PVIC particularily suitable to event-building applications, while
in control and monitoring applications RFM and direct mapping may be of
greater importance. The PVIC complements network oriented
interconnections such as ATM and FDDI which may cover a larger numer
of nodes and larger distances at the cost of increased software
complexity and larger latencies. The PVIC concept will be implemented
as a family of boards: a PMC 'PCI Mezzanine card, a short format
PC-type PCI card and a group of 6U VME cards implementing bridges
between the different PVIC tranmission media and between PVIC and VME.
First hardware is expected around the time of this conference.