ChimeraTK-DeviceAccess 03.20.00
•All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
Design: AsyncNDRegisterAccessor

Concept overview

Warning
This document is supposed to help with understanding the code structure. It also contains a lot of technical details that led to certain design decisions. As the structure is still in the flow, it might contain outdated information.

AsyncNDRegisterAccessors and the AsyncAccessorManager

  • The AsyncAccessorManager is a base clase which manages the subscriptions of accessors requested by the users.
  • For each AsyncAccessor that the user requests, an AsyncVariable is created interally, with 1:1 correspondence AsyncAccessor ↔ AsyncVar.
  • The AsyncAccessor is handed out to user. The AsyncVariable holds the a buffer which is used to fill data into the AsyncAccessor when sending.
  • The AsyncVariable knows how to fill its send buffer, i.e. it might be special for each AsyncAccessorManager implementation (see TriggeredPollDistributor and VarialeDistributor)
  • AsyncVar is only holding a weak pointer to the AsyncAccessor, since we want to be sensitive to the user discarding the latter

The TriggeredPollDistributor

  • The TriggeredPollDistributor is polling data via synchronous accessors when a trigger (interrupt) comes in.
  • The PolledAsyncVar implements filling the send buffer from an associated syncAccessor.
  • The TriggeredPollDistributor is using a transferGroup for efficiency. As the syncronous accessors and the transfer group already implement all the necessary conversion to user types, partial use or 1 D registers etc., we simply use one matching synchronous accessor for each PolledAsyncVar.

The SubDomain

The reverse ownership scheme in the NumericAddressedBackend

  • The triggers (resulting from interrupts) are distributed from left to right, mostly through weak pointers (dashed lines and empty arrow head)
  • For each interrupt handler thread there is one async::Domain which implements a central lock for the whole tree behind it. It protects the trigger distribution against concurrent subscriptions of accessors and the sending of exceptions. The Domain is of the backend data type "nullptr_t" because the triggers are conceptually void. The NumericAddressedBackend is holding weak pointers to the Domains. The Domains are created when they are needed, i.e. when an accessor down the tree is created. Other backends might hold shared pointers to Domains, depending on the implementation.
  • Behind each Domain there is exactly one (primary) SubDomain. The SubDomain basically is an adapter to the connected distrubutors (VariableDistributors, TriggeredPollDistributors and MuxedInterruptDistributors). Further SubDomains are behind the MuxedInterruptDistributors, one for each sub-interrupt.
  • SubDomains are identified by a qualified (i.e. heirarchical) ID (represented as a std::vector<size_t>). The primary SubDomain ID has a length 1, and for each hierarchy level a number is added. The Domain ID is a number, machting the first entry in the qualifiedAsyncSubDomainID. (qualifiedAsyncSubDomainID is abbeviated as qualifiedSubDomainID or qualifiedAsyncID, depending on the context). Example: The SubDomain [0,4] is responsible for sub-interrupt 4 in the MuxedInterruptDistributor [0] behind the primay SubDomain [0] in Domain 0 (see Canonical interrupt names).
  • The single instance of the DomainsContainter is responsible for distributing exceptions to all Domains. It is always holding weak pointers to the Domains.
  • VariableDistributors and TriggeredPollDistributors are derrived from the AsyncAccessorManager and are holding accessors to the backend to read data or do handshakes (red), which in turn are holding shared pointers to the backend (shared ownership). The backend must never (also not indirectly) hold a shared pointer to any of these entities because it would introduce a closed ownership circle and the backend could never be deleted properly.
  • The ownership is depicted via solid line arrows, mostly from right to left. In the end the AsyncNDRegisterAccessors handed out to the user (green) are holding the whole trigger distribution tree. (except for the DomainsContainer in blue)
  • When an accessor is requested, the request goes from the DomainsContainer through the Domain to the primary SubDomain, which creates all required MuxedInterruptDistributors, SubDomains, VariableDistributors or TriggeredPollDistributors when needed, and hands out the AsyncNDRegisterAccessor which holds the ownership. It shares the ownership with other AsyncNDRegisterAccessors.
  • When the last owning AsyncNDRegisterAccessors goes out of scope, the according distributors are deleted.
  • If there are no subscribers, i.e. AsyncNDRegisterAccessors being held by the application, the Domain shared pointers are empty and trigger distribution does nothing.

Vision: Using the VariableDistributor DOOCS backend

‍ATTENTION: This section is the envisioned use in the DOOCS backend. It is not implemented yet and the framework and this vision probably need a few more iterations to make it work.

  • The EqData object, which is send as an asynchonous event, is distributed in the case of DOOCS. It contains all the relevant data.
  • For each ZMQ-subscription there is one Domain with its SubDomain and one VariableDistributor<EqData>. The variable distributor takes care of interpreting the EqData and extracting the information to be filled into the subscribed AsyncNDRegisterAccessors.

The AsyncNDRegisterAccessor and the AsyncAccessorManager

The AsyncNDRegisterAccessor is a generic NDRegisterAccessor implementation for push-type data. It is used if accessors with AccessMode::wait_for_new_data are requested. See Technical specification: TransferElement B.8 for the requirements it is based on.

The AsyncNDRegisterAccessor is a generic implementation which can be used for all backends and contains the following components

  • a lock free queue to transport data of type NDRegisterAccessor<UserType>::Buffer, which contains the data value (2D vector of UserType), a VersionNumber and a DataValidity flag.
  • functions to push data and exceptions into the queue

AsyncNDRegisterAccessor are read-only. If you want to write to the register you have to get an Accessor without "wait_for_new_data". (N.B. in NumericAddressedBackends this is not possible, the push-type data is intrinsically read only there. For Doocs-Backends with ZMQ data it is possible to get an Accessor for writing.)

For each primary interrupt or event there is one async::Domain. All the distribution tree behind it belongs to that Domain.

  • For backends with void interrupts, there is always one primary InterruptDistributor associated to the Domain, followed by a TriggeredPollDistributor, a void-type VariableDistributor and/or an InterruptControllHandler. The latter has one InterruptDistributor per sub interrupt, which again can distribute to a TriggeredPollDistributor, a VariableDistributor and/or an InterruptControllHandler etc.
  • For push-type data (data events), there is a one distributor of the backend-specific source type, which then distributes further to the subscribed AsyncRegisterAccessors. – Example 1 (envisioned use case, not implemented yet): For Doocs, a VariableDistributor<EqData> is getting an EqData object which contains a double, and the has subsribers of user type doule, int and string. The VariableDistributor<EqData> in combination with the according AsyncVariables takes care of converting the data into the user types and filling the user-type send buffers. – Example 2 (envisioned use case, not implemented yet): EqData with an IFFF is distributed via an IFFFDistributor (also derived from AsyncAccessorManager like VariableDistributor and TriggeredPollDistributor.). You can have individual subscriptions for each component (one integer and three floats).
  • The VariableDistributor has a copy of the last distributed data. This is used as initial value for a new subscription.

The AsyncAccessorManager has three main functions:

  • It serves as a factory for AsyncNDRegisterAccessors
  • It contains a list of created asynchronous accessors and a subscribe/unsubscribe mechanism
  • It provides functions to act on all asynchronous accessors

Design decisions and implementation details

  • The data transport queue contains data that is already converted to the UserType, so the potentially expensive and slow operations (data readout, de-multiplexing, data conversion) are happening only once, before the data is distributed to multiple subscribers.
  • To unsubscribe, the NDRegisterAccessor is calling AsyncAccessorManager::unsubscribe() with its TransferElementID from the destructor.
  • The class AsyncVariable represents the sender side of the asynchronous accessor.(*) It provides functions to send data and exceptions, and it implements all the handling of the contained AsyncNDRegisterAccessor instance. In addition it provides shape information for creating AsyncNDRegisterAccessors to the AsyncAccessorManager.
  • A send buffer is contained in the AsyncVariable to avoid dynamic memory allocation.
  • The void notification queue in the TransferElement base class is set up as a continuation of the data transport queue, such that the read operations of the notification queue trigger the filling of the user buffer.

(*) C++ implementation detail: There is AsyncVariable as an untyped base class and AsyncVariableImpl, which is templated to a user type. Like this pointers to variables with different user types can be stored in one list.

Exception handling

  • Some classes like the TriggeredPollDistributor and the MuxedInterruptDistributors need to access the hardware.
  • As this code is called from within a thread, no exceptions must escape. However, in case of an exception the backend's setException() must be called to inform the user application.
  • TriggeredPollDistributor and the MuxedInterruptDistributors do HW access via regular, synchrous accessors, which call the backend's setException() when an exception occurs. They just suppress the exception coming out of the accessor.
  • In setException(), the exceptions must be distributed to all async accessors. This is done through the same three as the trigger distribution.
  • As creation of accessors and distribution must not happen at the same time and are in different threads, inevitably there are locks involved. In addition, the backend might use locks internally.
  • As setException() can occur during the regular distribution, this would be a recursive iteration of the tree if it happens in the same threads, which can lead to deadlocks or lock order inversions.
  • To avoid lock order inversion, recursive tree iteration is prevented. There is only one lock in the Domain, which protects the whole tree behind it.
  • To implement the exception distribution, there is a thread in the DomainsContainer. setException() in informing that thread that exceptions should be distributed and then returns, such that the distribution call which executed setException() can return. The lock in the Domain is freed, and the exception distribution thread can get it.

Interface for implementing backends

Requirement for each backend:

  • Provide (one or more) implementations of a VariableDistributor<BackendSpecificDataType>.
  • Provide a template-function 
      template<typename BackendSpecificUserType>
  std::pair<BackendSpecificUserType, VersionNumber> getAsyncDomainInitialValue(size_t asyncDomainId);
  • Implement the virtual function (Fixme: This is NumericAddressed only at the moment. Should be generalised) 
      std::future<void> activateSubscription(uint32_t interruptNumber, boost::shared_ptr<async::DomainImpl<BackendSpecificDataType>> asyncDomain);
    Backends deriving from NumericAddressedBackend create the interrupt handling thread here, and store the asyncDomain to call it from inside the thread.
  • Call DomainsContainer::subscribe() in getRegisterAccessor() to create and hand out AsyncNDRegisterAccesors.
  • Fixme: should move to DeviceBackendImpl: Activate all AsyncDomains in activateAsyncRead, and call activateSubscription() for it.

(*2) The reason to put this into the AsyncVariable is coming from the NumericAddressedBackend. It adds a synchronous accessor to the AsyncVariable which is used for reading and provides this information. The logic to generate this information is in the creation code for the synchronous accessor. Other backends might have similar advantages from having the functions the AsyncVariable, or at least they don't care much where the functions are implemented.

Implementation in the NumericAddressedBackend

Asynchronous registers in the map file

# name            nElements  address  nBytes  bar  width  bitInterpretation signed accessMode
APP.0.DATA_READY          0        0       0    0      0                  0      0 INTERRUPT1
APP.0.PUSH_DATA           4       12      16    1     32                  8      1 INTERRUPT2

The map file has a special access mode INTERRUPTx, in addition to the synchronous modes RW and RO. x is the numeric identifyer for a device interrupt. Registers with access mode INTERRUPT are treated as read-only. Sub-interrupts (INTERRUPTx:y, INTERRUPTx:y:z, etc.) are supported in combination with MuxedInterruptDistributors (see below).

For interrupts it is possible to have void-type entries. The width of these "registers" is 0 bits. For consistency all other fields also have to be 0. They don't make sense in this context and have to be 0 to avoid confusion.

Canonical interrupt names

For each interrupt and sub-interrupt it is possible to get a void accessor via the canonical interrupt name '!x:y:z:'. For instance for an interrupt with data on INTERRUPT3:9:4, it is possible to get void accessors for '!3', '!3:9' and '!3:9:4'. This is useful to check whether a primary interrupt has arrived, or any interrupt on the controller 3:9.