ARM Cortex ACE Protocol: Inner and Outer Domain Cache Coherency Behavior
The ARM ACE (AXI Coherency Extensions) protocol is a critical component in ensuring cache coherency across multi-core systems. One of the key aspects of ACE is the division of transaction share domains into Non-shareable, Inner, Outer, and System domains. These domains dictate how cache coherency operations, such as invalidations or snoops, propagate across different cores and caches. A common point of confusion arises when understanding the behavior of Inner and Outer domains, particularly in scenarios where multiple cores share cache lines across these domains.
In the context of ACE, the Inner domain is designed to limit the scope of cache coherency operations to a subset of cores, while the Outer domain extends this scope to a broader set of cores. For example, if Core0, Core1, and Core2 share Cache Line A, with Core0 and Core1 belonging to the Inner domain and Core0 and Core2 belonging to the Outer domain, an invalidation operation initiated by Core0 with ARDOMAIN set to Inner (01) will invalidate Cache Line A in Core1 but not in Core2. This behavior ensures that coherency operations are localized to the Inner domain, preventing unnecessary snoops or invalidations in the Outer domain.
However, this raises questions about the state of Cache Line A in Core0 after the invalidation. If Core0 invalidates Cache Line A in the Inner domain, the state of Cache Line A in Core0 may no longer be Unique, as the invalidation operation does not propagate to Core2 in the Outer domain. This can lead to inconsistencies if Core2 attempts to access Cache Line A without being aware of the invalidation in the Inner domain. Understanding these nuances is critical for designing systems that rely on ACE for cache coherency.
System Architecture Decisions and Domain Division in ARM ACE
The division of Inner and Outer domains in ARM ACE is not arbitrary but is instead a system architecture decision. The CPU defines which address regions belong to the Inner and Outer shareable domains, which directly impacts how coherency operations are broadcast externally. For example, a CPU might designate a specific cluster of cores as part of the Inner domain, while another cluster or external device might be part of the Outer domain. The interconnect plays a crucial role in propagating snoops and ensuring that coherency operations are correctly routed based on these domain definitions.
The ARM AXI specification (D1.6.1) provides examples of how domains can be set up. For instance, a system might have multiple Inner domains within a single Outer domain. This setup is common in multi-cluster systems, where each cluster operates as an Inner domain, but all clusters are part of a larger Outer domain. This hierarchical division allows for efficient coherency management, as coherency operations can be localized within a cluster (Inner domain) without affecting other clusters (Outer domain).
However, this flexibility also introduces complexity. System architects must carefully consider the implications of domain division on performance and coherency. For example, if multiple Inner domains are nested within an Outer domain, the interconnect must ensure that coherency operations are correctly propagated across all relevant domains. Misconfigurations can lead to coherency violations, where cores in different domains access stale or inconsistent data.
Implementing Cache Coherency and Domain Management in ARM ACE Systems
To address the challenges of cache coherency and domain management in ARM ACE systems, developers must implement robust strategies for data synchronization and cache management. One critical step is the use of Data Synchronization Barriers (DSBs) and Data Memory Barriers (DMBs) to ensure that coherency operations are correctly ordered and executed. For example, after invalidating a cache line in the Inner domain, a DSB can be used to ensure that the invalidation operation is completed before any subsequent memory accesses.
Cache management techniques, such as explicit cache invalidations and clean operations, are also essential. In the scenario where Core0 invalidates Cache Line A in the Inner domain, developers must ensure that Core2 in the Outer domain is aware of this invalidation if necessary. This can be achieved through software-managed coherency mechanisms, such as explicit cache flushes or invalidations in the Outer domain.
Additionally, system architects must carefully configure the interconnect to correctly propagate snoops based on domain definitions. This includes setting up address regions and domain mappings in the CPU and ensuring that the interconnect correctly routes coherency operations. Tools such as ARM’s CoreSight and AMBA Designer can be used to validate these configurations and identify potential coherency issues.
In summary, the division of Inner and Outer domains in ARM ACE is a powerful tool for managing cache coherency in multi-core systems. However, it requires careful system architecture decisions and robust implementation strategies to ensure that coherency operations are correctly propagated and executed. By understanding the behavior of these domains and implementing appropriate cache management techniques, developers can build efficient and reliable systems that leverage the full potential of ARM ACE.
