Cortex-A53 Write-Streaming Mode and BUS_ACCESS_LD Anomalies
The Cortex-A53 processor, a widely used ARM core in embedded systems, is known for its efficiency and performance in applications ranging from mobile devices to embedded controllers. However, when operating in write-streaming mode, particularly during memory operations such as memset, unexpected behavior in Performance Monitoring Unit (PMU) counters, specifically the BUS_ACCESS_LD event, can occur. This issue manifests as disproportionately high BUS_ACCESS_LD counts despite minimal actual read activity on the DDR memory controller ports. Understanding this anomaly requires a deep dive into the Cortex-A53’s memory subsystem, cache behavior, and PMU event monitoring mechanisms.
In write-streaming mode, the Cortex-A53 optimizes memory writes by bypassing the cache for streaming data, which is typically non-temporal and not expected to be reused soon. This mode is particularly useful for operations like memset or large memory copies, where caching the data would be inefficient. However, the interaction between write-streaming mode and the PMU’s BUS_ACCESS_LD event counter reveals subtle hardware-software interactions that can lead to misleading performance metrics.
The BUS_ACCESS_LD event is intended to count the number of load transactions on the bus, which typically correspond to read operations. However, in write-streaming mode, the observed BUS_ACCESS_LD counts are significantly higher than the actual read transactions measured at the DDR memory controller. This discrepancy suggests that the PMU may be miscounting or misinterpreting certain bus transactions during write-streaming operations. This issue is further complicated by the Cortex-A53’s errata, which notes inaccuracies in PMU counters for certain events, though BUS_ACCESS_LD is not explicitly mentioned.
Memory Subsystem Behavior and PMU Counter Inaccuracies
The root cause of the high BUS_ACCESS_LD counts in write-streaming mode lies in the intricate behavior of the Cortex-A53’s memory subsystem and the PMU’s event counting logic. When write-streaming is enabled, the processor bypasses the L1 and L2 caches for write operations, directly streaming data to the memory controller. This bypassing mechanism is designed to reduce cache pollution and improve performance for non-temporal data. However, the PMU’s BUS_ACCESS_LD counter may still increment during these operations due to several factors.
One possible cause is the PMU’s interpretation of bus transactions during write-streaming. Although the primary operation is a write, the memory subsystem may generate speculative or auxiliary read transactions to maintain cache coherency or prefetch data. These read transactions, even if they do not result in actual data transfers from memory, could be counted by the BUS_ACCESS_LD event. Additionally, the PMU may not distinguish between different types of bus transactions accurately, leading to overcounting of load events during write-streaming operations.
Another factor is the interaction between the Cortex-A53’s cache management policies and the PMU counters. In write-streaming mode, the L1 and L2 caches are bypassed, but the cache controllers may still perform background operations such as cache line fills or write-backs. These operations can generate bus transactions that the PMU interprets as load events, even though they are not directly related to the write-streaming operation. The errata notice for the Cortex-A53, which mentions inaccuracies in BUS_ACCESS and BUS_ACCESS_ST events, suggests that the PMU’s event counting logic may have inherent limitations or bugs that affect its accuracy in certain scenarios.
The DDR memory controller’s read byte count, which remains low during write-streaming operations, further supports the hypothesis that the high BUS_ACCESS_LD counts are not due to actual read transactions. Instead, they are likely the result of the PMU miscounting or misclassifying bus transactions. This behavior is consistent with the observed data, where the BUS_ACCESS_LD counts are close to the L2D cache access and write-back counts, indicating a potential correlation between cache management activities and PMU event counting.
Mitigating BUS_ACCESS_LD Inaccuracies and Optimizing Write-Streaming
To address the high BUS_ACCESS_LD counts in write-streaming mode, several troubleshooting steps and optimizations can be implemented. These steps focus on understanding the memory subsystem’s behavior, adjusting cache management policies, and mitigating PMU counter inaccuracies.
First, it is essential to verify the Cortex-A53’s configuration and ensure that write-streaming mode is correctly enabled. This involves checking the relevant control registers and ensuring that the memory regions used for memset operations are marked as non-cacheable or write-through. This configuration prevents the caches from being polluted with non-temporal data and ensures that write-streaming operates as intended.
Next, the cache management policies should be reviewed to minimize unnecessary bus transactions that could contribute to BUS_ACCESS_LD counts. This includes adjusting the cache line fill and write-back policies to reduce speculative or auxiliary read transactions. For example, disabling prefetching or reducing the cache line size can help mitigate the generation of extraneous bus transactions. Additionally, ensuring that the cache coherency protocols are correctly configured can prevent unnecessary cache operations that might trigger PMU events.
To address the PMU counter inaccuracies, it is recommended to cross-validate the BUS_ACCESS_LD counts with other performance metrics and hardware counters. This can be done by comparing the PMU data with the DDR memory controller’s read and write byte counts, as well as other cache-related events such as L1D_CACHE_REFILL and L2D_CACHE_WB. If the BUS_ACCESS_LD counts remain disproportionately high despite minimal read activity, it may be necessary to apply workarounds or software patches to compensate for the PMU’s limitations.
One potential workaround is to use alternative PMU events that are less prone to inaccuracies. For example, monitoring BUS_ACCESS_ST (store transactions) or other cache-related events can provide a more accurate picture of the memory subsystem’s behavior during write-streaming operations. Additionally, implementing custom performance monitoring routines in software can help track specific bus transactions and provide more granular insights into the system’s performance.
Finally, it is crucial to stay informed about any updates or errata related to the Cortex-A53’s PMU and memory subsystem. ARM periodically releases errata notices and software updates that address known issues and improve the accuracy of performance monitoring. Applying these updates and following ARM’s recommendations can help mitigate the BUS_ACCESS_LD inaccuracies and ensure optimal performance in write-streaming mode.
In conclusion, the high BUS_ACCESS_LD counts observed in Cortex-A53 processors during write-streaming operations are likely due to a combination of PMU counter inaccuracies and the memory subsystem’s behavior. By understanding the underlying causes and implementing the appropriate troubleshooting steps and optimizations, developers can mitigate these issues and achieve more accurate performance monitoring in their embedded systems.
