ARM EL2 Stage 2 Translation Configuration and Debugging Guide

ARM Cortex-A Series EL2 Stage 2 Translation Faults and RCU Stalls

The issue at hand involves a fault occurring during the configuration of Stage 2 address translation in the ARM Cortex-A series processor, specifically when attempting to set up a one-to-one mapping from Intermediate Physical Address (IPA) to Physical Address (PA) in the EL2 (Exception Level 2) hypervisor mode. The fault manifests as an RCU (Read-Copy-Update) stall, indicating a potential deadlock or severe latency in the system. The error message provided in the discussion points to a task (optee_example_h) running on CPU 2 that is stuck in a running state, with a call trace leading to __switch_to, suggesting a context switch issue. The core of the problem lies in the misconfiguration of the Stage 2 translation tables, particularly the VTCR_EL2 (Virtualization Translation Control Register at EL2) settings, including the TS0Z, PS, and SL0 fields.

The VTCR_EL2 register is critical for defining the behavior of Stage 2 translation, which is used by the hypervisor to manage guest physical address spaces. Misconfiguration of this register can lead to translation faults, incorrect memory mappings, or system stalls. The TS0Z field determines the size of the translation table walks, the PS field sets the physical address size, and the SL0 field defines the starting level of the page table walk. Incorrect settings in any of these fields can result in the hypervisor being unable to correctly translate IPAs to PAs, leading to the observed RCU stalls and system instability.

Misconfigured VTCR_EL2 Fields and Concatenated Table Alignment

The root cause of the issue is likely a combination of misconfigured VTCR_EL2 fields and improper alignment or layout of the Stage 2 translation tables. Unlike Stage 1 translation tables, Stage 2 tables can use concatenated tables, which require specific alignment and layout to function correctly. Concatenated tables allow for larger memory regions to be mapped by combining multiple translation tables, but this requires careful configuration of the VTCR_EL2 register and adherence to alignment requirements.

The TS0Z field in VTCR_EL2 controls the size of the translation table walks. If this field is set incorrectly, the hypervisor may attempt to access memory regions that are not properly mapped, leading to translation faults. Similarly, the PS field defines the physical address size, and an incorrect setting here can result in the hypervisor attempting to address memory beyond the physical limits of the system. The SL0 field determines the starting level of the page table walk, and if this is set incorrectly, the hypervisor may start the translation process at the wrong level, leading to incorrect mappings or faults.

Additionally, the use of concatenated tables in Stage 2 translation requires that the tables be properly aligned in memory. If the tables are not aligned correctly, the hypervisor may misinterpret the table entries, leading to incorrect translations or faults. The alignment requirements for concatenated tables are specified in the ARM Architecture Reference Manual, and failure to adhere to these requirements can result in the observed system stalls and RCU errors.

Correcting VTCR_EL2 Configuration and Ensuring Proper Table Alignment

To resolve the issue, the VTCR_EL2 register must be configured correctly, and the Stage 2 translation tables must be properly aligned and laid out. The following steps outline the process for correcting the configuration and ensuring proper table alignment:

1. Configure the VTCR_EL2 Register:

   • Set the TS0Z field to the appropriate value based on the desired translation table walk size. This value should be chosen based on the size of the memory regions being mapped and the granularity of the translations required.
   • Set the PS field to match the physical address size of the system. This ensures that the hypervisor does not attempt to address memory beyond the physical limits of the system.
   • Set the SL0 field to the correct starting level for the page table walk. This value should be chosen based on the structure of the translation tables and the desired translation granularity.

2. Align and Layout the Stage 2 Translation Tables:

   • Ensure that the Stage 2 translation tables are properly aligned in memory according to the requirements specified in the ARM Architecture Reference Manual. This includes aligning the base address of the tables and ensuring that the tables are laid out in a way that supports concatenation if needed.
   • Verify that the tables are correctly populated with valid entries that map the desired IPAs to PAs. This includes setting the appropriate attributes for each entry, such as memory type, access permissions, and shareability.

3. Validate the Configuration:

   • After configuring the VTCR_EL2 register and setting up the translation tables, validate the configuration by performing a series of tests to ensure that the hypervisor can correctly translate IPAs to PAs. This includes testing with different memory regions and verifying that the translations are correct.
   • Monitor the system for any signs of RCU stalls or other faults. If any issues are detected, revisit the configuration and alignment of the translation tables to identify and correct any errors.

4. Refer to the ARM Architecture Reference Manual and Example Code:

   • Consult the ARM Architecture Reference Manual for detailed information on the VTCR_EL2 register and the requirements for Stage 2 translation tables. This manual provides comprehensive guidance on the configuration and alignment of translation tables, as well as examples of correct usage.
   • Review example code from projects such as Hafnium, a Type 1 hypervisor for ARM processors, to see how Stage 2 translation is implemented in practice. The Hafnium project provides a reference implementation of Stage 2 translation that can be used as a guide for configuring and aligning translation tables.

By following these steps, the issue of RCU stalls and translation faults during Stage 2 address translation can be resolved. Proper configuration of the VTCR_EL2 register and careful alignment of the translation tables are critical to ensuring that the hypervisor can correctly translate IPAs to PAs and maintain system stability.