Approved ARM Compiler test suites include:
- The Plum Hall Validation Suite for C™
- The Plum Hall Validation Suite for C++ (Suite++®)
- The Plum Hall Validation Suite for the C++ Library
- In-House Validation Suites
- Edison Design Group (EDG) Front-End Test Suite
- Roguewave C++ Library Self Tests
- GNU Torture
- Standard Benchmark Tests
- ABI Test Suite
- Debug Tests
- Defect Management
- ISO Certification
- Keil Product Integration
The Plum Hall Validation Suite for C is the industry standard for the verification of C compilers. The Plum Hall Validation Suite for C uses both positive and negative test techniques to accomplish the following:
- Test basic conformance to the ISO c90 and c99 C language standards.
- Generate self-checking C programs that test permutations of operators and data types.
- Determine the size of certain compile time limits, e.g., significant length of identifiers or how deeply include files may be nested.
- Test for legal expressions by generating self-checking expressions of arbitrary complexity.
Suite++ is the industry leading conformance test for the ANSI/ISO C++ standard. Suite++ represents over 4,600 positive and negative test cases for compilers against ISO/IEC 14882:2003 conformance.
LibSuite++ represents over 2,500 test cases that specifically test the libraries for ISO/IEC 14882:2003 conformance.
- Diff Tests - For detecting differences in the code quality, ARM compares the code generated from new compiler versions with that of previously tested versions. This test suite generates an internal report showing differences against the previous version. ARM analyzes this report to verify if any code differences are unexpected.
- Regression Suite - When defects are fixed, tests are added to our regression tests suite to ensure the defect is not re-enabled by future compiler changes.
- Language Extensions - ARM maintains an in-house test suite for testing product language extensions which are not covered in the ANSI/ISO standard.
- C-Header Tests - Verifies the internal consistency of the standard C header files. They work by running the compiler over each header and having it dump out its internal state. The result is then examined to ensure consistency.
- Nullspace Tests - Measures consistency of expected compiler behavior and optimizations across semantically equivalent, but syntactically different source code.
- Math Library Tests - The Floating-Point and Math libraries experience directed tests of special cases (NaNs, infinites, underflow, etc.) as well as tests associated with known difficult values.
- Convergence Tests - This suite tests that the ARM Compiler can build itself. This is done in two stages, first using the ARM Compiler to build a semi-hosted version of itself targeting ARM, then using the semi-hosted version to build the second version, and comparing the final images and objects.
- Code Database (CodeDB) - CodeDB is a collection of open source applications and approved customer code used to verify that complex real-world applications can be built with the latest compiler version.
- Product Integration Tests - These tests ensure that the different product components have been successfully integrated and produce the preferred user experience. New tests are generated for each release based on new features in that release. In addition to new tests, product integration testing consists of more than 1,200 regression test cases.
EDG Front End software is tested using Gimpel's FlexeLint, Plum Hall, and Perennial commercial test suites. Additionally, the front-end software is subjected to ~25,000 local regression tests developed over the past 20 years.
A self-test suite supplied with the RogueWave C++ Standard Template Library (STL).
These tests form part of the GNU C compiler distribution. They include self-checking tests and also tests for the compiler front end.
Standard benchmarks such as EEMBC and Dhrystone are used to verify that the benchmark suites compile and execute properly.
This suite tests conformance to the ARM-THUMB Embedded Application Binary Interface (EABI) Procedure Call Standard (ATEPCS).
This tool checks that the ARM Compiler is generating valid DWARF debugging information, it analyzes source file information via embedded annotations using a purpose-designed language, DVAL (DWARF Validator Annotation Language).
Defects are numbered, documented, and prioritized in our defect database. Defects are regularly reviewed and targeted for correction based on priority. Defect data is secured via nightly backups and offsite storage.
ARM maintains ISO9001:2008 certification (number LRQ 0963114) which is applicable to design, development, and quality management of software solutions at ARM.
The ARM Compiler has been used in all versions of the Keil MDK-ARM (Microcontroller Development Kit) since its release in 2006. A full MDK-ARM version history is available for reference.