A major factor which determines the Success/Failure of current products/systems is Time to Market. As the time to market is becoming more critical than ever day after day , the early software development is becoming one of the most important component in today’s SOC / product development. Effectively Developing, Integrating and Validating firmware/software within the stipulated timeline is becoming so important that software teams now cannot wait till availability of silicon on development boards. Thus it is now absolute necessity to make available, accurate, fast and low cost Virtual Hardware Platforms, to software teams very early in the product development cycle
SystemC TLM2.0 based models are currently the best available/ widely used technology to create Virtual Platforms in today’s world. These models additionally are useful in architectural Exploration and also can act as golden standard Reference model for RTL / Netlist verification.
As the industry has now widely starting using the SystemC models in various stages of the SOC/ASIC product development cycle, their verification has also come out to an additional challenge. At CircuitSutra, we have long been involved in the development and verification of SystemC / TLM models . In this article we will discuss about various techniques that are most effective for SystemC / TLM model verification and their respective use cases.
The 3 most effective approaches for Model verification are -
VERIFICATION USING DIRECTED TESTS
Directed testing is a straightforward approach to functional verification. For writing directed tests we pick a feature, write a test to verify that feature and repeat the process until all the features are tested. However with increasing design size and complexity the number of directed tests required for verification are increasing exponentially.
Two kind of verification setup are feasible to enable the directed testing.
One simple setup is to develop the testbench for a specific IP model. This testbench should allow to read / write the value of registers of the IP, read / write the value of pins of the IP. CircuitSutra modeling methodology provides a python based unit testing framework.
The other setup is to get the virtual prototype of the complete SoC by connecting the IP models with the processor models as per the address map of SoC. The testcase is written in the form of bare metal embedded code, that is cross compiled and executed on the virtual prototype. This bare metal code, configures the IP for a certain functionality by writing to the registers of the IP, and checks the correctness of specific functionality by checking the value of related register & memory addresses etc..
At Circuitsutra we follow a hybrid approach for doing functional verification with directed tests. We have divided the verification process into three phases:-
i) Establish a baseline for directed testing:-
The goal of the first phase is to establish a solid baseline to support subsequent testing/verification. In this phase all possible testable features of IP Models are identified using the specification document and a list of generic test cases are developed for sanity testing of IP Models in order to ensure stability of test framework and that of IP Models code base. Directed tests in this phase don’t require implementation of a coverage model or a completed design or verification environment, so results in this phase are quick and easy to quantify.
ii) Developing direct test case:
This phase is a transition phase that builds on top of the directed tests and test/verification environment developed in phase i. In this phase test developed in phase 1 are modified and tested with random test data of various kinds in order build a strong productive environment for verification of IP models.
To support random inputs:-
At Circuitsutra we use an open source tool DGL in order to generate test data of various kinds. DGL is a data generation language developed by University of Florida that can be used to generate test data systematically, randomly, or a combination of both. The goal of DGL usage is to reduce the complexity of test cases, number of test cases by capturing multiple combinations under one test and capture more functional coverage. In order to use and generate test inputs using DGL we write test cases to test a set of input configuration and then generate multiple different combinations of that input configurations using DGL productions.
For analog and communication IP:-
To test analog IP such as ADC , A continuous analog input signal is required and for this at Circuitsutra we use generic systemC IP(port interface driver) to drive input port of analog module IP by using python VCD driver. To drive input port at particular time for defined duration VCD driver uses .vcd file ( contain port name, input value & time duration etc), VCD driver written in python read vcd file parameters and call wrapper API’s of port interface driver to drive input ports of analog devices .
SystemC TLM2.0 based models are currently the best available/ widely used technology to create Virtual Platforms in today’s world. These models additionally are useful in architectural Exploration and also can act as golden standard Reference model for RTL / Netlist verification.
As the industry has now widely starting using the SystemC models in various stages of the SOC/ASIC product development cycle, their verification has also come out to an additional challenge. At CircuitSutra, we have long been involved in the development and verification of SystemC / TLM models . In this article we will discuss about various techniques that are most effective for SystemC / TLM model verification and their respective use cases.
The 3 most effective approaches for Model verification are -
- Directed unit testing approach
- Using the already available RTL testbench to verify IP models.
- Using System Universal Verification Methodology(UVM) to verify complex SoC / IP Models.
VERIFICATION USING DIRECTED TESTS
Directed testing is a straightforward approach to functional verification. For writing directed tests we pick a feature, write a test to verify that feature and repeat the process until all the features are tested. However with increasing design size and complexity the number of directed tests required for verification are increasing exponentially.
Two kind of verification setup are feasible to enable the directed testing.
One simple setup is to develop the testbench for a specific IP model. This testbench should allow to read / write the value of registers of the IP, read / write the value of pins of the IP. CircuitSutra modeling methodology provides a python based unit testing framework.
The other setup is to get the virtual prototype of the complete SoC by connecting the IP models with the processor models as per the address map of SoC. The testcase is written in the form of bare metal embedded code, that is cross compiled and executed on the virtual prototype. This bare metal code, configures the IP for a certain functionality by writing to the registers of the IP, and checks the correctness of specific functionality by checking the value of related register & memory addresses etc..
At Circuitsutra we follow a hybrid approach for doing functional verification with directed tests. We have divided the verification process into three phases:-
i) Establish a baseline for directed testing:-
The goal of the first phase is to establish a solid baseline to support subsequent testing/verification. In this phase all possible testable features of IP Models are identified using the specification document and a list of generic test cases are developed for sanity testing of IP Models in order to ensure stability of test framework and that of IP Models code base. Directed tests in this phase don’t require implementation of a coverage model or a completed design or verification environment, so results in this phase are quick and easy to quantify.
ii) Developing direct test case:
This phase is a transition phase that builds on top of the directed tests and test/verification environment developed in phase i. In this phase test developed in phase 1 are modified and tested with random test data of various kinds in order build a strong productive environment for verification of IP models.
To support random inputs:-
At Circuitsutra we use an open source tool DGL in order to generate test data of various kinds. DGL is a data generation language developed by University of Florida that can be used to generate test data systematically, randomly, or a combination of both. The goal of DGL usage is to reduce the complexity of test cases, number of test cases by capturing multiple combinations under one test and capture more functional coverage. In order to use and generate test inputs using DGL we write test cases to test a set of input configuration and then generate multiple different combinations of that input configurations using DGL productions.
For analog and communication IP:-
To test analog IP such as ADC , A continuous analog input signal is required and for this at Circuitsutra we use generic systemC IP(port interface driver) to drive input port of analog module IP by using python VCD driver. To drive input port at particular time for defined duration VCD driver uses .vcd file ( contain port name, input value & time duration etc), VCD driver written in python read vcd file parameters and call wrapper API’s of port interface driver to drive input ports of analog devices .
To test communication module such as I2C, SPI, CAN etc , we have generic SystemC test bench which has two interfaces such as GPIO interface and Module interface. Test bench GPIO ports have connection with system GPIO ports to transfer module specific configuration data and store all received data in a structure. Once test bench receive data by module interface it start comparing received data with stored configuration and shows comparison result.
iii) Exhaustive Testing with random inputs and coverage measurement:-
Phase 3 starts when all basic and critical features of the IP Model have been verified rigorously and as a result, the design can reasonably be expected to handle the full scope of legal stimulus.
The goal in phase 3 is to exercise difficult corner conditions as efficiently as possible. Tests written in this phase are lightly constrained and focus more towards the code coverage of IP models. Additionally in this stage, a second class of directed tests is written as needed for focusing on error conditions and/or hard-to-reach corner scenarios of the IP model functionality.
Following figure represent the functional coverage progress during each phase.
Phase 3 starts when all basic and critical features of the IP Model have been verified rigorously and as a result, the design can reasonably be expected to handle the full scope of legal stimulus.
The goal in phase 3 is to exercise difficult corner conditions as efficiently as possible. Tests written in this phase are lightly constrained and focus more towards the code coverage of IP models. Additionally in this stage, a second class of directed tests is written as needed for focusing on error conditions and/or hard-to-reach corner scenarios of the IP model functionality.
Following figure represent the functional coverage progress during each phase.
Advantages and disadvantages of using this approach has been summarised below -
Advantages-
VERIFICATION USING RTL TESTBENCH / VERIFICATION ENVIRONMENT
Lots of advancement have happened in the the RTL verification methodologies over past decades. A very comprehensive verification of the RTL is done before the design goes for the tape out. The same verification setup, if used for the verification of SystemC models will ensure that SystemC models are of the equally high quality as the hardware.
This methodology is useful when the SystemC models are developed for the IP whose RTL is already available. For any new SoC design it is becoming mandatory to develop the virtual prototype, most of these new SoC designs reuses the existing well tested IPs. The SystemC models of these existing IPs may be verified using the existing RTL verification setup of these IPs.
For the new IPs also it makes sense to verify the SystemC models using the RTL verification environment, once the RTL design & verification is complete. In the initial phases the virtual prototype can be verified using directed tests, later when the RTL verification environment becomes available, the virtual prototype can be further verified using the RTL verification setup. Primary motivation for virtual prototype development is to enable pre-silicon embedded software development, however going forward more and more companies are using virtual prototypes for post silicon use cases as well. Semiconductor companies are required to provide the virtual prototype of their SoC to their customers, who utilize these for their system level verification and firmware development. For such post silicon use cases of the virtual prototype, expectation is that quality of the virtual prototype should be as good as real hardware.
In our use case, a RTL Verification environment can be shown as in the diagram below. All the RTL IPs in DUT were connected together and stimulus was applied at the DUT Boundary and signals were also observed at the DUT boundary .
Advantages-
- It is easier and quicker way for IP verification.
- Helps to find the redundant lines of code in IP model.
- With little modification tests written via this approach can be reused for successive versions of IP models.
- Possible to simulate hardware behaviour for analog devices.
- Able to verify communication IP with virtual slave device.
- This approach require availability of an integrated platform and toolset for compiling C/C++ based test cases, loading it, running and debugging it on the virtual platform.
- Manual writing of each test case takes more time as compared to automated constrained random testing.
- Additional tool support is required for external stimulus, debugging etc.
VERIFICATION USING RTL TESTBENCH / VERIFICATION ENVIRONMENT
Lots of advancement have happened in the the RTL verification methodologies over past decades. A very comprehensive verification of the RTL is done before the design goes for the tape out. The same verification setup, if used for the verification of SystemC models will ensure that SystemC models are of the equally high quality as the hardware.
This methodology is useful when the SystemC models are developed for the IP whose RTL is already available. For any new SoC design it is becoming mandatory to develop the virtual prototype, most of these new SoC designs reuses the existing well tested IPs. The SystemC models of these existing IPs may be verified using the existing RTL verification setup of these IPs.
For the new IPs also it makes sense to verify the SystemC models using the RTL verification environment, once the RTL design & verification is complete. In the initial phases the virtual prototype can be verified using directed tests, later when the RTL verification environment becomes available, the virtual prototype can be further verified using the RTL verification setup. Primary motivation for virtual prototype development is to enable pre-silicon embedded software development, however going forward more and more companies are using virtual prototypes for post silicon use cases as well. Semiconductor companies are required to provide the virtual prototype of their SoC to their customers, who utilize these for their system level verification and firmware development. For such post silicon use cases of the virtual prototype, expectation is that quality of the virtual prototype should be as good as real hardware.
In our use case, a RTL Verification environment can be shown as in the diagram below. All the RTL IPs in DUT were connected together and stimulus was applied at the DUT Boundary and signals were also observed at the DUT boundary .
To use the RTL testbench to verify the SystemC Model, the RTL IP was replaced by the Model IP in the DUT.
In the diagram above it is shown that the TB was directly connected to the ports/pins at model boundary and all the RTL IPs were bypassed. The connection was done by mapping the HDL paths of the Model IP with the corresponding signals in Testbench.
It requires to develop the specialized transactors, drivers / monitors etc.. to connect the pin level cycle accurate world to the transaction level, loosely timed models. This kind of setup requires co-simulation of SystemC / C models along with RTL. Similar setup can also be created using hybrid co-emulation methodologies.
RTL verification technology has evolved to be very advanced and state of art. A typical Hardware verification language such as System Verilog, Specman E etc already have features builtin to achieve the most effective verification. Some of these features are -
Pros and Cons of this approach has been summarised below -
Advantages-
Disadvantages -
VERIFICATION USING SYSTEMC UVM
As electronic systems are getting complex day by day so are their virtual prototypes or simulation models. SystemC helps creating models in short time as it has constructs required to model hardware easily along with lot more other features and uses abstract communication (TLM). But as models are getting complex verifying them also is taking considerable amount of time & effort. Till now as there is no standard methodology for system level verification, companies are following their own flows and testbench creation methods, where testbenches and stimulus needs to be created for each module and also again at system level.
UVM (Universal Verification Methodology) is a standard verification methodology which specifies how to construct a test environment which is modular, scalable, configurable and reusable. UVM (or its predecessors like OVM) implementation in SystemVerilog is kind of defacto for block level/RTL verification.
Even though SystemC is the standard system level design language till now it was lacking support of a unified verification methodology. UVM-SystemC standard promises to decrease the system level verification effort by implementing UVM methodology in SystemC. With this standard there is lot of possibility for reusing test infra from module level to system level. And also it helps in achieving better test coverage. Testbench components may be reused from system level to block level also. Tests and testbenches can also be reused across verification and validation.
As SystemC-AMS helps in modeling heterogeneous systems there is also some effort going on in extending UVM for SystemC-AMS also. This will help in verifying full system model (digital + mixed-signal) with a single testbench.
Summary
As Discussed above all the 3 techniques used have their unique use case.
Directed tests are very useful to verify IPs/System when models have been integrated into Virtual Platform. It allows verification engineer to verify models keeping the perspective of the firmware engineer. As the tests are run on the on-chip processor, the tests are very near to real life use cases. But verifying individual IPs is tedious with this approach.
On the other hand, using RTL Testbench is most effective in verification of IPs standalone. It can cover vast stimulus using constrained random and generating unthinkable scenarios to catch tough bugs. It is very cost effective as Test Bench is already available and verification can be quickly kickstarted. But It is very complex to verify whole system or integrated Virtual platform.
UVM SystemC is alternatively very efficient with both IP and System level verification. It is very structured methodology, created using most effective time tested methods for verification. It makes code reuse very easy. As it is standard, Maintenance is very easy. But It is very complex to build a verification environment, takes much extra development time when done from scratch. But once developed, modification is easy.
As we can see, All the said techniques are very effective in their respective use case. No technique is best and has its pros and cons. The best use of any technique can only be made by determining various factors like schedule , complexity of design, availability of resources, cost , Verification goals et cetera. At CircuitSutra we have been pioneering Model verification since many years and with dozen of projects successfully completed. We can help companies determine and deploy the most effective or all technique for complete verification of their SystemC Model/Virtual Platform.
It requires to develop the specialized transactors, drivers / monitors etc.. to connect the pin level cycle accurate world to the transaction level, loosely timed models. This kind of setup requires co-simulation of SystemC / C models along with RTL. Similar setup can also be created using hybrid co-emulation methodologies.
RTL verification technology has evolved to be very advanced and state of art. A typical Hardware verification language such as System Verilog, Specman E etc already have features builtin to achieve the most effective verification. Some of these features are -
- Exhaustive list of supported data types including dynamic arrays, associative arrays, queues, structs, unions , real, int etc for verification. As well as data types like bit, byte, vectored arrays, nets and regs for description of hardware.
- They supports full Object Oriented Programming which make it very easy to manage large code base of tests and environment.
- Have built in constraint specification which helps in generating useful random data for stimulus as according to constraints in design.
- Inbuilt full functional coverage modelling language without which it becomes almost impossible to measure the achievement verification of goals with random stimuli.
- Ability to fork of parallel threads of verification sequences and interprocess synchronisation using mutex and semaphores.
- Rich assertion language which makes specifying temporal property/conditions checks very easy.
Pros and Cons of this approach has been summarised below -
Advantages-
- RTL tests are already available, this saves a lot of test development effort.
- Minor to medium changes in the TB environment needed.
- Replacing RTL ip instances with SystemC ip models with some stitching code enough to start with
- Give jumpstart to model verification
- RTL simulators/Debug tools are already at state of art level , which helps a lot in quick effective debugging
- HVL like System Verilog etc. have inbuilt advanced features like Constraint Randomisation, Functional Coverage capturing, Assertions, OOPs which help with very effective and complete verification.
- As the everything is already in use for RTL verification, reuse saves lot of effort/money
- RTL verification engineers can perform Model verification without much knowledge of systemC.
Disadvantages -
- This technique is only most useful if RTL verification is already done / is in advanced stage. If model has to be verified early better to use SystemC for Verification.
- As RTL has tests have requirement for cycle/timing accuracy, all tests are not valid. Some tests are overkill and require workaround/debug in environment, which is not relevant to model verification.
- Most useful for IP level test. System level tests are a big challenge.
VERIFICATION USING SYSTEMC UVM
As electronic systems are getting complex day by day so are their virtual prototypes or simulation models. SystemC helps creating models in short time as it has constructs required to model hardware easily along with lot more other features and uses abstract communication (TLM). But as models are getting complex verifying them also is taking considerable amount of time & effort. Till now as there is no standard methodology for system level verification, companies are following their own flows and testbench creation methods, where testbenches and stimulus needs to be created for each module and also again at system level.
UVM (Universal Verification Methodology) is a standard verification methodology which specifies how to construct a test environment which is modular, scalable, configurable and reusable. UVM (or its predecessors like OVM) implementation in SystemVerilog is kind of defacto for block level/RTL verification.
Even though SystemC is the standard system level design language till now it was lacking support of a unified verification methodology. UVM-SystemC standard promises to decrease the system level verification effort by implementing UVM methodology in SystemC. With this standard there is lot of possibility for reusing test infra from module level to system level. And also it helps in achieving better test coverage. Testbench components may be reused from system level to block level also. Tests and testbenches can also be reused across verification and validation.
As SystemC-AMS helps in modeling heterogeneous systems there is also some effort going on in extending UVM for SystemC-AMS also. This will help in verifying full system model (digital + mixed-signal) with a single testbench.
Summary
As Discussed above all the 3 techniques used have their unique use case.
Directed tests are very useful to verify IPs/System when models have been integrated into Virtual Platform. It allows verification engineer to verify models keeping the perspective of the firmware engineer. As the tests are run on the on-chip processor, the tests are very near to real life use cases. But verifying individual IPs is tedious with this approach.
On the other hand, using RTL Testbench is most effective in verification of IPs standalone. It can cover vast stimulus using constrained random and generating unthinkable scenarios to catch tough bugs. It is very cost effective as Test Bench is already available and verification can be quickly kickstarted. But It is very complex to verify whole system or integrated Virtual platform.
UVM SystemC is alternatively very efficient with both IP and System level verification. It is very structured methodology, created using most effective time tested methods for verification. It makes code reuse very easy. As it is standard, Maintenance is very easy. But It is very complex to build a verification environment, takes much extra development time when done from scratch. But once developed, modification is easy.
As we can see, All the said techniques are very effective in their respective use case. No technique is best and has its pros and cons. The best use of any technique can only be made by determining various factors like schedule , complexity of design, availability of resources, cost , Verification goals et cetera. At CircuitSutra we have been pioneering Model verification since many years and with dozen of projects successfully completed. We can help companies determine and deploy the most effective or all technique for complete verification of their SystemC Model/Virtual Platform.
RSS Feed