What is a Trial?

When working with these ranges, you should aim for as tight an interval as possible. If it is too wide, it is difficult to see where the true average will lie. The more runs you perform in a Trial, the more your Confidence Interval should converge. Using ranges in a Trial in this way will indicate how accurate you can expect your results to be and allows us to ensure their reliability. The results at the end of a trial include a 95% or 99% confidence interval with SIMUL8 Standard edition. With SIMUL8 Professional edition you get the additional options of 80% 90% 98% 99.8% and 99.9%.

Why one run is never enough?

When using SIMUL8, if you only do one run, then reset the clock back to zero and re-run the simulation you will see exactly the same things happen on the screen, in the same sequence as the last run. Each time the simulation is re-started in the normal way, the same sequence of random numbers will be generated.

This is very useful when debugging as it means you can watch a simulation several times to understand exactly what is happening, without the issue being clouded by the random numbers changing each time. This is a crucial feature in helping you analyze the impact of changes to input parameters. It also means that you can be sure that any observed change that you see in your results are directly related to the changed input parameters entered and not random numbers.

When analyzing your simulation, if you run it only once, there is no way of knowing if the results are a good indication of reality, or if they have been over/under-estimated. For example, analyzing the results from just one run could mean that you predict less machine downtime than what would actually happen in the real system or an incorrect prediction on capacity requirements. This is one reason why it is important to run trials. Trials give you more realistic predictions as results are stated within confidence intervals.

How many runs then?

A general recommendation is to run a Trial with as many runs as possible. However, clearly there is a trade-off here as the more runs you do, the less analysis you can do in a given timeframe. As a rule of thumb, try starting with a trial of 30–50 runs depending on simulation size and complexity.



Alternatively, a better way to decide may be to gradually increase the number of runs in your Trial and monitor the range of a key result until a certain level of tolerance is reached. When a Trial has completed you can extend it to include a new number of runs to increase its accuracy and narrow confidence limits. This can be done without the need to re-run the previously executed runs. This can be quite a tedious and time consuming process to go through so we have automated this analysis for you and have developed the Trial Calculator feature.

SIMUL8 Trial Calculator

The Trial Calculator is a new feature that we introduced in our SIMUL8 2008 software.This makes it easy for the user to decide the number of runs you should conduct in order to ensure accurate trial results as it actually calculates this for you! This feature resulted from extensive research conducted by Dr Katy Hoad and Professor Stewart Robinson of Warwick University.

How does the Trial Calculator work?

This works by specifying a chosen tolerance for the key results you are interested in, and running until this tolerance is reached. This feature is accessed by clicking the 'Calculate Required Number of Runs' button within the Conduct Trial dialog window.

All results (KPIs) available for this calculation are given in the box on the left. To select some of these to meet your required level of tolerance, move them to the box on the right using the arrow buttons in the middle. The precision (% of the mean result) can be set too. The smaller the chosen precision the more runs will be required to achieve it, so it is recommended to run the calculator with a higher precision level first, to get an idea of runs required before tuning the precision number and KPIs chosen and running the calculator again.

Finally, before clicking 'OK' make sure you have highlighted the KPI in the box on the right you would most like to see graphed. After clicking 'OK' the calculator will run and Recommendations will be displayed once your required precision has been met. If you want to stop the calculator before this there is a button to do so called 'End After Current Run'.

In the Recommendation above, you can see 155 and 36 runs are advised to achieve the desired 25% accuracy precision level for this simulation. So overall, a minimum of 155 runs should be conducted to get the overall accuracy for the KPIs you have selected. However in the example below you can see that a much smaller number of runs are required to achieve a tighter accuracy precision level for the KPIs that you have chosen to analyze. A minimum of 15 runs are recommended to achieve the desired 5% accuracy precision level.

Further Information

For further information on the extensive research carried out relating to the Trial Calculator feature visit  Warwick University Research Site.

If you have any questions on this feature or require further information then please contact our support team at support@SIMUL8.com. We would be very happy to help you.