Is Your Mobile Phone Smart? Antutu AI Benchmark Public Beta Is Released

At present, the word AI is not new to us. Whether in terms of PC end, mobileend, mobile phone or car, it seems to be a bit out of date without mentioningAI. AI has become relevant to our lives, affecting every aspect of our lives.

Though AI is strictly an algorithm, the support from software and hardware hasa close relationship to it, with the integration between them bringing aperformance difference.

In order to provide you with an intuitive judgment on AI performance of yourmobile phone, Antutu formally released the "Antutu AI Benchmark"public beta, giving you a quantifiable standard to determine the difference inAI performances between different platforms.

However, it is important to note that thereis no unified standard about AI in the industry at present and different chipvendors have various understanding to AI. For example, Qualcomm processesAI-related operations through DSP inside SoC while Hisilicon Kirin performs AIoperations through an internal separated NPU. In addition, Samsung and MTK alsohave added separated AI chips in their next generation chips, which are knownas NPU for Samsung and APU for MTK.

Apart from differences in hardware, there is also no unified standard in SDK,and each vendor offers a SDK for its own AI chip, whereas SDK of Qualcomm iscalled SNPE while that of MTK, Hisilicon Kirin and NVIDIA are calledNeuoroPilot, HiAI and TensorRT respectively (SDK of Samsung has not beenpublished by far). These SDKs will be presented in the Antutu AI Benchmark.

Antutu has established a unified standard for testing AI Performance by workingwith the above manufacturers. The test is divided into two subcategories: imageclassification and object detection.

Among them, the image classification is based on the Inception v3 neuralnetwork, with 200 images as test data; and that object detection is based onthe mobilenet ssd neural network, with a video of 600 frames as test data.

Antutu recommends users to test in Wi-Fi status as the software installationpackage size is only 33 MB, but the data package is up to 160 MB. Data packagewill be automatically downloaded when users click start to test for the firsttime (there will be a prompt in non-Wi-Fi network status). A verification forintegrity of data package will be carried out after download is completed. Thenthe test will formally start.

Data package download is required only when first running Antutu AI Benchmark.In case of subsequent re-testing, the system will verify the integrity of datapackage each time other than downloading it again. But, in case of damage, thedata package needs to be re-downloaded prior to test.

In addition, different platforms have data packages with different sizes sothat Antutu converts the original neural network into the manufacturerssupported neural networks with SDKs provided by each manufacturer. Although thedata package size is different, the final test sample is identical and allchips are tested under a unified standard.

If the chip itself does not support AI-related algorithms, or the manufacturerprovided SDK is temporarily unable to support the network of Antutu AI Benchmark,image classification and object detection will use TFLite to call CPU foroperation, which is low in efficiency with poor performance.

Scoring Standard:

1. The score is related to speed and accuracy, namely, the faster speed andhigher accuracy enable a better final score.

2. When the speed is faster but with a pooraccuracy, Antutu AI Benchmark will adopt targeted penalty measures to keepfinal score from being influenced by cheating that decreasing the accuracy toincrease the speed, and vice versa.

Special Cases Description:

1. Because Antutu AI benchmark focuses onAI operational capability, there is no obvious difference in performance forthe same AI processors (for example, the DSP models of Qualcomm 845 and 710 arethe same, so the score difference between them is marginal).

2. Samsung has not yet released its AI SDK, so testing for Samsung chip is byits CPU, leading to poor performance. This situation will be improved afterSamsung releases its SDK.

3. The Hisilicon's HiAI engine object recognition currently uses TFLite toperform operations on the CPU, resulting in low scores. This situation willalso be improved after HiAI is upgraded by Hisilicon.

4. Mobile chip of NVIDIA completes AI algorithms through GPU with floatingpoint algorithms by TensorRT engine.

5. Android versions also have an impact on test results, and theoretically theperformance of the same chip on Android 9.0 will be higher than that in Android8.0, as Google has been optimizing support for AI at the system level.

Bonus:

After the release of Antutu AI benchmark official version, Antutu will releaseAntutu AI Mobile Tool at appropriate time to make things convenient fordevelopers to call hardware acceleration function on each platform and speed upthe development of mobile AI application industry.