CTS ( COMPATIBILITY TESTSUITE )USER GUIDEAndroid-ALL-CTS USER GUIDERev. 1.10July 23, 2011DISCLAIMERAll information and data contained in this material are without any commitment, are not to be considered as an offer for conclusion of a contract, nor shall they be construed as to create any liability. Any new issue of this material invalidates previous issues. Product availability and delivery are exclusively subject to our respective order confirmation form; the same applies to orders based on development samples delivered. By this publication, T elechips, Inc. does not assume responsibility for patent infringements or other rights of third parties that may result from its use.Further, Telechips, Inc. reserves the right to revise this publication and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes.No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of Telechips, Inc.This product is designed for general purpose, and accordingly customer be responsible for all or any of intellectual property licenses required for actual application. Telechips, Inc. does not provide any indemnification for any intellectual properties owned by third party.Telechips, Inc. can not ensure that this application is the proper and sufficient one for any other purposes but the one explicitly expressed herein. Telechips, Inc. is not responsible for any special, indirect, incidental or consequential damage or loss whatsoever resulting from the use of this application for other purposes.COPYRIGHT STATEMENTCopyright in the material provided by Telechips, Inc. is owned by Telechips unless otherwise noted.For reproduction or use of Telechips‟ copyright material, permission should be sought from Telechips. That permissi on, if given, will be subject to conditions that Telechips‟name should be included and interest in the material should be acknowledged when the material is reproduced or quoted, either in whole or in part. You must not copy, adapt, publish, distribute or commercialize any contents contained in the material in any manner without the written permission of Telechips. Trade marks used in Telechips‟ copyright material are the property of Telechips.Important NoticeThis product may include technology owned by Microsoft Corporation and in this case it cannot be used or distributed without a license from Microsoft Licensing, GP.For customers who use licensed Codec ICs and/or licensed codec firmware of mp3:“Supply of this product does not convey a license nor imply any right to distribute content created with this product in revenue-generating broadcast systems (terrestrial. Satellite, cable and/or other distribution channels), streaming applications(via internet, intranets and/or other networks), other content distribution systems(pay-audio or audio-on-demand applications and the like) or on physical media(compact discs, digital versatile discs, semiconductor chips, hard drives, memory cards and the like). An independent license for such use is required. For details, please visit ”.For customers who use other firmware of mp3:“Supply of this product does not convey a license under the relevant intellectual property of Thomson and/or Fraunhofer Gesellschaft nor imply any right to use this product in any finished end user or ready-to-use final product. An independent license for such use is required. For details, please visit ”.For customers who use Digital Wave DRA solution:“Supply of this implementation of DRA technology does not convey a license nor imply any right to this implementation in any finished end-user or ready-to-use terminal product. An independent license for such use is required.”For customers who use DTS technology:“Supply of this implementation of DTS technology does not convey a license, exhaust DTS‟ rights in the implementation, or imply a right under any patent, or any other industrial or intellectual property right of DTS to use, offer for sale, sell, or import such implementation in any finished end-user or ready-to-use final product. Notice is hereby provided that a license from DTS is required prior to such use.”“This product made under license to U.S. Patents 5,451,942; 5,956,674; 5,974,380; 5,978,762; 6,487,535; 6,226,616 and/or foreign counterparts.”"© 1996 –2010 DTS, Inc.”For customers who use Dolby technology:"Supply of this Implementation of Dolby technology does not convey a license nor imply a right under any patent, or any other industrial or intellectual property right of Dolby Laboratories, to use this Implementation in any finished end-user or ready-to-use final product. It is hereby notified that a license for such use is required from Dolby Laboratories."Revision HistoryTABLE OF CONTENTSContents1 Introduction ......................................................................................................................................................................... 1-11.1 How does the CTS work? ......................................................................................................................................... 1-12 Setup environment .............................................................................................................................................................. 2-22.1 Set up Test PC .......................................................................................................................................................... 2-22.1.1 JDK ................................................................................................................................................................. 2-22.1.2 Preparing Android SDK ................................................................................................................................... 2-32.1.3 Setup Path ...................................................................................................................................................... 2-32.1.4 CTS installation............................................................................................................................................... 2-42.1.5 Setup CTS configuration ................................................................................................................................. 2-42.1.6 Setup target device ......................................................................................................................................... 2-52.2 Start CTS .................................................................................................................................................................. 2-62.2.1 Start CTS ........................................................................................................................................................ 2-62.2.2 Execute a Command ...................................................................................................................................... 2-72.3 Customizing CTS .................................................................................................................................................... 2-102.3.1 Add and Use a Plan ...................................................................................................................................... 2-102.3.2 Debugging .................................................................................................................................................... 2-113 Clearing problems ............................................................................................................................................................. 3-133.1 ADB Connection ..................................................................................................................................................... 3-133.2 CTS Connection ..................................................................................................................................................... 3-134 Appendix ........................................................................................................................................................................... 4-144.1 Checking peripherals .............................................................................................................................................. 4-144.2 Checking applications ............................................................................................................................................. 4-144.3 References ............................................................................................................................................................. 4-141 IntroductionCTS (Compatibility Test Suite) is a tool to test the compatibility of android devices. This checks whether APIs required in the Android standard SDK exist and they work properly. This is necessary to guarantee application programs made according to the Standard SDK development guide properly work with CTS certified Android compatibility devices. Therefore, Pass in CTS is the basic condition for accessing to the Android market. In addition, the Google policy requires not only market programs but also various applications called GMS (Google Mobile Service) such as Gmail, Maps, Search, YouTube, etc to pass a Compatibility Test in order to get a license.Since CTS is a software based test, it is not sufficient. Therefore, you can find a guide to the composition of actual devices. If you refer to the document “Compatibility Definition” which can be obtained from Google once you execute a NDA with Google, you will see the document explains well peripherals, input and output configuration of devices, and application operation style required for being certified as an Android compatible device. This device guide seems to update according to the market trend or the Android development status. 1.1 How does the CTS work?The CTS is an automated testing harness that includes two major software components:1. The CTS test harness runs on your desktop machine and manages test execution.2. Individual test cases are executed on attached mobile devices or on an emulator. The test cases arewritten in Java as JUnit tests and packaged as Android .apk files to run on the actual device target.- Copied to /compatibility/cts-intro.html2 Setup environmentrefer to /compatibility/cts-intro.html2.1 Set up Test PCyou will need to use Linux and Ubuntu (10.04 and later)●OS : Linux, Ubuntu 10.042.1.1 JDKYou can download JDK from and set up it at /root/ of Test PC system.●If you wish to test Gingerbread, set up JDK 6Fie Name : jdk-1_5_0_19-linux-i586.bin/javase/downloads/index_jdk5.jsp●If you wish to test Froyo, set up JDK 5File Name : jdk-6u24-linux-i586.binhttps:///is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/ViewFilteredProducts-SingleVariationTypeFilter2.1.2 Preparing Android SDK1. Download TCC Android platform from GIT server to Android building machine(Linux PC) through “repo sync”.2.Run “make sdk” at “local path : ~/Android_platform_home/” of Android build machine (Linux PC),then Android-SDK_xxxx.xxxx_linux-x86.zip is generatedat “local path : ~/Android_platform_home/out/host/linux-x86/sdk/”3.Copy Android-SDK_xxxx.xxxx_linux-x86.zipfrom “local path : ~/Android_platform_home/out/host/linux-x86/sdk/” of Android building systemto “root_path: /home/(user_ID)/” of CTS test PC(Linux OS),then decompress it.2.1.3 Setup PathAdd the followings in ~/.bashrc or ~/.bash_profile file. (local path)2.1.4 CTS installationrefer to /compatibility/downloads.html1. Download TCC Android platform from GIT server to Android building machineLinux PC) through “repo sync”.If you already downloaded it, you can skip this step.2. Run “make cts” at “local path : ~/Android_platform_home/” of Android build machine (Linux OS),then android-cts.zip is created at “local path : ~/Android_platform_home/out/host/linux-x86/cts/”.3. Copy android-sdk.zipfrom “local path : ~/Android_platform_home/out/host/linux-x86/cdk/” of Android building systemto “root_path: /home/(user_ID)/” of CTS test PC(Linux OS),then decompress it.2.1.5 Setup CTS configurationFor Froyo and older versions, edit the startcts script in android-cts directory.In the script, you will find SDK_ROOT=NOT_CONFIGURED.Change NOT_CONFIGURED to the root path of the android-sdk-linux_86 as below.Note > Gingerbread or beyond doesn‟t this configuration.2.1.6 Setup target deviceAfter your machine(CTS test PC) set up1. Attach at least one device (or emulator) to you machine2. For CTS 2.1 R2 and beyond, setup your device (or emulator) to run the accessibility tests:adb install –r android-cts/repository/testcases/CtsDelegatingAccessibilityService.apkOn the device, enable Settings > Accessibility > Accessibility > Delegating Accessibility Service3. For CTs 2.3 R4 and beyond, setup your device to run the devices administration testsadb install –r android-cts/repository/testcases/CtsDeviceAdmin.apkOn the device, enable all the android.deviceadmin.cts.* device administratorsunder Settings > Location & security > Select devices administrators.4. Launch the CTS. The CTS test harness loads the thst plan onto the attached devices.For each test in the test harness.The test harness pushes a apk file to each device, executes the test through instrumentation,And records test resultsThe test harness removes the apk file from each device.2.2 Start CTSNow, Setup of CTS environment is finished.Please connect PC to target android device through USB interface cable.2.2.1 Start CTSRun “adb device” and “startcts” at path “local_path : ~/android-cts/tool/”. If target device is connected with CTS,”Device (0123456789ABCDEF) connected” is displayed.If not, “Device (0123456789ABCDEF) disconnected” is displayed.2.2.2 Execute a CommandIf you enter help, you will see various options. For example, if you want to test one certain packet, you can do as follows.그림 3 CTS plans list# ls –-planHere, you can see the basically provided plans The plans should be formed in xml file and placed in repository -> plans .# ls –-plan AndroidYou can check the packages in Android Plan.그림 4 fail result for local test of Signature planIf you choose Signature over a couple of plans and execute the CTS test, you will get the result as above. This figure shows the result is …fail‟.The following figure shows the result is …pass‟.그림 5 pass result for local test of Ref App그림 6 generated location path of CTS test result filesIf CTS test is completed, in the local-path “local_path : ~/android-cts/repository/results” directory, CTS test timing becomes the title of the directory and the results are saved to the directory. If you enter into your wanted directory and check testResult.xml, you can easily check data that you want to see.그림7 tResult.xml file viewed with firefox[web browser※If you execute CTS test, it takes a long time (approx. 8 hours) to test around each of 21,000.2.3 Customizing CTS2.3.1 Add and Use a PlanYou can use this when you want to remove unnecessary packages and test only necessary packages.그림8 screen of selecting planscts_host> add –plan MyTest: A new plan called MyTest is created. At this time, you can select a package.cts_host> start –plan MyTest: You can use the created MyTest plan as above.2.3.2 DebuggingIf a result of CTS test is …fail‟, you need debugging to analyze causes for the failure.The log as the following figure will remain.그림9 debugging message for failYou sometimes might have some difficulty with finding a problem from a platform source. If so, directly refer to CTS source in order to easily access to debugging.A path is divided per package in cts/tests/test.a.Enter a log message in the location of a file where an error occurs.b.After Rebuild, install CTS on a test PC. (Refer to 2-(1)-⑤.)c.Reset the environment setting of the CTS (Refer to 2-(2)-①).d.Re-execute the CTS and check the log message. (Refer to 2-(2)-②.)For example, in the above figure CameraTest.java file exist in the following path.3 Clearing problems3.1 ADB ConnectionProblem : If ADB of a device is unstable, time out can be happened frequently.Solution :-Use a shield cable.-Connect a target board directly to a test PC, not use an extension USB port.3.2 CTS ConnectionCTS reboots a device periodically during CTS test.Problem : For some reasons, CTS can not be connected after rebooting the device.Solution :-Physically reboot a device until the message “connected”on the CTS appears.-With adb kill-server, turn off adb and turn on it with adb start-server.4 Appendix4.1 Checking peripheralsTo run CTS successfully, target device has to be equipped with several peripherals.GPS/G-sensor/Compass1. Check Wi-FiEnable Settings > Wi-Fi >Wireless & networks > Wi-FiEnable Settings > Wi-Fi >Wireless & networks > Wi-Fi setting2. Check Location & security settingsEnable Settings > Location & security settings > Use GPS satellites4.2 Checking applicationsBesides peripherals, some applications has to be installed and some feature has to be selected to run CTS successfully, 1. Check Display SettingEnable Settings > Display Setting > Auto-rotate screenCheck Settings > Display Setting > Enable Screen timeout : Max (30 minute)2. Check AccessibilityEnable Settings > Accessibility > Accessibility > Delegating Accessibility Service3. Check ApplicationsEnable Setting > Applications > USB debuggingEnable Setting > Applications > Allow mock locations4. Check Text-to-speech settingsCheck up on TTS engines (ex. Pico TTS).4.3 References1. Android2.1 Compatibility Definition ( Android 2.1 CTS r1 Open Handset Alliance )2. Compatibility Test Suite Framework User Manual ( Android 2.1 CTS r2 Open Handset Alliance )3. Android 2.2 Compatibility Definition4. Android 2.3 Compatibility Definition3. /android/mydroid/development/pdk/docs/instrumentation_framework.html。
