Technical Specification Group Services and System Aspects TSGS#22(03)0679 Meeting #22, Maui, Hawaii, USA, 15-18 December 2003 Agenda Item:7.4.3S4-0307473GPP TSG-SA4 Meeting #29 Tdoc Tampere, November 24 – 28.Source: TSG SA WG4 (France Telecom R&D)Test plan for 3G packet switched conversation tests - Title:Phase 2: Comparison of quality offered by differentspeech coders.Document For: ApprovalAgenda Item: 7.4.31 IntroductionThis document proposes a conversation test plan to compare the quality obtained with several different speech coders, over packet switched networks.The different speech coders used in this test areAdaptive Multi-Rate Narrow-Band (AMR-NB), in modes 6.7 kbit/s and 12.2 kbit/s, Adaptive Multi-Rate Wide-Band (AMR-WB), in modes 12.65 kbit/s and 15.85 kbit/s,ITU-T G.723.1, in mode 6.4 kbit/s,ITU-T G.729, in mode 8 kbit/s,ITU-T G.722, in mode 64 kbit/s, with packet loss concealment and,ITU-T G.711, with packet loss concealment.As there is no standardized packet loss concealment, plc for G.711 and G.722 are proprietary algorithms.The simulated network will include two values of IP packet loss.The test will be done in one test laboratory, only, but in two different languages.This discussion gives references, conventions and contacts, section 3 details the test methodology, including test arrangement and test procedure,Annex A contains the instructions for the subjects participating to the conversation tests.Annex B contains the description of results to be provided to the Analysis Laboratory (if any) by the testing laboratories.Annex C contains the list of statistical comparisons to be performed.2. References, Conventions, and Contacts 2.1Permanent DocumentsITU-T Rec.P.800 Methods for Subjective Determination ofTransmission QualityITU-T Rec. P.831 Subjective performance evaluation of network echo cancellersITU-T Rec. G.711 Pulse code modulation (PCM) of voicefrequenciesITU-T Rec. G.729 Coding of speech at8 kbit/s using conjugate-structure algebraic-code-excited linear-prediction (CS-ACELP)ITU-T Rec. G.723.1 Speech coders : Dual rate speech coder formultimedia communications transmitting at 5.3and 6.3 kbit/sITU-T Rec. G.722 7 kHz audio-coding within 64 kbit/s2.2 Key AcronymsAMR-NB Adaptive Multi-Rate Narrowband Speech CodecAMR-WB Adaptive Multi-Rate Wide-band Speech CodecMOS Mean Opinion Score2.3 Contact NamesThe following persons should be contacted for questions related to the test plan.Section ContactPerson/Email Organisation Address Telephone/FaxExperiments and results analysis L. GrosLaeticia.gros@francetelecom.comFrance Telecom R&D 2, Avenue P. Marzin,22307 Lannion CédexFranceTel : +3329605 0720Fax : +33296051316AOB PaoloUsaiai@etsi.fr ETSI MCC 650 Route des Lucioles06921 Sophia Antipolis CedexFranceTel: 33 (0)4 92 94 42 36Fax: 33 (0)4 93 65 28 172.4 ResponsibilitiesEach test laboratory has the responsibility to organize its conversation tests. The list of Test laboratories participating to the conversation test phase.Lab Company Language1 France Telecom R&D FrenchFrance Telecom R&D Arabic3. Test methodology3.1 IntroductionThe protocol described below evaluates the effect of degradation such as delay and dropped packets on the quality of the communications. It corresponds to the conversation-opinion tests recommended by the ITU-T P.800 [1]. First of all, conversation–opinion tests allow subjects passing the test to be in a more realistic situation, close to the actual service conditions experienced by telephone customers. In addition, conversation-opinion tests are suited to assess the effects of impairments that can cause difficulty while conversing (such as delay).Subjects participate to the test by couple; they are seated in separate sound-proof rooms and are asked to hold a conversation through the transmission chain performed by means of networks simulators and communications are impaired by means of an IP impairments simulator part of the CN simulator, as the figure below describes it. 3.2 Test arrangement3.2.1 Description of the proposed testing systemThis contribution describes a networks simulator for the characterization of the different speech codecs when the bitstream is transmitted over a PS network. The procedure to do the conversational listening test has been earlier described in [1].Figure 1 describes the system that is going to be simulated:Figure 1: Packet switch audio communication simulatorThis will be simulated using 5 PCs as shown in Figure 2.Terminal Figure 2: Simulation PlatformPC 1 and PC 5: PCs under Windows OS with VOIP Terminal SimulatorSoftware of France Telecom R&D.PC 3: PCs under WinNT OS with Network Simulator Software(NetDisturb).Basic Principles:The platform simulates a packet switch interactive communication between two users using PC1 and PC5 as their relatives VOIP terminals. PC1 sends encoded packets that are encapsulated using IP/UDP/RTP headers to PC5. PC1 receives these IP/UDP/RTP audio packets from PC5.3.2.2 France Telecom Network simulatorThe core network simulator, as implemented, works under IPv4.Figure 3 shows the possible parameters that can be modified, but, in this test, only "loss Law" will have two values, all the others settings being fixed.Figure 3: IP simulator interfaceOn both links, one can choose delay and loss laws. Both links can be treated separately or on the same way. For example, delay can be set to a fixed value but can also be set to another law such as exponential law.3.2.3 Headsets and Sound CardTo avoid echo problems, it has been decided to use headsets, instead of handsets. The monaural headsets are connected to the sound cards of the PCs supporting the AMR simulators.The sound level in the earphones can be adjusted, if needed, by the users. But, in practice, the original settings, defined during the preliminary tests, and producing a comfortable listening level, will not be modified. The microphones are protected by a foam ball in order to reduce the "pop" effect. It is also suggested to the user to avoid to place the acoustic opening of the microphone in front of the mouth.3.2.4 Test environmentEach of the two subjects participating to the conversations is installed in a test room. They sit on an armchair, in front of a table. The test rooms are acoustically insulated. All the test equipments are installed in a third room, connected to the test rooms. The background noise level is checked by a sound level meter. The measurement microphone, connected to the Sound level meter is located at the equivalent of the center of the subject's head. The noise level is A weighted.3.2.5 Calibration and test conditions monitoringSpeech levelBefore the beginning of a set of experiment, the end to end transmission level is checked subjectively, to ensure that there is no problem. If it is necessary to check the speech level following procedure will apply. An artificial mouth placed in front of the microphone of the Headset A, in the LRGP position -See ITU-T Rec. P.64-, generates in the artificial ear (according to ITU-T Rec. P57) coupled to the earphone of the Head set B the nominal level defined in section 4.3. If necessary, the level is adjusted with the receiving volume control of the headset. The similar calibration is done by inverting headsets A and B.DelayThe overall delay (from the input of sound card A to the output of sound card B) will be adjusted for each test condition taking into account the delay of the related codec in order to have a fixed delay around 250ms. This value of 250ms is close to the hypothetical delay computed for AMR and AMRWB through the UMTS network.3.3 Test ConditionsactorsCondition ExperimentalconditionsIPMode(Packet loss ratio)1 0%AMR NB 6,7kbit/s2 0%AMR-NB 12,2 kbit/s3 0%AMR-WB12,65 kbit/s4 0%AMR-WB15,85 kbit/s5 0%G. 723.16,4 kbit/s6 0%G.7298 kbit/s7 0%G.72264 kbit/s + plc8 0%G.711 + plc9 3%AMR NB 6,7kbit/s10 3%AMR-NB 12,2 kbit/s(delay 300 ms)11 3%AMR-WB12,65 kbit/s12 3%AMR-WB15,85 kbit/s13 3%G. 723.16,4 kbit/s14 3%G.7298 kbit/s15 3%G.72264 kbit/s + plc16 3%G.711 + plcListening Level 1 79 dBSPLListeners 32 Naïve Listeners per languagesubjects/group2Groups 16Rating Scales 5tableLanguages 1SeeListening System 1 Monaural headset (flat response in the audiobandwidth of interest: 50Hz-7kHz). The other ear isopen.Listening Environment Room Noise: Hoth Spectrum at 30dBA (as defined byITU-T, Recommendation P.800, Annex A, sectionA.1.1.2.2.1 Room Noise, with table A.1 and FigureA.1),1 ReferencesTdoc S4-030564- Test Plan for the AMR Narrow-Band Packet switchedConversation testTdoc S4-030565- Test Plan for the AMR Wide-Band Packet switchedConversation testENDAnnex A Example Instructions for the conversation testTable : Instructions to subjects.INSTRUCTIONS TO SUBJECTSIn this experiment we are evaluating systems that might be used for telecommunication services.You are going to have a conversation with another user. The test situation is simulating communications between two mobile phones. All the situations will correspond to silent environment conditionAfter the completion of each call conversation, you will have to give your opinions on the quality, by answering to the following questions that will be displayed on the screen of the black box in front of you. Your judgment will be stored. You have 8 seconds to answer to each question. After "pressing" the button on the screen, another question will be displayed. You continue the procedure for the 5 following questions.Question 1: How do you judge the quality of the voice of your partner?Excellent Good Fair Poor Bad Question 2: Do you have difficulties to understand some words?All the time Often Some time totimeRarely Never Question 3: How did you judge the conversation when you interacted with your partner?Excellent interactivity (similar to face-to-face situation) Goodinteractivity(in few moments,you were talkingsimultaneously,and you had tointerruptyourself)Fair interactivity(sometimes, youwere talkingsimultaneously,and you had tointerruptyourself)Poor interactivity(often, you weretalkingsimultaneously,and you had tointerruptyourself)Bad interactivity(it wasimpossible tohave aninteractiveconversation)Question 4: Did you perceive any impairment (noises, cuts,…)? In that case, was it:No impairment Slightimpairment, butnot disturbing ImpairmentslightlydisturbingImpairmentdisturbingVery disturbingImpairmentQuestion 5: How do you judge the global quality of the communication?Excellent Good Fair Poor Bad From then on you will have a break approximately every 30 minutes. The test will last a total of approximately 60 minutes.Please do not discuss your opinions with other listeners participating in the experiment.Annex B: Example Scenarios for the conversation testThe pretexts used for conversation test are those developed by the Ruhr University (Bochum, Germany) within the context of ITU-T SG12 . These scenarios have been elaborated to allow a conversation well balanced within both participants and lasting approximately 2’30 or 3’, and to stimulate the discussion between persons that know each other to facilitate the naturalness of the conversation. They are derived from typical situations of every day life: railways inquiries, rent a car or an apartment, etc. Each condition should be given a different scenario.Examples coming from ITU-T SG 12 COM12-35 "Development of scenarios for short conversation test", 1997• Scenario 1 : Pizza service Subject 1:Your Name :ClemenceReason for the call 1 large PizzaCondition which should be applied to the exchange of information For 2 people,Vegetarian pizza preferredInformation you want to receive from your partner ToppingPriceInformation that your partner requires Delivery address : 41 industrystreet,OxfordPhone : 7 34 20Question to which neither you nor your partner will have information.You should discuss and find asolution that is acceptable to both ofyou.How long will it take?Subject 2:Your Name :Pizzeria RomaInformation from which you should select the details which your partner requiresPizzas 1 person 2 persons 4personsToscana (ham, mushrooms, tomatoes, cheese)3.2£ 5.95£10.5£Tonno (Tuna, onions, tomatoes, cheese)3.95£ 7.5£13.95£Fabrizio (salami, ham, tomatoes, cheese)4.2£ 7.95£14.95£Vegetarian (spinach, mushrooms, tomatoes, cheese)4.5£ 8.5£15.95£Information you want to receive from your partner Nameaddresstelephone numberQuestion to which neither you nor your partner will haveinformation.You should discuss and find asolution that is acceptable toboth of you.• Scenario 2 : Information on flights Subject 1:Your Name :ParkerReason for the call Intended journey: London Heathrow Düsseldorf Condition which should be applied to the exchange of information On June 23rd,Morning flight,Direct flight preferredInformation you want to receive from your partner Departure :ArrivalFlight numberInformation that your partner requires Reservation : 1 seat, Economy classAddress: 66 middle street, SheffieldPhone: 21 08 33Question to which neither you nor your partner will have information. You should discuss and find a solution that is acceptable to both ofyou.From which airport is it easier to getinto Cologne center : Düsseldorf orCologne/BonnSubject 2:Your Name :Heathrow flight informationInformation from which you should select the details which your partner requiresFlight schedule Lufthansa British AirwaysLufthansa Flight number LH 2615 BA 381 LH 413London Heathrow departure6:30 6:35 8:20Brussels arrival Brussels departure 7:35 8:00Düsseldorf arrival 7:35 9:05 9:25Information you want to receive from your partner Nameaddresstelephone numbernumber of seatsClass : Business or EconomyQuestion to which neither you nor your partner will haveinformation.You should discuss and find asolution that is acceptable toboth of you.ITU-T SG 12 COM12-35 "Development of scenarios for short conversation test", 1997Annex C: Results to be providedFor contractual purposes, the information which needs to be provided is defined here.The information required from each test Laboratory is a table containing the following information for each of the conditions in the experiment:The "Mean Opinion Score (MOS)" obtained for all the subjects.When the conditions are symmetrical, the mean value is calculated from all the result for the two test rooms..For the dissymmetric conditions, the mean is calculated on the two test conditions, each result cumulating the results obtained in each condition of background noise.The Standard Deviation of the "MOS" obtained for all the subjects, for each test condition. The specific statistical comparisons are specified in Annex C.Annex D: Data analysis and presentation of resultsD.1 Calculation of MOS and Standard DeviationThe (overall) MOS/DMOS for confounded subjects for condition C (Yc) can then be obtained from:Y T Yc c,t t 1T ==∑1The standard deviation (S) for condition C, denoted as Sc, can be calculated as:S c = 1 L T 1 (XY ) c,l,t c 2l 1L t T ×−−==∑∑1Finally, the confidence interval (CI) at the (1-α) level can be calculated for T L N ×= as:N S )(CI c1,1c −−=N t αD.2 Presentation of Basic Statistical ResultsThe test results should be reported by the test Laboratory and the Global Analysis Laboratory as follows:Calculate and tabulate "Mean Opinion Scores" for the (opinion scales, Standard Deviations and Confidence Intervals as shown in Table E.1.Table C.1 - Layout for presentation of test results.D.3 Thorough analysisTwo statistical analyses should be conducted on the data obtained with these subjective scales. The first analysis consists in a Multiple ANalysis OF VAriance (MANOVA), which globally indicates the possible effect of the experimental factors (i.e., different conditions). Then, a specific ANOVA should be run on each dependent variable (the five scales) to test if there is an effect of a specific experimental factor for a given subjective variable. In other words, these statistical analyses indicate if the differences observed between the MOS obtained for the different conditions are significant, for one given dependant variable (ANOVA) or for the whole of dependant variables (MANOVA). Finally, Pearson's linear correlations should be computed between the results of all subjective variables, to see which are those preponderant or dependent on others.。