Tenth Quarterly Progress Report

November 1, 1997 through January 31, 1998

NIH Project N01-DC-5-2103

Speech Processors for Auditory Prostheses

Prepared by

Mariangeli Zerbi, Dewey Lawson and Blake Wilson

Center for Auditory Prosthesis Research

Research Triangle Institute

Research Triangle Park, NC 27709

Contents

I. Introduction

II. Effects of Upward Extension of the Frequency Range Analyzed by CIS Processors

III. Plans for the Next Quarter

IV. Acknowledgments

Appendix 1: Summary of Reporting Activity for this Quarter

I. Introduction

One of the principal objectives of this project is to design, develop, and evaluate speech processors for implantable auditory prostheses. Ideally, the processors will represent the information content of speech in a way that can be perceived by implant patients. Another principal objective is to develop new test materials for the evaluation of speech processors, given the growing number of cochlear implant subjects enjoying levels of performance too high to be sensitively measured by existing tests.

Work in the present quarter included:

• Studies with subject ME2 (October 27 to November 14), who has full insertions of Med El COMBI 40+ implants on both sides. The studies included psychophysical measures of sensitivities to differences in the amplitudes and timing of stimuli delivered to the two implants and evaluation of a wide range of speech processing strategies designed to exploit such sensitivities and/or the additional sites of stimulation supported by two implants. The speech reception tests were conducted with materials in German, the subject's native language. Additional investigators participating in the studies with ME2 included Stefan Brill from the University of Innsbruck, Sigfrid Soli from the House Ear Institute in Los Angeles, and Joachim Müller from the Julius-Maximilians-Universität in Würzburg, Germany. Stefan Brill served as a Guest Co-Investigator and also served as a translator for the subject. Mr. Brill had made major contributions (along with Otto Peter and Erwin Hochmair) to the development of the interface system for simultaneous laboratory control of two Med El implants, and also to the design of the studies with ME2. Sigfrid Soli is a consultant for our project, and participated in the studies with ME2 for the week of November 3. Dr. Soli is an internationally recognized authority on binaural processing and on measures of speech reception, for both unilateral and bilateral stimulation. He and Dr. Michael Nilsson, also with the House Ear Institute, incorporated German language test materials into a system they have developed for use in studies of binaural processing and perception. That system, with those incorporated materials, was used in the studies with ME2. Dr. Müller participated in the studies for one and one half weeks beginning on November 3. He also served as a translator for ME2. Dr. Müller was the implanting surgeon for ME2.
• Ongoing studies with Ineraid subject SR2 (usually one morning per week). Studies in this quarter included investigation of effects of various "conditioner" stimuli, such as high rate pulse trains with uniform pulse amplitudes, on the neural representations and perception of superposed "deterministic" stimuli, such as a low rate pulse train with pulse amplitudes greater than that of the conditioner. The investigation included recordings of intracochlear evoked potentials and psychophysical measures of decays in loudness of conditioners presented over long periods (e.g., minutes).
• Development and coding of new speech processing strategies using high-rate conditioner stimuli in addition to the normal outputs of CIS processors.
• A visit by Margo Skinner (various times during the week of January 5). Among the topics of discussion were (1) choice and reliability of speech test materials for studies with implant patients, (2) measures of electrode positions in implanted patients, (3) possible psychophysical and speech reception correlates of electrode position, and (4) use of intracochlear EP recordings for inferring the physiological status of the auditory nerve in implant patients on an electrode-by-electrode basis.
• A trip to Iowa City by Wilson (January 27 and 28), primarily to discuss coordination between the University of Iowa and RTI in studies with 10 subjects implanted, or to be implanted, at the University of Iowa with CI24M devices on both sides.
• Presentation of project results in an invited lecture by Wilson at the University of Iowa (RTI experience from studies with two bilateral subjects).
• Continued preparation for studies with the Iowa bilateral subjects, including (1) development of an interface system, with the cooperation of Cochlear Ltd., for simultaneous laboratory control of two CI24M implants, and (2) modification of speech processor and psychophysical test code for use with the interface system.
• Verification of correct operation of an interface system developed by us years ago for laboratory control of the implanted receiver in the Clarion device, in anticipation of studies with subjects having Clarion implants on one side.
• Modification of speech processor and psychophysical test code for use with the Clarion interface system.
• Further development of instrumentation in the evoked potentials laboratory, along the lines indicated in Quarterly Progress Report 9 for this project.
• Continued analysis of speech reception, evoked potential, and psychophysical data from prior studies.
• Continued preparation of manuscripts for publication, especially one in collaboration with Jay Rubinstein and Paul Abbas, on ways in which spontaneous activity can be reinstated in the auditory nerves of implant patients.

In this report we describe in detail studies to evaluate possible effects of extending the overall frequency range analyzed and represented by CIS processors. (Preliminary findings were presented in Quarterly Progress Report 1 for this project.)

Results from other studies and activities indicated in the list above, such as results from the studies with bilateral subject ME2, will be presented in future reports.

II. Effects of Upward Extension of the Frequency Range Analyzed by CIS Processors

Introduction

In tests with CIS processors we have observed consistent differences in consonant identification scores between female and male voices. This has led us to a careful comparison of the representation of male and female voices by our CIS processor implementations. Typical CIS processors have had an input spectral range of 350 to 5500 Hz. We have determined that this eliminates useful information from the high frequency consonants of female speakers and some low frequency voicing information from both male and female consonants.

Background

We have compiled a list of most frequent consonant confusions among data for four cochlear implant subjects using CIS processors that spanned our customary overall frequency range of 350Hz to 5500Hz. Tables 1 and 2 indicate the relative incidence of those confusions (among 40 presentations of each token across four subjects), and processor manipulations that might reduce their incidence. Among these proposed improvements are including an additional high frequency channel (one covering 5656Hz to 9000Hz), using no pre-emphasis equalization (allowing more voicing information in channel one), and including an additional low frequency channel (80Hz to 350Hz).

The four subjects represented in these data included two with 6-electrode Ineraid implants and two with percutaneous versions of the Nucleus 22-electrode implant. Preliminary data for three of these four subjects -- NP1, NP2, and SR3 -- were presented in QPR1 of the current contract. More complete data for subject SR10 subsequently replaced those presented for subject SR2 in QPR1. The tests of identification of 16 medial consonants were conducted in a hearing alone condition with no feedback as to correct or incorrect responses. Each subject was using the six-channel CIS processor that had yielded his/her best overall performance to date. The processing hardware was identical except for stimulus current sources. All the processors used full-wave rectification in estimating energy levels for representation on each channel, and the same type of compression table for mapping stimuli to each subject's thresholds and most comfortable loudness levels.

Table 1. Possible ways to reduce principal confusions among medial consonants uttered by a female speaker.

Table 2. Possible ways to reduce principal confusions among medial consonants uttered by a male speaker.

It seemed that nine of these principal confusions might be reduced by the inclusion of higher frequency information in such CIS processors.

Processors and subjects.

These observations led us to study the potential benefits of extending the overall frequency range represented by CIS processors at the high-frequency end. We present the results of five 22 electrode percutaneous subjects' performances studied so far. We compared processors analyzing an overall frequency range of 350Hz to 5500Hz to otherwise identical processors analyzing a range of 350Hz to 9500Hz, to examine the benefits or disadvantages of extending the frequency range higher.

With each of five subjects we compared two six channel processors which were identical except for the overall frequency range. The processors all supported good performance by their users. While they were not necessarily the best ones for each subject, they were the best pair of processors differing only in the frequency ranges. The six channel processors all had 12^th order Butterworth bandpass filters and full wave rectification. In every case the overall frequency range was divided into six logarithmically equal bands. Pulses were all presented positive phase first and in a staggered order of stimulation across electrodes. Four of the subjects' processors had 4^th order low-pass envelope smoothing filters with a 200Hz cutoff, while subject NP2 had a 4^th order smoother with a 400Hz cutoff. Electrode choice, pulse width, and stimulation rate all varied among, but not withn, subjects. The processors with the extended frequency range used a 20 bit oversampling A/D while the other processors used a 12 bit A/D. Table 3 gives more detail about each pair of processors compared in this report.

Table 3. Processor descriptions

Results

The performance of 3 subjects was compared with tests of identification of 16 medial consonants, while tests of identification of 24 medial consonants were used for the other two subjects because of their higher level of overall performance. The results within subjects are summarized in Tables 4 and 5.

Table 4. Percent correct scores for the 16 medial consonant identification tests

Table 5. Percent correct scores for the 24 medial consonant identification tests

The male test scores hardly changed with the frequency expansion; the difference between the two scores is insignificant except for subject NP1. On the other hand, the test scores for the female voice all increased by significant amounts when the upper frequency region was included. Many of the confusions in both frequency ranges are the same. For the female voice some confusions are resolved with the extended frequency range while a couple of new ones appear. We compared the most frequent consonant confusions within the two frequency ranges for the 16 consonant test and the 24 consonant test separately. The aggregate matrices from all the male voice 16 consonant tests are shown in Tables 6 and 7 for the normal and extended frequency ranges, respectively. In these and all subsequent matrices the row indicates the presented consonant token and the column the response. [Use of a boldface label distinguishes voiced th from the unvoiced form that will be included in the 24 consonant matrices.] Consonants with less than 60% correct identification score for all three subjects combined are considered "frequently misidentified". In those cases the correct identification scores are shown in red in the aggregate matrices and the percentages noted in Tables 8a and 8b, where the most frequent confusions are listed. These most frequent confusions are labeled with presented token first, then incorrect response, and also appear in bold in the aggregate matrices of Tables 6 and 7. [The same pattern – of aggregate matrices for the two overall frequency ranges, followed by lists of frequently misidentified tokens and most frequent confusions – will be followed for the other conditions of talker gender and consonant test to follow.]

Table 6. Aggregate matrix for male voice 16 medial consonant tests for subjects NP3, NP4, and NP5 with processors using an overall frequency range of 350Hz to 5500Hz.

Table 7. Aggregate matrix for male voice 16 medial consonant tests for subjects NP3, NP4, and NP5 with processors using an overall frequency range of 350Hz to 9500Hz.

Table 8a. For frequency range of 350Hz to 5500Hz, male talker, each token was presented 30 times (summation for all 3 subjects)

Tables 8b. For frequency range of 350Hz to 9500Hz, male talker, each token was presented 30 times.

The /sh/-/s/ confusion occurred less often with the extended frequency range processors for all subjects, even though for the male voice both consonants seem to be well represented in the 350Hz to 5500Hz frequency range. Other confusions where hardly affected for these subjects by the expansion of the frequency range. (There have been some instances, for other subjects not in this group, where the male consonant test scores decrease somewhat with the increase of the maximum analyzed frequency to 9500Hz.)

Table 9. Aggregate matrix for female voice 16 medial consonant tests for subjects NP3, NP4, and NP5 with processors using an overall frequency range of 350Hz-5500Hz.

Table 10. Aggregate matrix for female voice 16 medial consonant tests for subjects NP3, NP4, and NP5 with processors using an overall frequency range of 350Hz - 9500Hz.

Table 11a. For frequency range of 350Hz to 5500Hz, female talker, each token was presented 30 times.

Tables 11b . For frequency range of 350Hz to 9500Hz, female talker, each token was presented 35 times.

For the female voice 16 consonant test results the most frequent confusions that we attributed to the upper frequency bound's being too low were resolved. These included: /z/-/th/, /s/-/f/, /t/-/k/, and /sh/-/s/. The /l/-/n/ confusion remained the same while some new confusions appeared, namely /d/-/g/, /th/-/v/, and /m/-/n/. In examining these confusions before the present study, we did not expect that expanding the upper band would help subjects distinguish between /l/-/n/, /th/-/v/, and /m/-/n/. Actually it seemed to hurt, possibly because of the frequency band in each channel being wider or because of where the frequency bands for each channel now happen to fall. Thus, we would expect that some of these problems would be resolved with a higher number of channels spanning the frequency range of 350Hz - 9500Hz. In other studies with this same group of subjects, in fact, we saw a dip in test scores for 8 channel processors and then scores went back up for 11 channels. The /d/-/g/ confusion also was present in the 350Hz - 5500 Hz data but increased with the frequency extension.

Table 12. Aggregate matrix for male voice 24 medial consonant tests for subjects NP1 and NP2 with processors using an overall frequency range of 350Hz - 5500Hz.

Table 13. Aggregate matrix for male voice 24 medial consonant tests for subjects NP1 and NP2 with processors using an overall frequency range of 350Hz - 9500Hz.

Table 14a. For frequency range of 350Hz to 5500Hz, male talker, each token was presented 35 times (summation of all 2 subjects)

Table 14b. For frequency range of 350Hz to 9500Hz, male talker, each token was presented 20 times.

Percent correct scores for subjects NP1 and NP2 did not differ much with the male voice for the two frequency ranges. The most frequent confusions include many of the same tokens for both frequency ranges.

Table 15. Aggregate matrix for female voice 24 medial consonant tests for subjects NP1 and NP2 with processors using an overall frequency range of 350Hz -5500Hz.

Table 16. Aggregate matrix for female voice 24 medial consonant tests for subjects NP1 and NP2 with processors using an overall frequency range of 350Hz - 9500Hz.

Table 17a. For frequency range of 350Hz to 5500Hz, female talker, each token was presented 25 times.

Tables 17b. For frequency range of 350Hz to 9500Hz, female talker, each token was presented 30 times.

Errors for the female 24 consonant test where scattered for both frequency ranges. But the /s/ token clearly was better resolved once the high frequency information in the 350Hz-9500Hz range was included. The /z/ token, even though not fully resolved, benefited from the increased frequency range also. Once these consonants are easier to identify, other consonants that were confused with them are better resolved as well, /f/ for example.

Comparison of processor outputs

We now compare the outputs of two processors -- one using a frequency range of 350Hz to 5500Hz and the other a range of 350Hz to 9500Hz. The figures were produced by a device developed in our laboratory – called the wavegrabber – for the verification of correct processor operation and archiving of output waveforms. The processors analyzed are identical to the ones used by NP2, except for the absence of compression. In the following figures the scales are zoomed in order to show more detail for each consonant, the scale is indicated on each figure heading. Figures 1a and 1b compare the output of the female token /s/ for both frequency ranges. It is obvious that the /s/ high frequency energy present in the processor including higher frequency analysis (1b) is largely absent in the other processor (1a). The /s/ in the 350Hz-5500Hz processor would have sounded very quiet, like the token /f/ with which it was most frequently misidentified by the subjects. The token /f/ is shown in Figures 2a and 2b for comparison. There is no indication that the extension of the frequency would help identify /f/ other than, as mentioned above, other tokens with which it was frequently misidentified are well resolved by the higher frequency range processor. Figures 3a and 3b compare the outputs for the female token /z/ for both frequency ranges. Again we see that for this particular talker the 5500Hz upper limit cuts off most of the high frequency energy in the consonant.

As we can see in the test results, however, there were some disadvantages in increasing the frequency range, partly due to the widening of the frequency bands of each channel and shifting of frequency bandpass edges. A possible example of this is the confusion between female voice tokens /d/ and /g/ seen with the 16 consonant test subjects using the expanded frequency processor. Figures 4a and 4b show the token /d/ in both frequency ranges and Figures 5a and 5b show the token /g/. In both cases the consonant's burst energy gets split between two bands by the 9500Hz processor. Also notice how the shape of the envelope of channel 1 gets smoothed by the 9500Hz processor, particularly for the /d/ token. This also is evident in several other tokens. For a six channel processor spanning 350Hz-5500 Hz, the 3db points for the bandpass filters are: 350, 554, 877, 1387, 2196, 3475, and 5500 Hz. For the six channel processor spanning 350Hz-9500 Hz, the 3db points are at: 350, 607, 1052, 1823, 3161, 5480, and 9500 Hz.

Figures 6, 7, and 8 are the processors responses to tokens /m/, /n/, and /l/ respectively. Expanding the frequency range decreased the scores for these tokens, possibly due to the widening of the first bandpass filter. Even though the /l/ token has strong voicing information in channel 1 for both processors, it is frequently confused with the /n/.

Some limited testing has been done with a frequency range of 150Hz to 9500Hz. Only NP1 and NP2 were tested with processors which included these low frequencies. Their test scores were very poor for this condition. These limited data show that with the 150 Hz lower frequency limit /m/ and /n/ tokens frequently are misidentified while /l/ becomes more distinct. Limited data for these two subjects also indicate that simply removing the equalization filter (at least when using an overall frequency range of 350Hz-9500Hz), improves /l/ token identification with a little decrement in identification of other tokens. The overall test scores for this condition were equivalent to the test scores of an otherwise identical processor with the equalization filter. Further studies with different equalization filters may prove useful in improving identification of consonants characterized by small low frequency energies.

On the male tests, the frequency expansion improved the /s/-/sh/ distinction, probably by providing more redundant information. Figures 9 and 10 display the processor responses for /s/ and /sh/, respectively, for both frequency ranges.

Figure 1a & 1b Processor response to female token input /s/, 1^st exemplar, using scale x:480-840, y: ± 2.9 for both. Part a (left) is for a processor spanning the frequency range 350-5500Hz , while b is for a processor spanning the frequency range 350-9500Hz.

Figure 2a & 2b Processor response to female token input /f/, 1^st exemplar, using scale x:460-740, y: ± 2.9 for both. Part a (left) is for a 350-5500Hz processor and b for a 350-9500Hz processor.

Figure 3a & 3b Processor response to female token input /z/, 1^st exemplar, using scale x:460-760, y: ± 2.9 for both. Part a (left) is for a 350-5500Hz processor and b for a 350-9500Hz processor.

Figure 4a & 4b Processor response to female token input /d/, 1^st exemplar, using scale x:480-780, y: ± 2.9 for both. Part a (left) is for a 350-5500Hz processor and b for a 350-9500Hz processor.

Figure 5a & 5b Processor response to female token input /g/, 1^st exemplar, using scale x:480-780, y: ± 2.9 for both. Part a (left) is for a 350-5500Hz processor and b for a 350-9500Hz processor. Notice how the energy from channel 4 in the narrow range processor gets distributed between channels 3 and 4 in the expanded frequency processor. Also channel 1 here looks more distinct than channel 1 in the expanded version.

Figure 6a & 6b Processor response to female token input /m/, 1^st exemplar, using scale x:460-740, y: ± 2.9 for both. Part a (left) is for a 350-5500Hz processor and b for a 350-9500Hz processor.

Figure 7a & 7b Processor response to female token input /n/, 1^st exemplar, using scale x:460-740, y: ± 2.9 for both. Part a (left) is for a 350-5500Hz processor and b for a 350-9500Hz processor.

Figure 8a & 8b Processor response to female token input /l/, 1^st exemplar, using scale x:460-740, y: ± 2.9 for both. Part a (left) is for a 350-5500Hz processor and b for a 350-9500Hz processor.

Figure 9a & 9b Processor response to male token input /s/, 1^st exemplar, using scale x:376-716, y: ± 4.9 for both. Part a (left) is for a 350-5500Hz processor and b for a 350-9500Hz processor.

Figure 10a & 10b Processor response to male token input /sh/, 1^st exemplar, using scale x:376-716, y: ± 4.9 for both. Part a (left) is for a 350-5500Hz processor and b for a 350-9500Hz processor.

Conclusions

In general, upward extension of the overall frequency range to 9500 Hz does not degrade overall processor performance for these subjects, while producing substantial improvements in medial consonant identification for the female voice.

9500 Hz probably is a higher upper frequency limit than necessary to achieve these results. Examination of spectrographs of the female voice tokens in the Iowa videodisc recordings indicates that 8000 Hz would be sufficient. Other voices, including those of children, should be included in future studies, as should various alternatives to the present preemphasis equalization filter.

III. Plans for the Next Quarter

Our plans for the next quarter include the following:

• Studies with Ineraid subjects SR15 (week of February 9), SR9 (two weeks beginning on February 23), and SR16 (week of March 9), principally to evaluate speech reception performance with their portable Innsbruck/RTI processors, following six months or more of daily use for each subject. The studies also will include measures of channel interaction using recordings of intracochlear evoked potentials, as in prior studies with other subjects. Intracochlear EPs also will be recorded for stimuli used in prior psychophysical studies of pitch scaling for subjects SR15 and SR16 (such recordings already have been made for subject SR9). Additional studies for all three subjects will include (1) measures of stimulus levels corresponding to threshold and most comfortable loudness in the context of multichannel stimulation; (2) comparison of speech processors using mapping functions derived from those measures versus speech processors using mapping functions derived from standard single-channel measures of threshold and MCL; (3) evaluation of proportional reductions in target thresholds across channels in deriving mapping functions; (4) evaluation of "n-of-m" processors; and (5) evaluation of additional strategies designed especially for implant patients at the low end of the performance spectrum.
• Ongoing studies with Ineraid subject SR2 (usually one morning per week). Studies anticipated for the next quarter include investigation of several new processor designs, such as processors using "conditioner" stimuli in addition to the normal outputs of CIS processors and such as processor using high rate stimuli in various ways.
• Resumption of studies with one or more subjects implanted with the Clarion device. The studies will include comparisons among (1) the Clarion implementation of a CIS-like strategy in conjunction with monopolar electrodes, (2) the CIS-like strategy in conjunction with "enhanced bipolar" electrodes, (3) the "simultaneous analog strategy" (SAS) of the Clarion device in conjunction monopolar electrodes, and (4) the SAS in conjunction with enhanced bipolar electrodes. All processors will use the same number of channels. The processors will be implemented and applied using our laboratory speech processor system and fitting procedures.
• Initial studies at RTI with the first of ten subjects implanted at the University of Iowa with CI24M devices on both sides (provisionally scheduled for either the week beginning on March 30 or the week beginning on April 13). The initial studies will focus on psychophysical measures of electrode ranking within and between implants, and of sensitivities to differences in the timing and amplitude of stimuli delivered to the two sides.
• Continued studies with subject NU-4, a local patient with standard transcutaneous Nucleus implants on both sides.
• Preparation of an invited poster, to be presented at the NIH Bioengineering Symposium, Bethesda, MD, February 27 and 28, 1998.
• Continued analysis of speech reception, evoked potential, and psychophysical data from prior studies.
• Continued preparation of manuscripts for publication, especially invited chapters for two books on cochlear implants, one edited by John Niparko and the other edited by Noel Cohen and Susan Waltzman.

IV. Acknowledgments

We thank subjects ME2 and SR2 for their participation in the studies of this quarter. We also are most grateful for the many and important contributions made by co-investigators Stefan Brill and Joachim Müller, and by consultant Sigfrid Soli, to the studies with bilateral subject ME2.

Appendix 1. Summary of Reporting Activity for this Quarter

Reporting activity for the last quarter, covering the period of November 1, 1997 through January 31, 1998, included the following:

Publication

Wilson BS, Finley CC, Lawson DT, Zerbi M: Temporal representations with cochlear implants. Am J Otol 18: S30-34, 1997.

Invited Presentation

Wilson BS: Review of studies at RTI with recipients of bilateral cochlear implants. University of Iowa, Department of Otolaryngology, Head & Neck Surgery, Iowa City, IA, January 27, 1998.