Sage Journals: Discover world-class research

Abstract

Future hearing devices could exploit brain signals of the user derived from electroencephalography (EEG) measurements, for example, for fitting the device or steering signal enhancement algorithms. While previous studies have shown that meaningful brain signals can be obtained from ear-centered EEG electrodes, we here present a feasibility study where ear-EEG is integrated with a live hearing device. Seventeen normal-hearing participants were equipped with an individualized in-the-ear hearing device and an ear-EEG system that included 10 electrodes placed around the ear (cEEGrid) and 3 electrodes spread out in the concha. They performed an auditory discrimination experiment, where they had to detect an audible switch in the signal processing settings of the hearing device between repeated presentations of otherwise identical stimuli. We studied two aspects of the ear-EEG data: First, whether the switches in the hearing device settings can be identified in the brain signals, specifically event-related potentials. Second, we evaluated the signal quality for the individual electrode positions. The EEG analysis revealed significant differences between trials with and without a switch in the device settings in the N100 and P300 range of the event-related potential. The comparison of electrode positions showed that the signal quality is better for around-the-ear electrodes than for in-concha electrodes. These results confirm that meaningful brain signals related to the settings of a hearing device can be acquired from ear-EEG during real-time audio processing, particularly if electrodes around the ear are available.

Keywords

Ear-EEG cEEGrid hearing aid auditory evoked potentials

Introduction

Although hearing devices can help to overcome even severe hearing problems, they are often not optimally adapted to the user. To improve the listening outcome in all situations and for optimal individualization, future hearing devices could exploit brain signals of the user derived from electroencephalography (EEG) measurements (Popelka & Moore, 2016). Specific applications could be automatic objective fitting of the device based on neural responses (Finke, Billinger, & Büchner, 2017; Lunner & Neher, 2013) or real-time steering of signal enhancement algorithms based on the decoded direction of attention (Bleichner, Mirkovic, & Debener, 2016; O’Sullivan et al., 2017, 2015). The integration of EEG into hearing devices needs to be convenient for the user, so that it can be used without stigmatization in everyday situations. In the recent years, several approaches have been presented that allow to record EEG reliably in and around the ears, referred to as ear-EEG (Bleichner & Debener, 2017; Debener, Emkes, De Vos, & Bleichner, 2015; Goverdovsky, Looney, Kidmose, & Mandic, 2016; Looney et al., 2012). It has been shown repeatedly that ear-EEG can record a wide variety of cognitive processes related to auditory perception and auditory attention (Debener et al., 2015; Looney et al., 2012; Mikkelsen, Kappel, Mandic, & Kidmose, 2015; Mirkovic, Bleichner, De Vos, & Debener, 2016). However, to the authors’ best knowledge, ear-EEG has never been evaluated in a situation where it was integrated with a live ear-level hearing device.

We therefore present a first feasibility evaluation of integrating ear-EEG with a live electroacoustic hearing device. We combined a recently presented experimental high-fidelity in-the-ear hearing device (Denk, Hiipakka, Kollmeier, & Ernst, 2017) with an around-the-ear electrode array consisting of 10 electrodes arranged in a C-shape (cEEGrid; Bleichner & Debener, 2017; Debener et al., 2015; www.ceegrid.com) and 3 electrodes distributed in the concha (in-concha electrodes). To evaluate auditory perception through the hearing device, we examined whether perceivable switches in the electroacoustic transmission properties of the hearing device between repeated presentations of otherwise identical stimuli can be identified in the ear-EEG.

For identification of the switches in the EEG data, we specifically focused on the amplitudes in the latency ranges of the N100 and P300 event-related potential (ERP). The N100 is an ERP component that can be detected in response to an auditory stimulus and has a maximal amplitude between 50 to 150 ms after sound onset (Näätänen & Picton, 1987). The N100 component shows a clear amplitude reduction for repeated sounds in a sequence, but amplitude increases again when a deviant sound is presented (Barry, Cocker, Anderson, Gordon, & Rennie, 1992). The property of an amplitude reduction for repeated identical sounds and the amplitude recovery when a nonidentical (deviant) sound is presented provides an objective means to study whether sequential sounds are perceived as different or identical. The P300 with a maximal amplitude at around 300 to 500 ms after stimulus onset is a second ERP component that is elicited by task-relevant, deviant sounds and reflects the more conscious evaluation and categorization of the stimulus (Polich, 2007). For the N100, we expected that the repetition of identical sounds leads to an amplitude reduction, while a deviant sound leads to an amplitude increase. For the P300, we expected a higher amplitude for a repeated deviant sound compared with a repeated identical sound. Importantly for the integration of EEG in hearing devices, these ERPs are comparatively robust responses that can be expected to be also recorded reliably under everyday situations outside of a lab context. ERPs can be exploited, for example, for automated fitting of hearing devices (Finke et al., 2017; Lunner & Neher, 2013).

The position of ear-EEG electrodes is a crucial factor for future integration into one combined ear-level hearing device. There appears to be a trade-off between the compact positing of the electrodes to assure ease of use and optimal sensitivity to the brain signal of interest. Therefore, we compared the signal properties of the around-the-ear and in-concha electrodes. Due to the larger interelectrode distance of the around-the-ear electrodes compared with the in-concha electrodes and a finer angular coverage of bipolar channel orientations (Bleichner & Debener, 2017), we expected a larger signal amplitude, better channel independence, and a better signal-to-noise ratio (SNR) for around-the-ear than in-concha recordings (Bleichner et al., 2015).

Methods

Participants

Seventeen subjects (age 28.4 ± 5.4, 10 male, 7 female) with clinically normal hearing participated in the study. Normal-hearing participants were used here as we were interested in the general feasibility of our approach. The study was conducted in agreement with the declaration of Helsinki and was approved by the local ethical committee of the University of Oldenburg (Drs. 5/2015). Before active participation, each participant gave written informed consent.

Acoustic Setup

The participants were equipped with a prototype hearing device as presented by Denk et al. (2017). It consists of an individual soft silicone earmold with an integrated set of electroacoustic transducers as shown in Figure 1. External sound is captured with the inbuilt pickup microphone located in the concha, processed, and played back on the inbuilt loudspeaker. Real-time processing was performed on a laptop running the Master Hearing Aid platform (Grimm et al., 2006), which was connected to the transducers through an Multiface II soundcard (RME, Haimhausen, Germany) with an input–output delay of 7.8 ms.

Figure 1.

Left: Photograph of the setup in the ear of a subject, with the cEEGrid glued around the ear using double-side adhesive tape. In the concha, the earmold containing the transducers of the hearing device (Denk et al., 2017) and three additional electrodes (black wires) are placed. Center: Schematic view of the layout in the ear. Gray circles indicate electrodes with their according nomenclature; red symbols mark the positions of electroacoustic transducers. The shaded area marks the part of the earmold which is inserted into the ear canal. Electrode CLF has been placed at one of the indicated alternative locations in the individual ears, depending on which was feasible. Right: Subject wearing superaural headphones (AKG K-1000) which provide sufficient free space for the hearing device and electrodes.

The hearing device was automatically calibrated in situ for each user to account for individual variations of the external ear (see Denk et al., 2017, for details). The aim was to provide acoustic transparency, that is, the pressure at the eardrum with the device inserted approximates the pressure at the eardrum that can be observed with an open ear (i.e., an unoccluded ear canal). The processing chain (here a finite impulse response filter) was individually adapted in a way that the superposition of electroacoustically generated sound and a direct sound component leaking through the vented earpiece approximated the open-ear condition. Thereby, an additional microphone located at the inner surface of the device pointing toward the eardrum was utilized to estimate the pressure at the eardrum.

Stimuli were presented on super-aural headphones (K-1000, AKG, Vienna, Austria), which are also shown in Figure 1. The special design assured that neither the electrodes nor the hearing device was touched by the headphones. This setup represents a sound-source coupling to the ear similar to the free field (Møller, Hammershøi, Jensen, & Sørensen, 1995). The stimuli were presented monaurally on the right ear only, which was equipped with a hearing device, whereas the left ear was fully occluded.

EEG Setup

EEG was acquired with a cEEGrid including 10 around-the-ear electrodes arranged in a C-shape (Bleichner & Debener, 2017; Debener et al., 2015) and 3 electrodes positioned at different positions in the concha (in-concha electrodes) by insertion into bores in the earmold (see Figure 1). Only electrodes positioned at the right ear were considered, that is, the side where the hearing device was located, to emulate a fully integrated system. The in-concha electrodes were Ag/AgCl miniaturized (2 × 4 mm) ton electrodes (i.e., shaped cylindrical, EasyCap GmbH, Herrsching, Germany) as used in an earlier study (Bleichner et al., 2015). After skin preparation with an abrasive gel and alcohol, a small amount of electrolyte gel (Abralyt HiCl, Easycap GmbH, Germany) was applied to the electrodes. The cEEGrids were fixed around the ear with a double-sided adhesive tape and the ton electrodes were inserted into the earmold up to skin contact after a drop of electrolyte gel was administered into the bores. All electrodes were connected to a wireless mobile 24-channel DC EEG amplifier (SMARTING, mBrainTrain, Belgrade, Serbia; custom modification for cEEGrid acquisition) positioned at the back of the head. EEG data were recorded with 24 bit resolution and 500 Hz sampling rate; electrodes R4a and R4b of the cEEGrid (see Figure 1) served as ground and reference, respectively. Signals were wirelessly transmitted to a recording computer through a Bluetooth connection. Although the used system is a laboratory-state prototype, there is no principal reason that electrodes in this layout cannot be included in a real hearing device or a fully mobile prototype.

A schematic drawing of the recording setup is shown in Figure 2. Acoustic stimulation and experimental control was implemented in MATLAB (Mathworks, Inc, Natick, MA) on the same laptop that was also used to send EEG triggers synchronously to audio stimulation via Lab Streaming Layer, a software framework for data acquisition (Swartz Center for Computational Neuroscience & Kothe, 2015). The subjects including the laptop were seated in a booth that was acoustically but not electromagnetically shielded. The EEG data as well as the event marker sent by the audio processing laptop were recorded using the recording computer located outside the booth using the LabRecorder Software from Lab Streaming Layer.

Figure 2.

Schematic depiction of the general experimental setup. The subjects were seated in a booth (indicated by thick gray line), where they conducted the experiments autonomously on a laptop that controlled the stimuli presentation, real-time audio processing for the hearing device, and EEG triggering through LSL.

Paradigm

Participants listened to sequences of four sounds. In each trial, four identical sounds were presented via the headphones. In half of the trials, the filter setting of the hearing device was switched between the third and the fourth sound, resulting in a perceivable deviation. Participants indicated whether they perceived an acoustic difference between the third and fourth sound by pressing buttons (y/n) on a keyboard within 1 s after the end of the fourth sound. Two different filter settings of the hearing device were utilized for each subject; details on the filter design are described in Hearing Device Settings.

Since the EEG evaluation assumes the subjects’ attention, data of subjects with a poor task performance were discarded. Only data from subjects whose responses fulfilled the following criteria were included in the analysis: (a) response correct (identical trials indicated as equal and deviant trials indicated as different) in more than 80% of all identical and deviant trials and (b) response given in more than 90% of all trials.

Stimuli

Three different stimuli were used: White noise (referred to as Noise), a logatome spoken by a female voice (Sass, from the OLLO corpus referred to as Speech; Meyer, Jürgens, Wesker, Brand, & Kollmeier, 2010), and the combination of both with an SNR of 5 dB (referred to as Speech-in-Noise). White noise was selected since it best supports a detection of the generated deviations, whereas Speech and Speech-in-Noise represent more complex and realistic stimuli. For all stimuli, a bandpass filter between 0.1 and 12 kHz was applied to match the frequency range of the hearing device. The identical waveforms were played in all four repetitions of one trial to avoid random variations of the spectra of the noise stimuli.

Each sound of the Noise condition had a duration of 500 ms. For the Speech-in-Noise condition, the logatome was placed in the middle of the 500 ms noise segment (see Figure 3). For the Speech condition, the logatome was placed at the same point in time as in the Speech-in-Noise condition to keep the speech onset time the same for these two conditions. For the Speech condition, this resulted in a longer period of silence before and after the logatome compared with the Speech-in-Noise condition. One trial consisted of four sounds presented sequentially with an interval of 300 ms between the 500 ms stimulus windows (beginning marked by S1 to S4 in Figure 3).

Figure 3.

Temporal recording of acoustic stimuli, obtained in a measurement on a dummy head. The example shows a trial for the Speech-in-Noise stimulus, including a switch from the hearing device output Filter F1 in the first three sounds (S1 to S3) to F2 in the last sound (S4). The blue line indicates the pressure at the artificial eardrum, which consists out of the direct sound leaking through from headphone playback and the output of the hearing device, indicated by the magenta curve (amplitudes not to scale). The different spectral profiles of F1 and F2 change the apparent power of the hearing device output; however, due to interference with the direct sound, the pressure at the eardrum has a similar level with both filter settings. Stimuli are 500 ms long with 10 ms ramps at the beginning and end, and separated by pauses of 300 ms, the range where the Speech stimulus is active is marked separately. O1 to O4 mark the onsets of the hardware noise before the first stimulus and after briefly turning it off between stimuli; S1 to S4 mark the onsets of the stimuli.

The audio stimuli are shown in Figure 3, which shows an example of a Speech-in-Noise stimulus measured at the artificial eardrum of a dummy head. The dummy head consisted of a custom adjustable ear canal simulator (Hiipakka, Tikander, & Karjalainen, 2010) that was attached to KB1065/1066 pinnae (G.R.A.S., Holte, Denmark) and mounted in a modified show-window mannequin. The dummy head was equipped with the same individualized earpieces as the subjects.

Each 16 deviant trials including a filter-switch in both possible orders of the two utilized filters and 16 nondeviant trials in either filter setting were presented. This was repeated for each stimulus (Noise, Speech, and Speech-In-Noise). Thus, in total, 192 trials were presented in randomized order, subdivided in four blocks of the same length. The experiment included additional conditions with a comparable number of trials, the results of which are not considered in this work. One session lasted about 90 min, which included individual adjustments of the hearing device filters prior to presentation of the stimuli (see next section). Prior to each block, the subjects had the opportunity to take a short break while remaining seated.

Hearing Device Settings

The deviation between repeated sounds was created by two different adjustments of the hearing device output filter. In one adjustment, the output filter of the hearing device was adjusted by individual calibration of the hearing device prior to the main experiment (Filter F1, see Acoustic Setup section). In the other adjustment, the output filter resulting from equivalent calibration of the system on a dummy head was used (F2, same dummy head as for the recordings of Figure 3). Due to different ear geometries and the tightness of fit, the filter responses varied. Hence, the spectral profile of sound arriving at the eardrum and, in the perceptual domain, the timbre was notably different between the two conditions. Alternative cues that may arise from differences in loudness were compensated through an additional broadband gain applied to the Filter F2, which was individually adjusted by means of an adaptive 1-up-1-down procedure conducted with the Noise stimulus prior to the main experiment.

A life hearing device was used, which made it almost inevitable to have a perceptible noise floor that originates mainly from the pickup microphone and fills the pauses. If the last sound is different, the output filter of the device is switched between playback of stimuli, whereas the operational setting is kept constant in nondeviant trials. To avoid sudden transient modification in the hardware noise coloration in deviant cases, the hearing device output was briefly deactivated 130 ms after presentation of Sounds 1 to 3 (20 ms pause, with 10 ms ramps, referred to as O2 to O4, see also Figure 3). The length of the deactivation was adjusted such that the pause was as brief as possible, while it still perceptually separated the noise floor before and after in different sound events. The hearing device was activated 300 ms prior to presentation of the first sound of each trial (referred to as O1) and turned off 130 ms after the last sound.

EEG Analysis

The analysis was performed offline with EEGLAB (Delorme & Makeig, 2004) and MATLAB. For the statistical analysis, we used RStudio (RStudio, Inc., Boston, MA), for the planned comparisons regarding the stimulus onsets, p-values below .05 were considered as indication of statistical significance. For the exploratory analysis regarding the onset of the hardware noise, we corrected for multiple comparisons (i.e., two comparisons). In these cases, p-values below .025 were regarded as statistically significant.

The data from each block were filtered prior to segmentation between 0.1 Hz and 20 Hz with a consecutive high-pass filter (filter order 500) and low-pass filter (filter order 100, windowed sinc FIR filters with linear phase). Epochs were extracted for the entire trial (−1000 ms to 4000 ms) relative to the onset of the first sound (S1). Furthermore, epochs were extracted for the individual sounds (−500 ms to 1000 ms) relative to the onset of the respective Sounds S1 to S4. After first inspection of the results (see also Discussion section), we also extracted epochs of the same length locked to the onsets of the hearing device O1 to O4, that is, the moments when the hardware noise started. Epochs dominated by artifacts were identified using the probability criteria implemented in EEGLAB (standard deviation: 2) and rejected from further analysis. On average, 27.1% (SD = 6.7%) of the trials were rejected. Averaging across the remaining epochs resulted in ERP waveforms.

The main contrast of interest was between identical and deviant sounds (S4). We expected a difference in amplitude of the N100, with a larger amplitude for the deviant compared with the identical condition. In addition, we expected a larger P300 for the deviant sound compared with the identical condition. Furthermore, we expected a decrease of the N100 amplitude from S1 to S2. The amplitude of the N100 was quantified by computing the mean amplitude for a 40 ms window around the N100 peak latency (±20 ms). The N100 latency was calculated as the peak latency of the N100 amplitude in response to S1. The same temporal window with respect to the stimulus onset was used to compute the amplitude in S2 to S4. The amplitude of the P300 was quantified by computing the mean amplitude for the time window of 230 ms to 430, relative to S4 and O4, respectively. Based on prior experience (Bleichner & Debener, 2017), we computed the signal difference between the mean of two electrodes above and the mean of two electrodes below the ear ((R2 + R3)/2) – ((R6 + R7)/2); see Figure 1 for the position of the electrodes. The resulting signal is in the following referred to as vertical bipolar cEEGrid channel and utilized to evaluate the ERPs in response to the switch in hearing device processing. All analyses were performed with the electrodes on the right side, that is, on the ear where the stimuli were presented (cf. EEG setup).

Influence of Electrode Position

A second objective of this study was to compare the signal at different electrode positions in and around the ear. To evaluate this issue, we computed measures of effect size, signal to noise ratio, and between-channel similarities for three different electrode configurations.

Electrode configurations

In addition to the vertical bipolar cEEGrid channel, the signal quality was evaluated for three electrode configurations: (a) all individual electrodes of the cEEGrid referenced to the original reference (Grid), (b) all in-concha electrodes locally rereferenced to the other in-concha electrodes (Concha), and (c) all cEEGrid electrodes rereferenced to the mean of the in-concha electrodes (Grid/Concha).

Effect size

The grand average ERPs (mean over all participants and trials) were computed for identical and deviant sounds relative to O4 for the Speech stimulus. The effect size, that is, the difference between identical and deviant presentation, was measured as Hedges’ g (Hedges, 1981) for the first negative peak (±20 ms) and for the P300 window (230 ms to 430 ms). Hedges' g is a variation of Cohen's d (Cohen, 1988) but reduces the estimation error for smaller samples by correcting the pooled variance. The interpretation of Hedges' g is analogous to Cohen's d, and effect size above 0.8 (absolute value) is considered as large.

SNR comparison

To assess the quality of the ERPs for the N100 response to the first sound (S1), an SNR was calculated for the Speech stimulus in each participant separately. The SNR was computed for the N100 component by dividing the root mean square of the N100 component at peak latency ± 20 ms by the root mean square of the estimated noise in the same time window scaled to dB (20log/10 (signal/noise)). The noise was estimated using a plus–minus procedure (Schimmel, 1967): The time signals of all trials are averaged after the polarity of every other trial is reversed. Assuming a signal that is coherent across repetitions, the resulting average is an estimate of the noise. For each electrode configuration, the individual channel with the maximal SNR was selected for each participant.

Interelectrode correlation

Furthermore, we assessed the influence of the electrode configuration on the between channel similarity. We therefore calculated the between-channel correlation coefficient between the individual channels of the Grid and Concha configurations, and the in-concha electrodes referenced to the recording reference (R4b). As we were interested in the overall signal similarity independent of the task and a specific neural source, all available data of the recording session were used and segmented into epochs of 1 s each. To prevent that the correlation is primarily driven by nonneural artifacts (e.g., eye blinks), epochs that contained artifacts were discarded (EEGLAB joint probability criterion, 2 standard deviations). The correlation coefficient was computed between all channel pairs for each epoch, and then averaged over epochs and participants.

Results

Psychophysical Results

Figure 4 shows the subjective discrimination results as indicated by the individual participants. Generally, identical and deviant trials were recognized with high accuracy. On average, identical sequences were indicated as such in 90.0% of all trials, and sequences where the last sound was deviant were indicated as different with 93.3% accuracy. In 3.6% of all trials, no response was given in the response window. These observations verified the clear audibility of the hearing device processing switch and the practicality of the paradigm. The task performance of three subjects (E6, E8, and E13) did not fulfill the criteria given in the Paradigm section (also marked in Figure 4). Consequently, their data were excluded from further analysis due to a suspected low level of attention.

Figure 4.

Subjective discrimination results, pooled over stimuli. For Identical and Deviant trials, the individual subjects’ responses are grouped to equal, different, and none. Data from three subjects (E6, E8, and E13) were excluded from further analysis because at least one of the criteria was not fulfilled: Response correct (i.e., equal for identical and different for deviant trials) >80%, no response (average over both conditions) <10%. The criteria are marked by dashed horizontal lines and colored vertical arrows.

EEG Results

Piloting results

Extensive piloting tests had been conducted where the same ear-EEG setup and the same or a similar paradigm, but varying acoustic stimulation were utilized. In particular, sound presentation was also done on distant loudspeakers only, while the hearing device was in place but disconnected from the sound card. Electromagnetic cross-talk from the hearing device or headphone to the EEG electrodes could thus be ruled out by comparing the EEG recordings from loudspeaker stimulation to the hearing device or headphone stimulation. Generally, the piloting results verified that the stimulus-locked signal components observed in the ear-EEG electrodes do originate from brain activity.

N100 amplitude reduction

The N100 amplitude reduction was assessed for the vertical bipolar cEEGrid channel (see EEG Analysis section). Figure 5(a, top) shows the grand average ERPs, averaged over all trials for the Noise and Speech-in-Noise stimuli (Mark: Noise and Speech-in-Noise have the same onset and offset time of the noise). Figure 5(a, bottom) shows the grand average ERP for the Speech stimulus. A negative deflection around 154 ms after stimulus onset (for S1) is clearly visible. A strong amplitude reduction of the N100 is evident for S2 and S3 relative to S1. Regarding the expected N100 amplitude reduction between S1 and S2, we computed a repeated measures analysis of variance (ANOVA) with the factors Stimulus Type (Speech, Speech-in-Noise, and Noise) and Sound (S1 and S2). The results are shown in Figure 6(a) and are also apparent in Figure 5(b) for the Speech condition. We found a significant main effect of Sound on the N100 amplitude, F(1, 13) = 13.22, p = .003, with a larger N100 amplitude for S1 compared with S2, and a significant effect of Stimulus Type on the N100 amplitude, F(2, 26) = 4.31, p = .024, and no significant interaction, F(2, 26) = .87, p = .431.

Figure 5.

(a) Top: Grand average ERP for Noise (solid line) and Speech-in-Noise (dashed line) condition, combined for identical and deviant trials. Bottom: Grand average ERP for Speech (solid line). Clearly visible is the N100 response roughly 150 ms after sound onset (S1), as well as after the onset of the hardware noise (O1). (b) Grand average ERP for S1 (black), S2 (dark gray), and S3 (light gray) for the Speech stimulus. Indicated in gray is the time window that was used for estimating the N100 amplitude. (c) Grand average ERP for identical (gray line) and deviant filters (black line) for the Speech stimulus (O4). Indicated as a gray background is the time window that was used for estimating the N100 amplitude (first) and the P300 amplitude (second).

Figure 6.

(a) N100 Amplitude in respect to stimulus onset for S1 and S2 for the individual stimuli. The N100 amplitude is reduced for S2 relative to S1 for all stimuli. (b) N100 amplitude in respect to device onset (hardware noise onset) for O1 and O2 for the individual stimuli. (c) N100 amplitude for identical and deviant condition in respect to device onset (O4) for the individual stimuli. Boxes and whiskers denote the 25% to 75% quantiles and full data range, respectively; the horizontal line indicates the median, circles denote outliers.

Also apparent for all stimuli (Figure 5(a)) is a negative deflection prior to stimulus onset with a latency that fits the onset of the hardware noise when the hearing device is switched on (O1). The N100 amplitude was decreased for the subsequent onsets (O1 to O2). We computed a repeated measures ANOVA on the N100 in response to the device onset with the factors Stimulus Type (Speech, Speech-in-Noise, and Noise) and Sound (O1, O2). The results are shown in Figure 6(b) and are also apparent in Figure 5(b) for the Speech condition. The N100 amplitude is larger for O1 compared with O2, analog to what we found for S1 and S2. However, the main effect of Sound on the N100 amplitude does not reach significance for O1 compared with O2, F(1, 13) = 4.39, p = .056. There was no effect of Stimulus Type on the N100 amplitude, F(2, 26) = 1.32, p = .284, and no significant interaction, F(2, 26) = 0.02, p = .982.

Filter switch

The identification of the filter switch is first assessed for signals from the vertical bipolar cEEGrid channel. Regarding the expected N100 amplitude reduction of the S4-identical relative S4-deviant condition, we computed a repeated measures ANOVA with the factors Stimulus Type (Speech, Speech-in-Noise, and Noise) and Filter-Switch (S4-identical and S4-deviant). We find a significant main effect of Filter-Switch on the N100 amplitude, F(1, 13) = 6.16, p = .027, with a larger N100 amplitude for S4-identical as compared with S4-deviant. This is contrary to the expectation of S4-deviant having a larger amplitude compared with S4-identical. There was no main effect of Stimulus Type on the N100 amplitude, F(2, 26) = 2.32, p = .118, and no significant interaction, F(2, 26) = 1.69, p = .204.

We also examined the N100 amplitude in response to the device onset (O4) and computed a repeated measures ANOVA with the factors Stimulus Type (Speech, Speech-in-Noise, and Noise) and Filter-Switch (O4-identical, O4-deviant). The results are shown in Figure 6(c) and are apparent for the Speech stimulus in Figure 5(c). We found a significant main effect of Filter-Switch on the N100 amplitude, F(1, 13) = 15.36, p = .002, with a larger N100 amplitude for O4-deviant compared with O4-identical. There was no significant effect of stimulus type on the N100 amplitude, F(2, 26) = 0.32, p = .738, and no significant interaction, F(2, 26) = 1.21, p = .314. The independence on the stimulus here was expected, as the hardware noise is independent of the stimulus type.

Finally, we analyzed P300 amplitudes in response to the device onset and computed a repeated measures ANOVA with the factors Stimulus Type (Speech, Speech-in-Noise, and Noise) and Filter-Switch (O4-identical and O4-deviant). The results are shown in Figure 7 and are apparent in Figure 5(c) for the Speech stimulus. We found the expected significant main effect of Filter-Switch on the P300 amplitude, F(1, 13) = 12.25, p = .004, with a larger P300 amplitude for O4-deviant compared with O4-identical. There was a significant effect of Stimulus Type on the P300 amplitude, F(2, 26) = 6.21, p = .006, and no significant interaction, F(2, 26) = 0.01, p = .984.

Figure 7.

P300 in respect to device onset (hardware noise onset) at O4 for the individual stimuli, comparison of identical and deviant trials. Over all stimuli, we observe a higher amplitude in the P300 window for the deviant sounds compared with the identical sounds. Boxes and whiskers denote the 25% to 75% quantiles and full data range, respectively; the horizontal line indicates the median, circles denote outliers.

Electrode comparison

The results of the effect size (i.e., difference in ERP between identical and deviant trials on O4) comparison in dependence on the electrode layout are shown in Figure 8(a). The largest effect sizes for the N100 and P300 time window were obtained for the Grid configuration (on average of the individual channels). The Grid electrodes behind and above the ear showed a medium effect size (but below the threshold for large effects of 0.8), while the effect size for the below-the-ear channels was relatively small. Some individual Grid channels located behind the ear exceeded the effect size that was observed with the vertical bipolar cEEGrid channel utilized in the previous section. The in-concha electrodes referenced locally to other in-concha and the Grid electrodes referenced to Concha electrodes also showed small effect sizes.

Figure 8.

(a) Mean effect size (Hedges’ g, absolute values) for the N100 (left) and P300 (right) between conditions (O4, deviant/identical). The color lines from the insets in the center indicate the electrodes and the respective reference. The cross or black line in the Grid condition marks the vertical bipolar cEEGrid channel that has been utilized for the Filter-Switch discrimination in the previous section. The in-concha channels (Concha) were locally rereferenced to the other in-concha channels. The cEEGrid channels (indicated as colored dots) were either rereferenced to the R4b (white electrode with black dot, Grid condition) or to the mean of the in-concha channels (Grid/Concha condition). (b) Maximal SNR (maximum of channels in each condition) for the N100 in respect to the stimulus onset (S1) of the Speech stimulus, individual subjects grouped. The in-concha electrodes were locally rereferenced to other in-concha electrodes (Concha), for cEEGrid channels, references to the below the ear channels or the original reference (Grid), for the Grid/Concha condition the cEEGrid channels were rereferenced to the mean of the in-concha channels.

The SNR analysis of electrodes at different positions and referencing schemes is shown in Figure 8(b). The largest SNR was observed in the cEEGrid electrodes (Grid configuration, M = 16.18, SD = 6.16), which was significantly higher than the maximum SNR for in-concha electrodes referenced locally to in-concha (Concha configuration, M = 7.55, SD = 6.70); paired t-test, t(13) = 4.10, p = .001. The maximal SNR for channels in the Grid configuration was not significantly different from the maximal SNR for cEEGrid electrodes referenced against the in-concha electrodes (Grid/Concha configuration), t(13) = −0.98, p = .341.

The analysis of the interelectrode correlation patterns is shown in Figure 9(a). For the cEEGrid channels, the median correlation was r = .40 (min = −.16, max = .75). The smaller the angle between the respective electrodes (relative to the reference electrode), the higher their signals were correlated (r = −.94, p < .001; Figure 9(b)). The in-concha channels (referenced to R4b of the cEEGrid) were highly correlated to each other, with a median interchannel correlation of r = .99 (min = .99, max = .99). The correlation between cEEGrid channels and in-concha EEG channels was low, with a median interchannel correlation of r = .12 (min = −.06, max = .23). The median interchannel correlation for the three possible in-concha bipolar channels (Concha configuration, i.e., each in-concha channel referenced to the two other channels) was r = .42, but showed a wide range (min = −.14, max = .99).

Figure 9.

(a) Between-channel correlation of all around-the-ear and in-concha channels (referenced to electrode R4b). The four below-the-ear (R5 to R8) and above-the-ear (R1 to R4) electrodes are more highly correlated to each other than to the electrodes of the other group. The in-concha electrodes (CLB, CLF, and CU) are highly correlated to each other and largely uncorrelated to the electrodes around the ear. (b) Inter-electrode angle difference (in respect to reference electrode) relative to the inter-electrode correlation. The larger the inter-electrode angle, the lower the inter-electrode correlation (r = −.942). The insets exemplify the relation of inter-electrode correlation and inter-electrode angle.

Discussion

ERPs and Identification of the Filter Switch

We observed robust ERPs in response to the onset of the stimuli, including N100- and P300-like structures that could be exploited to identify a switch in the filter of the hearing device. For the N100, we found the expected amplitude reduction in response to repeated sounds. For a deviant last sound, the expected increase in the N100 amplitude compared with an identical last sound was not present in our data. However, this result can be fully explained by considering the hardware noise of the hearing device. It is important to note that the manipulation of the filter settings in our experiment lead to a change in the spectral properties, both for the actual stimuli and the hardware noise. That is, a filter-switch between S3 and S4 became already apparent at the moment the device was switched back on (O4). To avoid sudden transient of the audible noise floor on a filter-switch, we had decided to briefly switch the hearing device on and off between presentation of stimuli while the filter was switched (or not). The onset of the hardware noise was very salient, particularly the first on-switch prior to presentation of the first sound which lead to a N100 (Figure 5, 100 ms after O1) that was comparable in size to the N100 to stimulus onset (Figure 5, 100 ms after S1). The N100 also showed a reduction in amplitude from the first (O1) to the second hardware noise onset (O2). Most importantly, we did find a difference in the N100 amplitude between identical and deviant trials with respect to the moment when the device was switched back on between O3 and O4. The task for the participants was to detect a difference in the last sound of the sequence and to respond with a button press. The expected difference was not observed in the latency range with respect to the onset of the last sound (S4). The P300, which could be associated with the more conscious detection of the difference of the acoustic properties, was consequently also locked to O4. In conclusion, the EEG results suggest that the decision (deviant/identical) could already be made at the time O4, that is, based solely on the hardware noise of the hearing device. Once the change in the hardware noise was detected, no further auditory processing of the actual stimulus was necessary. It should specifically be noted that the time at which the subject was able to make the decision was revealed purely by the EEG analysis and cannot be resolved based on the present psychophysical data.

The conclusion that subjects made the discrimination based on the hardware noise is supported by the comparison of the ERP amplitudes for the different stimuli. The largest amplitude change for both the N100 and P300 between identical/deviant last sounds is present for the Speech condition. The time between device onset and stimulus onset was largest for this condition. For the Speech condition, the hardware noise was audible for a longer duration and could therefore more easily be used to identify the filter condition. Consequently, the respective ERPs could be estimated more reliably as separate ERPs in response to different sound onsets overlapping to a smaller degree compared with the Noise and Speech-in-Noise condition. Especially in the P300 latency range, the ERP in response to O4 for the Noise and Speech-in-Noise stimuli appears to largely overlap with the ERP in response to S4, leading to small P300 median amplitudes (cf. Figure 7). This also explains the significant dependence of the P300 amplitude in response to O4 on the stimulus, although the sound the response is locked to is independent of the stimulus.

Admittedly, these results were not intended. We aimed to study the influence of the filter settings on the perception of the stimuli, and the hardware noise was regarded as a disturbance. However, our results provide relevant practical insights for future EEG studies regarding aided hearing. The hardware noise of hearing devices is a common issue that can often be neglected for hearing impaired subjects, but becomes relevant in normal-hearing subjects as in this study. Changes in the acoustic properties of the hardware noise are readily perceivable by the participants, leading to unwanted effects that bias the results. Apparently, avoiding transient cues when switching the processing setting by briefly turning off the hearing device were not sufficient to suppress such disturbances. For future similar studies, we recommend to temporally separate any switches in the device operation from stimulus presentations by several hundred milliseconds, such that the cortical responses to both events are temporally separated. If problems specifically originate from audible hardware noise, another option is to additionally present noise that perceptually masks this cue, as successfully applied in Denk et al. (2017).

Signal Quality in the Individual Electrode Positions

The second objective of our study was to compare the signal quality in the around-the-ear and in-concha electrodes. For around-the-ear recordings, we used a cEEGrid, and for the in-concha recording, we used small ton electrodes. We found the larger effect sizes for the switch identification and the better SNRs for the cEEGrid electrodes. Given the effect sizes for the filter switch discrimination, the electrodes above and behind the ear referenced to a behind-the-ear location seem to be most informative.

Regarding the between-channel correlation analysis, we found that the signals recorded at different cEEGrid electrodes show differences in the correlation scores. The between-channel correlation depends on the angle between two electrodes (relative to the reference electrode), the smaller the angle the higher the signals were correlated. This is also in line with earlier studies where we find that different cEEGrid channels are sensitive to different neural sources (Bleichner & Debener, 2017). When referenced against a cEEGrid electrode located behind the ear, the concha channels showed very high between-channel correlations (r > .95). The correlation to the other cEEGrid electrodes, however, was small. Also, when the in-concha channels were rereferenced to one of the other in-concha channels (i.e., local bipolar reference), the interchannel correlation was much smaller (r ∼ .4). These results underline the importance of the choice of reference for ear-EEG recordings.

The small interelectrode distance in the concha and the small angle between in-concha electrodes relative to the far away cEEGrid reference results in highly correlated signals, by using a local reference; however, distinct signals can be picked up also in the concha. This is in line with earlier studies that have shown that distinct neuronal activity can be picked up at different electrode sites in the ear canal and concha (Mikkelsen et al., 2015). Furthermore, the in-concha electrodes were largely uncorrelated to the around-the-ear-channels. Together with results of earlier studies and the results reported here that show that meaningful information is recorded by the in-concha electrodes (Mikkelsen et al., 2015), we take this as evidence that cEEGrid and in-concha electrodes capture nonredundant information. How much independent information can be obtained at the different recording sites depends on the position of the neural source and interindividual anatomical differences.

The observed SNRs were generally higher in the around-the-ear electrodes than in the in-concha electrodes. However, we also found that the SNR of the in-concha electrodes can be as high as the SNR of around-the-ear electrodes in some participants. We used in-concha electrodes that were not tightly integrated with the earmold but were plugged into a bore, in addition to an electrode technology that was different between around-the-ear and in-concha electrodes. These factors might have contributed to the lower SNR for the in-concha electrodes. Another relevant factor could be amplifier noise (Kidmose, Looney, Ungstrup, Rank, & Mandic, 2013). The signal amplitude decreases as the electrode distance decreases, that is, the overall signal amplitude of the concha channels is small. It is hence possible that the amplifier noise (input noise 1 µV, according to amplifier specification) dominated the signal, which could also have led to the observed reduction in SNR. For future work, identical electrode technology should be used to unambiguously compare the SNR between in-concha versus around-the-ear EEG. Furthermore, an amplifier with even lower low input noise should be used to potentially improve the SNR of channels that are spatially close to each other.

Conclusions

We studied the feasibility of using around-the-ear (cEEGrid; Bleichner & Debener, 2017; Debener et al., 2015) and in-concha EEG electrodes integrated with a live hearing device (Denk et al., 2017) to objectively monitor sound perception. Although the used system was a laboratory-state prototype, there is no principal reason that electrodes in this layout cannot be included in a real hearing device. Specifically, we switched the processing settings of the hearing device between repeated sounds and studied the effects on the N100 and P300 waves of the ERPs. Besides the identification of such filter switches in the ERPs, we evaluated the signal quality of around-the-ear and in-concha electrodes.

The results show that high-quality ERPs can be obtained from ear-EEG during hearing device activity without crosstalk from the electroacoustic system. The filter switch of the hearing device could be identified in the ERP responses. The EEG results revealed that the subjects detected the switch already in the hardware noise of the hearing device preceding the actual stimulus, although transient changes were avoided—an issue that should be considered in future applications of EEG together with aided hearing. Importantly, the EEG provided additional insight in the perception process that was not apparent in the current psychophysical data, namely the point in time where the deviation was perceived by the subject.

Signals from in-concha and around-the-ear electrodes were largely uncorrelated and appear to capture nonredundant information. Also, individual around-the-ear electrodes were less correlated among each other than the in-concha electrodes. For the correlation between electrode channels, the position of the reference electrode and the angle between electrode-reference vectors seem to be the most dominant factor, rather than the distance. The achievable SNR appears to be higher in the around-the-ear electrodes than in the in-concha electrodes, although the different electrode technologies utilized here may have biased the result. The effect size for the switch discrimination was larger when only around-the-ear electrodes were utilized than for sole utilization of in-concha electrodes. Altogether, according to our results, an array of electrodes distributed around (most importantly above and behind) the ear, potentially extended by one (or more) electrode(s) in the concha or ear canal, is highly promising for applications in hearing devices.

In summary, our results show that using ear-EEG in combination with live hearing devices is possible. EEG signals acquired in close proximity of the ear contain relevant neural information that may be harvested to realize brain-computer interface technology that is integrated into hearing devices of the future.

Footnotes

Acknowledgments

The authors thank Reiner Emkes for technical support,Bojana Mirkovic for practical support in handling the cEEGrids,and Tobias Herzke for support on the Master Hearing Aid.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research,authorship,and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research,authorship,and/or publication of this article: This work was funded by the DFG through the Cluster of Excellence EXC 1077 Hearing4all (Task Groups 4 “Algorithms for Hearing Instruments” and 7 “BCI for Hearing Aids”) and Research Unit FOR1732 Individualized Hearing Acoustics .

References

Barry

R. J.

Cocker

K. I.

Anderson

J. W.

Gordon

Rennie

(1992) Does the N100 evoked potential really habituate? Evidence from a paradigm appropriate to a clinical setting. International Journal of Psychophysiology 13(1): 9–16. doi:10.1016/0167-8760(92)90014-3.

Bleichner

M. G.

Debener

(2017) Concealed, unobtrusive ear-centered EEG acquisition: cEEGrids for transparent EEG. Frontiers in Human Neuroscience 11: 163. doi:10.3389/fnhum.2017.00163.

Bleichner

M. G.

Lundbeck

Selisky

Minow

Jäger

Emkes

De Vos

(2015) Exploring miniaturized EEG electrodes for brain-computer interfaces. An EEG you do not see? Physiological Reports 3(4): doi:10.14814/phy2.12362.

Bleichner

M. G.

Mirkovic

Debener

(2016) Identifying auditory attention with ear-EEG: cEEGrid versus high-density cap-EEG comparison. Journal of Neural Engineering 13(6): 066004. doi:10.1088/1741-2560/13/6/066004.

Cohen

(1988) Statistical power analysis for the behavioral sciences, 2nd ed. Hillsdale, NJ: L. Erlbaum Associates.

Debener

Emkes

De Vos

Bleichner

(2015) Unobtrusive ambulatory EEG using a smartphone and flexible printed electrodes around the ear. Scientific Reports 5: 16743. doi:10.1038/srep16743.

Delorme

Makeig

(2004) EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods 134(1): 9–21. doi:10.1016/j.jneumeth.2003.10.009.

Denk, F., Hiipakka, M., Kollmeier, B., & Ernst, S. M. A. (2017). An individualised acoustically transparent earpiece for hearing devices. International Journal of Audiology. Advance online publication. doi:10.1080/14992027.2017.1294768.

Finke

Billinger

Büchner

(2017) Toward automated cochlear implant fitting procedures based on event-related potentials. Ear and Hearing 38(2): e118–e127. doi:10.1097/AUD.0000000000000377.

10.

Goverdovsky

Looney

Kidmose

Mandic

D. P.

(2016) In-ear EEG from viscoelastic generic earpieces: Robust and unobtrusive 24/7 monitoring. IEEE Sensors Journal 16(1): 271–277. doi:10.1109/JSEN.2015.2471183.

11.

Grimm, G., Herzke, T., Berg, D., & Hohmann, V. (2006). The master hearing aid: A PC-based platform for algorithm development and evaluation. Acta acustica united with Acustica, 92(4), 618–628.

12.

Hedges

L. V.

(1981) Distribution theory for glass’s estimator of effect size and related estimators. Journal of Educational Statistics 6(2): 107–128. doi:10.2307/1164588.

13.

Hiipakka

Tikander

Karjalainen

(2010) Modeling the external ear acoustics for insert headphone usage. Journal of the Audio Engineering Society 58(4): 269–281.

14.

Kidmose

Looney

Ungstrup

Rank

M. L.

Mandic

D. P.

(2013) A study of evoked potentials from ear-EEG. IEEE Transactions on Biomedical Engineering 60(10): 2824–2830. doi:10.1109/TBME.2013.2264956.

15.

Looney, D., Kidmose, P., Park, C., Ungstrup, M., Rank, M., Rosenkranz, K., & Mandic, D. (2012). The in-the-ear recording concept: User-centered and wearable brain monitoring. IEEE Pulse, 3(6), 32–42. doi:10.1109/MPUL.2012.2216717.

16.

Lunner, T., & Neher, T. (2013). Automatic real-time hearing aid fitting based on auditory evoked potentials. Retrieved from http://www.google.com/patents/US20130101128.

17.

Meyer

B. T.

Jürgens

Wesker

Brand

Kollmeier

(2010) Human phoneme recognition depending on speech-intrinsic variability. The Journal of the Acoustical Society of America 128(5): 3126–3141. doi:10.1121/1.3493450.

18.

Mikkelsen

K. B.

Kappel

S. L.

Mandic

D. P.

Kidmose

(2015) EEG recorded from the ear: Characterizing the ear-EEG method. Frontiers in Neuroscience 9: 438. doi:10.3389/fnins.2015.00438.

19.

Mirkovic

Bleichner

M. G.

De Vos

Debener

(2016) Target speaker detection with concealed EEG around the ear. Frontiers in Neuroscience 10: 349. doi:10.3389/fnins.2016.00349.

20.

Møller

Hammershøi

Jensen

C. B.

Sørensen

M. F.

(1995) Transfer characteristics of headphones measured on human ears. Journal of the Audio Engineering Society 43(4): 203–217.

21.

Näätänen

Picton

(1987) The N1 wave of the human electric and magnetic response to sound: A review and an analysis of the component structure. Psychophysiology 24(4): 375–425. doi:10.1111/j.1469-8986.1987.tb00311.x.

22.

O’Sullivan

J. A.

Power

A. J.

Mesgarani

Rajaram

Foxe

J. J.

Shinn-Cunningham

B. G.

Lalor

E. C.

(2015) Attentional selection in a cocktail party environment can be decoded from single-trial EEG. Cerebral Cortex 25(7): 1697–1706. doi:10.1093/cercor/bht355.

23.

O'Sullivan, J., Chen, Z., Herrero, J., McKhann, G. M., Sheth, S. A., Mehta, A. D., & Mesgarani, N. (2017). Neural decoding of attentional selection in multi-speaker environments without access to clean sources. Journal of Neural Engineering, 14(5), 056001. doi:10.1088/1741-2552/aa7ab4.

24.

Polich

(2007) Updating P300: An integrative theory of P3a and P3b. Clinical Neurophysiology 118(10): 2128–2148. doi:10.1016/j.clinph.2007.04.019.

25.

Popelka

G. R.

Moore

B. C. J.

(2016) Future directions for hearing aid development. In: Popelka

G. R.

Moore

B. C. J.

Fay

R. R.

Popper

A. N.

(eds) Hearing aids, Cham, Switzerland: Springer, pp. 323–333. doi:10.1007/978-3-319-33036-5_11.

26.

Schimmel

(1967) The (+) reference: Accuracy of estimated mean components in average response studies. Science (New York, N.Y.) 157(3784): 92–94.

27.

Swartz Center for Computational Neuroscience & Kothe, C. (2015). Lab streaming layer (LSL). Schwartz Center for Computational Neuroscience. Retrieved from https://github.com/sccn/labstreaminglayer.

Event-Related Potentials Measured From In and Around the Ear Electrodes Integrated in a Live Hearing Device for Monitoring Sound Perception

Abstract

Keywords

Introduction

Methods

Participants

Acoustic Setup

EEG Setup

Paradigm

Stimuli

Hearing Device Settings

EEG Analysis

Influence of Electrode Position

Electrode configurations

Effect size

SNR comparison

Interelectrode correlation

Results

Psychophysical Results

EEG Results

Piloting results

N100 amplitude reduction

Filter switch

Electrode comparison

Discussion

ERPs and Identification of the Filter Switch

Signal Quality in the Individual Electrode Positions

Conclusions

Footnotes

Acknowledgments

Declaration of Conflicting Interests

Funding

References