Does Sight Predominate Sound? Electrophysiological Evidence for Multisensory Mismatch Negativity Correlation
When being presented with consistent and repetitive sensory stimuli, the human brain creates a predictive “memory trace” against which subsequent stimuli are compared. When later stimuli do not match this predictive model, a highly localized negative shift in the brain polarity occurs. This respo...
Saved in:
| Published in: | Нейрофизиология |
|---|---|
| Date: | 2013 |
| Main Authors: | , , |
| Format: | Article |
| Language: | English |
| Published: |
Інститут фізіології ім. О.О. Богомольця НАН України
2013
|
| Online Access: | https://nasplib.isofts.kiev.ua/handle/123456789/148247 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Journal Title: | Digital Library of Periodicals of National Academy of Sciences of Ukraine |
| Cite this: | Does Sight Predominate Sound? Electrophysiological Evidence for Multisensory Mismatch Negativity Correlation / J.C. Horvath, L. Schilberg, J. Thomson // Нейрофизиология. — 2013. — Т. 45, № 5. — С. 476-484. — Бібліогр.: 44 назв. — англ. |
Institution
Digital Library of Periodicals of National Academy of Sciences of Ukraine| _version_ | 1859707981412696064 |
|---|---|
| author | Horvath, J.C. Schilberg, L. Thomson, J. |
| author_facet | Horvath, J.C. Schilberg, L. Thomson, J. |
| citation_txt | Does Sight Predominate Sound? Electrophysiological Evidence for Multisensory Mismatch Negativity Correlation / J.C. Horvath, L. Schilberg, J. Thomson // Нейрофизиология. — 2013. — Т. 45, № 5. — С. 476-484. — Бібліогр.: 44 назв. — англ. |
| collection | DSpace DC |
| container_title | Нейрофизиология |
| description | When being presented with consistent and repetitive sensory stimuli, the human brain creates a
predictive “memory trace” against which subsequent stimuli are compared. When later stimuli
do not match this predictive model, a highly localized negative shift in the brain polarity
occurs. This response, known as the mismatch negativity (MMN), is believed to represent
a pre-attentive deviance-detection mechanism that serves to provide direct attention toward
unanticipated events. At present, there are conflicting data as to whether visually generated
and auditorily generated MMNs interact, or whether they are mediated by independent
sensory-specific networks. We present compelling evidence that visual and auditory MMNs
are strongly correlated, and that, upon presentation of dual-sensory “audiovisual” deviants,
this synergy is heavily dictated by an individual’s unique visual response. This finding is
suggestive of inhibitory interaction between the visual and auditory MMN networks. The
characterization of this correlation helps one to explain (and explain away) much conflicting
data published to date and opens the door to many questions regarding individual perception.
Під впливом стійких та повторних сенсорних стимулів
мозок людини створює предиктивну енграму, з якою порівнюються наступні подразники. У випадку, коли останні стимули не відповідають створеній предиктивній моделі, відбувається особливо локалізоване негативне зміщення
мозковій полярності. Вважають, що ця відповідь, відома як негативність розузгодження (НР), є преатентивним механізмом девіантності та детектування, забезпечуючим концентрацію прямої уваги на непередбачуваних подіях. Нині існують суперечливі дані щодо того, що процеси «візуально-»
та «аудіогенерованої» НР безпосередньо взаємодіють або ж
що така взаємодія опосередковується незалежними сенсорно специфічними нервовими мережами. Ми подаємо переконливі свідчення про те, що процеси зорового та слухового
НР чітко корелюють. В разі пред’явлення подвійних сенсорних «аудіовізуальних» девіантів така синергія здебільшого
диктується унікальною зоровою відповіддю особи. Отримані нами дані вказують на гальмівну взаємодію процесів НР
у зорових та слухових нейронних мережах. Характеристика
такої кореляції допомагає розтлумачити (та аргументувати)
багато що із суперечливих відомостей, опублікованих нині,
та розв’язати багато складних питань щодо індивідуального сприйняття.
|
| first_indexed | 2025-12-01T03:38:43Z |
| format | Article |
| fulltext |
NEUROPHYSIOLOGY / НЕЙРОФИЗИОЛОГИЯ.—2013.—T. 45, № 5476
UDC 616-073.7+616.079.4
J. C. HORVATH1,2, L. SCHILBERG3,4, and J. THOMSON2
DOES SIGHT PREDOMINATE SOUND? ELECTROPHYSIOLOGICAL EVIDENCE
FOR MULTISENSORY MISMATCH NEGATIVITY CORRELATION
Received January 14, 2013.
When being presented with consistent and repetitive sensory stimuli, the human brain creates a
predictive “memory trace” against which subsequent stimuli are compared. When later stimuli
do not match this predictive model, a highly localized negative shift in the brain polarity
occurs. This response, known as the mismatch negativity (MMN), is believed to represent
a pre-attentive deviance-detection mechanism that serves to provide direct attention toward
unanticipated events. At present, there are conflicting data as to whether visually generated
and auditorily generated MMNs interact, or whether they are mediated by independent
sensory-specific networks. We present compelling evidence that visual and auditory MMNs
are strongly correlated, and that, upon presentation of dual-sensory “audiovisual” deviants,
this synergy is heavily dictated by an individual’s unique visual response. This finding is
suggestive of inhibitory interaction between the visual and auditory MMN networks. The
characterization of this correlation helps one to explain (and explain away) much conflicting
data published to date and opens the door to many questions regarding individual perception.
Keywords: event-related potentials (ERPs), mismatch negativity (MMN), multisensory
perception, multisensory attention.
1 University of Melbourne School of Psychological Sciences, Melbourne,
Australia.
2 Harvard Graduate School of Education, Cambridge, Massachusetts, USA.
3 Berenson-Allen Center for Noninvasive Brain Stimulation, Beth Israel
Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts,
USA.
4 University of Maastricht, Faculty of Psychology and Neuroscience,
Maastricht, the Netherlands.
Correspondence should be addressed to J. C. Horvath
(e-mail: jch155@mail.harvard.edu).
INTRODUCTION
The auditory mismatch negativity (aMMN) is an
event-related potential (ERP) component elicited by
any discernible violation in an otherwise consistent
chain of auditory stimuli [1-3]. Peaking at several
frontocentral scalp locations and approximately 100-
150 msec post-violation under conditions of both
attention and inattention, the aMMN is believed
to reflect a pre-attentive auditory sensory memory
that stores the characteristics of the standard stimuli
against which any incoming sound is compared and
determined to be “typical” or “deviant” [4-6]. It is
largely believed that the major neural source of the
aMMN is temporally located (bilateral auditory cortex)
with a secondary frontal source involved in initiating
an involuntary attentional switch to the deviant sound
[7-10].
Similarly to the aMMN, the visual mismatch
negativity (vMMN) is an ERP component elicited by
any discernible violation in an otherwise consistent
chain of visual stimuli (for review, see [11, 12]).
Despite several years of mild debate, the existence of
the vMMN has been confirmed by a number of studies
describing a negative deflection over the occipital pole
peaking approximately 100-300 msec post-violation
under both attentive and inattentive conditions
[13-18] (a debate regarding the temporal characteristics
of this component has recently arisen [12]; we will
explore this item in the Discussion). Like the aMMN,
the vMMN is theorized to reflect a pre-attentive visual
sensory memory “regularity/violation” detection
process [12, 14, 19, 20].
Despite their similarities, the aMMN and vMMN are
believed by many researchers to be exclusive processes
generated and mediated by largely independent neural
networks [21, 22]. This uni-sensory hypothesis finds
support not only in the unique scalp localizations of
the individual MMN components but also in discrete
data obtained from several experiments. For instance,
under control conditions within a McGurk effect
MMN paradigm, Sams et al. [23] simultaneously
NEUROPHYSIOLOGY / НЕЙРОФИЗИОЛОГИЯ.—2013.—T. 45, № 5 477
DOES SIGHT PREDOMINATE SOUND?
presented a consistent auditory speech stimulus
(/pa/) with either a red (standard) or a green (deviant)
visual circle. The deviant visual stimulus did not
elicit an aMMN and had no discernible effect on
frontal recordings, thereby suggesting dissociation
between the two sensory modalities. More recently,
Besle et al. [6] presented subjects with either pure
auditory, pure visual, or paired audiovisual standard/
deviant combinations. These authors reported that the
response to audiovisual deviants included both frontal
and occipital components, suggesting the auditory and
visual deviance-detection mechanisms were operating
in a parallel manner and separately.
However, additional data obtained from the
same experiments appear to support the opposite
supposition: The aMMN and vMMN are correlated.
Returning to Sams et al. [23], subjects were presented
under experimental conditions with the same auditory
speech stimulus (/pa/) time-paired, in this case, with
either a congruent standard visual stimulus (a person
mouthing /pa/) or an incongruous deviant visual
stimulus (a person mouthing /ka/; the McGurk effect
[24]). Sams [23] reported that, under this condition,
the deviant visual stimulus generated a clear aMMN
despite the fact that the auditory stimulus remained
unchanged. Similarly, Besle et al. [6] reported that
the vMMN elicited by audiovisual deviants was
statistically different from the vMMN elicited by visual
deviants alone, with two distinct peaks appearing at
the occipital pole during audiovisual deviants, thereby
suggesting an audiovisual interaction.
Despite a growing body of evidence supporting the
correlative deviant detection hypothesis utilizing both
the McGurk effect [25-28] and the ventriloquist illusion
[29, 30], several researchers maintain the sensory-
specific MMN assumption, citing the special status of
speech effects and the inherent nature of “illusions”
to circumvent typical neural function [31, 32]. Adding
fuel to this debate is the interexperimental variability
reported by many audiovisual MMN researchers.
For instance, Nyman et al. [21] presented subjects
with either auditory or simultaneous audiovisual
standard/deviant combinations. This research group
reported no difference in the timing or amplitude of
the evoked aMMN under either condition, a finding
since replicated by several authors [6, 32]. Utilizing a
similar paradigm, other researchers, however, reported
attenuated aMMN responses in the presence of visual
deviations [33-35]. Accordingly, two questions loom
large: Are the aMMN and vMMN correlated or do
they represent independent uni-sensory deviant
detection processes? What is to account for the
discrepant findings reported both within and between
experiments utilizing seemingly similar audiovisual
oddball paradigms?
To examine these questions, we designed an
oddball paradigm whereby audiovisual standards (AV)
were interrupted by deviations to either the auditory
domain (A′V), or the visual domain (AV′), or both
simultaneously (A′V′). To avoid the earlier cited
possible confounds of speech or illusion effects, we
paired simple auditory beeps (deviating in pitch) with
a simple checkerboard pattern (deviating in a color
pattern). Our hypothesis was rather straightforward:
If the aMMN and vMMN are correlated, then the
neural activity measured in response to the dual-
sensory A′V′ should be different from the sum of the
activity measured in response to the uni-sensory A′V
and AV′. Conversely, if the two MMN components are
independent, then the A′V′ response should reflect a
simple summation of the A′V and AV′ responses.
METHODS
Participants. Eighteen healthy subjects (10 men
and 8 women; ages 21-29, M = 23.8 years, s.d. = 3.96)
volunteered to participate in this study. All subjects
were right-handed with normal hearing and normal or
corrected-to-normal vision.
Stimuli and Procedure. The standard auditory
stimulus (A) was a 1,000 Hz sinusoidal tone played
for 100 msec (including 25-msec-long rise and fall
times). The deviant auditory stimulus (A′) was a
1,200 Hz sinusoidal tone with the same timing
parameters. Tones were played at a constant intensity
(70 dB) through a central forward-facing central
speaker located under the computer monitor.
The standard visual stimulus (V) was a 10 × 10 cm
checkerboard pattern with twenty-five 2 × 2 cm
internal squares. Alternating squares were either white
or dark gray (67% black). The deviant visual stimulus
(V′) was the same checkerboard; alternating squares
were, however, either green (RGB values of 181, 230,
and 29) or white. All stimuli were presented
agains t a b lack background and cons is ted
of simultaneous presentat ion of an auditory
and v i s ua l f e a t u r e . Each s t imu lu s wa s
presented for 100 msec with constant ISIs of
300 msec (off-set to on-set).
Following EEG set-up, subjects were seated in a
dark soundproof room 80 cm from a computer screen.
NEUROPHYSIOLOGY / НЕЙРОФИЗИОЛОГИЯ.—2013.—T. 45, № 5478
J. C. HORVATH, L. SCHILBERG, and J. THOMSON
Prior to stimuli presentation, a fixation cross appeared
in the center of the computer screen, and subjects
were asked to stare at the cross for the duration of the
study. Subjects completed a total of eight stimulus-
presentation blocks, each lasting approximately 5 min.
Between blocks, subjects were allowed a 2-min-long
break. Overall, subjects were exposed to 6,268 total
stimuli; among them were 5,656 AV stimuli (~90%),
132 A′V stimuli (~2.5%), 132 AV′ stimuli (~2.5%),
and 348 A′V′ stimuli (~5%) presented in a randomized
order.
EEG Recording and Analysis. Stimuli were
presented using E-Prime 2.0 software (Psychology
Software Tools Inc., USA) with event codes
synched with the ERP recording system. EEG was
continuously recorded via a QuickAmp amplifier
with a system bandpass 0.016 to 70 Hz and a
500 sec–1 digital sampling rate. Signals were recorded
using the BrainRecorder software program and saved
for future analysis. Thirty Ag/AgCl ring electrodes
were held in place by a fitted elastic cap and placed
according to the international 10-20 system at scalp
sites Fz, F3, F4, F7, F8, FC1, FC2, FC5, FC6,
FT9, FT10, CZ, C3, C4, T7, T8, CP1, CP2, CP5,
CP6, TP9, TP10, Pz, P3, P4, P7, P8, Oz, O1, and
O2. Electrode sites were prepped with alcohol and
NuPrep conductance gel. Vertical and horizontal eye
movements were recorded via two electrodes placed
at the left eye.
Data were digitally filtered off-line with a high-
pass filter of 0.01 Hz and a low-pass filter of
30 Hz and analyzed using an average reference. After
filtering, samples were segmented into 500-msec-
long epochs (starting 100 msec pre-stimulus onset).
Deviant trials occurring within 3 sec of a previous
deviant trial were discarded (as ample time is
needed for subjects to re-acclimate to the standard
stimuli). Trials with peak-to-peak EOG amplitudes
exceeding 200 mV were discarded to avoid blink
or eye-movement contaminations. The remaining
epochs were baseline-corrected and averaged,
and a final waveform was constructed. Average
peak (AP) measurements were performed between
110-145 msec post-stimulus onset (presented in
mV); AP values were not rectified to reflect the
componential polarity. Accordingly, negative and
positive values represented negative and positive
polarity, respectively. One-way ANOVAs were run
between average peak values at matching electrode
sights across each condition.
Following initial grand average analysis, each
subject’s averaged A′V′, A′V, and AV′ responses were
subtracted from his/her unique averaged AV response
at each electrode sight. This value represented the
differential response between typical and deviant
responses. These values were utilized in all correlation
analyses.
RESULTS
Initial Analysis: Grand Average Comparison.
Uni-sensory A′V and AV′ difference responses
reveal large and easily recognizable aMMN and
vMMN components (Fig. 1). With regard to A′V
stimuli, a significant negative deflection over
several frontal electrodes (primarily Fz, FC1,
and F4) appeared approximately 90 msec post-
stimulus onset and attenuated approximately
100 msec later. With regard to AV′ stimuli, a significant
negative deflection over several occipital electrodes
(primarily Oz, O1, and O2) appeared approximately
100 msec post-stimulus onset and attenuated
approximately 100 msec later. These neural responses
correlate well with the MMN characteristics described
in the literature and suggest our stimuli were effective
in eliciting individual MMN responses. In addition, a
small negative deflection appeared in response to A′V′
deviants over several frontal channels, and a larger
negative deflection appeared over the occipital pole.
Both of these deflections began approximately 90 msec
post-stimulus onset and attenuated approximately
100 msec later.
Peak-value descriptive and difference values
for electrodes F4 and Oz can be seen in Table 1.
One-way ANOVA showed a significant difference
between average responses at F4 across conditions
[F(3, 68) = 25.368, P < 0.001, h2 = 0.53]. Post
hoc analysis using the Bonferroni correction
for multiple comparisons revealed a significant
difference between AV and AV′ (MD = 1.52,
P < 0.01), AV and A′V (MD = 1.83, P < 0.001), A′V′ and
AV′ (MD = 2.27, P < 0.001), A′V′ and A′V
(MD = 1.08, P = 0.048), and AV′ and A′V
(MD = 3.35, P < 0.001). No significant difference
was found between AV and A′V′ (MD = 0.747,
P = 0.376). One-way ANOVA showed a significant
difference between average responses at Oz
across conditions [F(3, 68) = 23.564, P < 0.001,
h2 = 0.51]. Post hoc analysis using the Bonferroni
correction for multiple comparisons revealed
a significant difference between AV and A′V′
NEUROPHYSIOLOGY / НЕЙРОФИЗИОЛОГИЯ.—2013.—T. 45, № 5 479
DOES SIGHT PREDOMINATE SOUND?
(MD = 6.25, P < 0.001), AV and AV′ (MD = 8.25,
P < 0.001), A′V′ and A′V (MD = 6.80, P < 0.001),
and AV′ and A′V (MD = 8.80, P < 0.001). No
significant difference was found between AV
and A′V (MD = 0.55, P = 1.00) or A′V′ and AV′
(MD = 2.00, P = 0.749). These results suggest that
there was no difference between vMMN responses to
the pure visual and dual audiovisual deviants; however,
there was a difference between aMMN responses to
the pure auditory and dual audiovisual deviants.
Secondary Analysis: Visual and Auditory
Responses to A′V′ Stimuli . Examination of
the individual difference data (as obtained by
subtracting deviant values from standard values
at each electrode) revealed s trong negat ive
correlation between responses at Oz (visual) and F4
(auditory) during A′V′ presentation [r(17) = –0.83,
P < 0.001; Fig. 2A]. This correlation suggests
that, during A′V′ stimuli presentation, the stronger
an individual’s occipital negativity, the weaker
his/her frontal negativity will be. A correlation
analysis between individual difference data at Oz
0
0
0 0
0 0
0
0
0 0
0 0
4
4
4 4
µV µV µV
A′V′AV′A′V
F4
Oz
4 4
–4
–4
–4 –4
–4 –4
400
400
400 400
msec
msec
400 400
F i g. 1. Averaged potential waveforms elicited by audiovisual standards (AV), as compared to A′V, AV′, and A′V′ deviants over electrodes
F4 and Oz from 100 msec pre-stimulus to 400 msec post-stimulus for all 18 subjects. Negative values are plotted upwards.
Р и с. 1. Усереднені форми хвиль потенціалів, викликаних аудіовізуальними стандартними сигналами (AV) порівняно з їх
девіантами (A'V, AV' та A'V'), у межах від 100 мс перед пред’явленням стимулів до 400 мс після їх пред’явлення (у дослідження
були залучені 18 людей; відведення від F4 та Oz).
Average Peak (AP) Values (110-145 msec) at Electrodes F4 and Oz across Each Condition
Величини середніх максимумів (110–145 мс), відведених від F4 та Oz, в умовах пред’явлення стандартних стимулів (AV) та
їх девіантів (A'V, AV' та A'V')
Standard
and deviant
audiovisual (AV)
stimuli
Peak value descriptive and difference values
F4 AP (µV) F4 difference (µV) Oz AP (µV) Oz difference (µV)
AV –1.24 ± 0.95 – 3.48 ± 1.71 –
AV′ 0.29 ± 1.51 1.52 ± 1.49 –4.77 ± 5.43 –8.25 ± 4.88
A′V –3.06 ± 0.91 –1.83 ± 0.78 4.03 ± 1.59 0.55 ± 0.63
A′V′ –1.98 ± 1.26 –0.75 ± 1.14 –2.77 ± 4.99 –6.24 ± 4.45
Footnote. Means ± s.d. are shown.
NEUROPHYSIOLOGY / НЕЙРОФИЗИОЛОГИЯ.—2013.—T. 45, № 5480
J. C. HORVATH, L. SCHILBERG, and J. THOMSON
during the A′V′ and pure AV′ deviants revealed a very
strong relationship [r(17) = 0.96, P < 0.001; Fig. 2B].
A similar correlation analysis between individual
difference data at F4 during the A′V′ and the pure
A′V deviants revealed a weak insignificant correlation
[r(17) = 0.35, P = 0.159; Fig. 2C]. Extrapolated, these
facts suggest that a response to pure visual deviants
dictates how one responds to audiovisual oddballs;
however, a response to pure auditory oddballs does
not correlate with responses to mixed oddballs.
Tertiary Analysis: Visual and Auditory Responses
during AV′ and A′V Presentation. A close look at
Fig. 1 reveals a significant positive deflection over
frontal regions during AV′ presentation. To determine
if the vMMN strength impacted this deflection, we ran
correlation analysis between the difference values at
Oz and F4 under AV′condition. We found very strong
correlation in this case [r(17) = 0.87, P < 0.001;
Fig. 3A]. A similar analysis run between these
electrodes under A′V condition revealed significant
but somewhat weaker correlation [r(17) = 0.64,
P < 0.001; Fig. 3B]. Taken together, these findings
suggest that there is an inhibitory connection between
occipital and frontal regions during MMN elicitation.
Interestingly, this connection seems to show stronger
activation during visual deviance detection.
DISCUSSION
Grand-average analysis revealed that a classical aMMN
was elicited at F4 in response to A′V stimuli, and that
a classical vMMN was elicited at Oz in response to
AV′ stimuli. These findings confirm that the stimuli
utilized were effective. More interestingly, our data
suggest that dual-sensory deviance (A′V′) elicited
a vMMN response with a highly attenuated aMMN
response. Furthermore, the dual-deviance vMMN was
not significantly different from the pure vMMN. This
finding goes against the proposed summation theory
(see [6]) and suggests that, in the presence of the
specific dual-sensory stimuli we utilized, the visual
deviance-detection mechanism appears to interact
with the auditory deviance-detection mechanism.
Correlative examination of the data set revealed
three additional interesting findings. First, there
appears to be strong negative correlation between
responses at the occipital pole and frontal sites
during A′V′ presentation. More specifically, our data
suggest that the stronger an individual’s response to
the visual dimension of a dual-sensory deviant is,
µV µV µV
2
2 4 6 8 10 12 14 16 18 0
0
1
–1
–2
–3
–4
–5
2
2 4 6 8 10 12 14 16 18
2 4 6 8 10 12 14 16 18
–10
–15
–20
–25
–5
4
A B C
–4
–2
0
–6
–8
–10
–12
–14
–16
F i g. 2. Correlation between individual average peak (AP) values across conditions. A) Oz (dark line) and F4 during A′V′; data sorted
according to Oz values. B) Oz during A′V′ (dark line) and AV′; data sorted according to A′V′ values. C) F4 during A′V′ (dark line) and A′V;
data sorted according to A′V′ values.
Р и с. 2. Кореляція між величинами індивідуальних середніх максимумів в умовах пред’явлення стимулів A'V'.
µV µV
10
5
A B
2
0
4 6 8 10 12 14 16 18
2
0
–1
–2
–3
–4
–5
1
2
3
4
4 6 8 10 12 14 16 18
–5
–10
–15
–20
–25
F i g. 3. Correlation between individual average peak (AP) values
within conditions. A) Oz (dark line) and F4 during AV′; data sorted
according to Oz values. B) F4 (dark line) and Oz during A′V; data
sorted according to F4 values.
Р и с. 3. Кореляція між величинами індивідуальних середніх
максимумів в умовах пред’явлення стимулів A'V та AV'.
NEUROPHYSIOLOGY / НЕЙРОФИЗИОЛОГИЯ.—2013.—T. 45, № 5 481
DOES SIGHT PREDOMINATE SOUND?
the weaker their response to the auditory dimension
will be (and vice versa). Second, there appears to be
very strong positive correlation between the vMMN
amplitude under both AV′ and A′V′ conditions. Put
another way, our data suggest that an individual’s
response to a pure visual deviant will almost perfectly
predict his/her response to the visual dimension of an
audiovisual deviant (and, by extension, the auditory
dimension as well). Interestingly, this correlation
did not exist between the average aMMN amplitudes
under A′V and A′V′ conditions. Third, there appears
to be correlation between the pure MMN strength
and positive deflection at the opposing MMN site.
More specifically, the amplitude of an individual’s
vMMN (Oz) can strongly predict the amount of
positive deflection in the frontal regions (F4), and
vice versa (although to a lesser extent). Unfortunately,
the specific location of these positive defections is
difficult to be determined. As such, it is uncertain
whether these deflections represent inhibitory cross-
talk between sensory specific deviance detection
networks or simply regional patterns recorded by our
analyzed electrodes.
To date, researchers have utilized an integrative
sensory memory approach to explain any aMMN/
vMMN interact ion. Put s imply, i t has been
hypothesized that auditory and visual informations
interact and form a combined audiovisual signal, at
least in part before the pre-attentive MMN deviance
detection process occurs. This hypothesis finds support
in recent evidence suggesting that dual-sensory
integration is realized very early in the process of
sensory analysis [36-38]. However, an early interaction
effect does not bring us any closer to explaining: Why,
after sensory combination, the visual domain seems
to dictate signal processing? To address this unique
finding, we developed two possible theories. The first
is that of a singular deviance-detection network. If the
aMMN and vMMN spring from a singular network, one
would expect the activity in each node to be reflected
in the activity of the other node(s). This is close to
what we see: During audiovisual deviance detection,
the amplitude of the vMMN negatively fluctuates
with the aMMN amplitude. However, beyond this,
the singular network theory falls short. If both MMNs
were generated by a unified network, then one would
expect to see equal yet opposing fluctuations in the
network under conditions of unisensory deviance.
Although we see a strong frontal positive shift
during exposure to pure visual deviants, we do not
see an equally strong occipital positive shift during
exposure to pure auditory deviants. Additionally, the
unitary network concept does help us to explain why
the vMMN seems to dictate the aMMN action, but
not vice versa. Because of these shortcomings, we
feel a second explanation is more apt: The individual
MMN networks possess inhibitory connections. These
connections appear to be bidirectional, although
slightly stronger frontal-going than occipital-going.
This explanation would not only explain the positive
deflections during single-sensory deviations but would
also explain why, under conditions of audiovisual
deviation, the visual modality appears to assume
precedence. To test this theory, one could present pure
auditory and visual deviants in very close succession
(<100 msec) to determine if there is any response
attenuation.
The strong variation in vMMNs between our
subjects (as elicited by both the A′V′ and AV′ stimuli)
is certainly worth noting (Fig. 2B). Despite these
wildly different responses, the vMMN amplitude
was still found to correspond strongly to the aMMN
amplitude during audiovisual deviants, but not to
the aMMN amplitude during pure auditory deviants.
More specifically, the larger an individual’s occipital
negativity was in response to pure visual deviants,
the smaller his/her frontal negativity in response
to audiovisual deviants was (and vice versa).
Interestingly, this variation may help us to answer a
question asked in the Introduction (What is to account
for the discrepant findings reported both within and
between experiments utilizing seemingly similar
audiovisual oddball paradigms?). As a reminder,
Besle et al. [6] described both frontal and occipital
negativities following audiovisual deviants. It is
possible that the Besle’s participant group displayed
relatively small pure vMMN amplitudes. If this was
the case, one would expect somewhat larger frontal
negativity amplitudes during audiovisual deviant
presentation, which, following grand-averaging,
might certainly suggest a dual-negativity. Another
example: Whereas Sittiprapaporn [32] reported no
vMMN response to audiovisual deviants, Stekelenburg
and Vroomen [33] found no aMMN response to
audiovisual deviants. Again, it is possible that subjects
in the former study might display very small (or no)
pure vMMN responses, whereas subjects of the latter
group might display very large pure vMMN responses.
If this was the case, one would certainly expect to
find no vMMN or aMMN in response to audiovisual
deviants, respectively. Unfortunately, to determine the
validity of these suppositions, additional protocols
NEUROPHYSIOLOGY / НЕЙРОФИЗИОЛОГИЯ.—2013.—T. 45, № 5482
J. C. HORVATH, L. SCHILBERG, and J. THOMSON
examining the effects of auditory-only and visual-only
deviations should be undertaken. Stekelenburg and
Vroomen [33] did not report a visual-only protocol,
and Sittiprapaporn [32] did not discuss the effects of
visual-only deviants in frontal electrode sites.
There are two final points worth briefly discussing
with regard to our findings. The first is that of
attention. As participants in this study were instructed
to “look at the fixation cross” for the entire duration of
the study, it is quite possible that different participants
attended to different aspects of the presented stimuli.
Whereas this might explain the variability between our
subjects with regard to the vMMN amplitude (those
attending to the visual dimension registered a larger
vMMN), we feel this argument is unfounded for two
reasons. First, despite some early debate [1, 39], it
has long been established that the deviance-detection
mechanism (and, by extension, the MMN component)
is pre-attentive and shows minimal (if any) attenuation
across varied conditions of overt or covert attention
(for review, see [40, 41]). Second, any sensory-specific
attention effects would likely be reflected in the pure
vs dual-sensory deviant responses (as participants
would only have a “choice” of the preferred sensory
modality during the audiovisual deviants). However,
occipital responses under both the A′V′ and AV′
conditions were nearly identical (Fig. 2B). For these
reasons, we, again, do not feel attention diversion is
an explanation for (or shortcoming of) our findings.
A second point worth discussing is the temporal
characteristics of the vMMN response. As alluded
to in the Introduction, there is ongoing debate as to
the precise temporal characteristics of the vMMN.
Whereas many researchers reported a distinct occipital
negativity peaking approximately 100-200 msec post-
deviant onset (for review, see [11]), it has recently
been suggested that this component is merely a
refractory effect, and that the true vMMN does not
peak until 250-400 msec post-deviant onset (for
review, see [12]). Attempts to resolve this question
across varying visual domains via utilization of the
equiprobable paradigm [42] have led to dissimilar
conclusions (early vMMN [14, 16, 43] and late
vMMN [20, 44]). As we utilized a standard oddball
paradigm, we do not feel our results speak to this
debate. However, Czigler et al. [14] utilized visual
color deviants in their equiprobable paradigm (very
similar to that in our study) and concluded that the
early negativity reflected the true vMMN. As such, we
geared our analysis to reflect these findings.
Therefore, we have found compelling evidence
that, with our utilized stimuli, the vMMN and aMMN
appear to be correlated, and that this relationship
may be strongly dictated by the response within
the visual modality. We found that vMMNs elicited
by our visual-only and audiovisual deviants do
not differ from each other significantly, and that
occipital negativity corresponds strongly to frontal
positivity. In addition, we found that the vMMN
response (and, by extension, the audiovisual deviant
response) varies strongly between individuals. This
variation could serve as an explanation for some
of the conflicting data reported in the literature.
Finally, we noted that , under condit ions of
uni-sensory deviant presentation, there appears to be
considerable correlation between the MMN amplitude
and positive deflection at the opposing sensory MMN
site. Whether this fact represents inhibitory cross-talk
or a more regional pattern, remains unknown. Future
research is expedient to explore this relationship
utilizing source localization protocols and exploring
the precise temporal relationship between the MMN
negativity and correlated positivity.
The study was carried out in compliance with generally
accepted international and institutional ethical standards.
All subjects volunteered to participate in this study after
providing written informed consent.
The authors, J. C. Horvath, L. Schilberg, and J. Thomson,
have no conflict of interest.
Дж. К. Хорват1,2, Л. Шилберг3,4, Дж. Томсон2
ЧИ ДІЙСНО ЗІР ПРЕВАЛЮЄ НАД СЛУХОМ? ЕЛЕКТРО-
ФІЗІОЛОГІЧНИЙ ДОКАЗ КОРЕЛЯЦІЇ МУЛЬТИСЕНСОР-
НОЇ НЕГАТИВНОСТІ РОЗУЗГОДЖЕННЯ
1 Інститут психологічних наук при Мельбурнському Уні-
верситеті, Мельбурн (Австралія).
2 Гарвардський університет, Кембрідж, Массачусетс
(США).
3 Беренсон Ален-центр атравматичної стимуляції мозку, до-
слідницький медичний Бет Ізраель-центр, Гарвардська ме-
дична школа, Бостон, Массачусетс (США).
4 Маастрихтський Університет (Нідерланди).
Р е з ю м е
Під впливом стійких та повторних сенсорних стимулів
мозок людини створює предиктивну енграму, з якою по-
рівнюються наступні подразники. У випадку, коли остан-
ні стимули не відповідають створеній предиктивній моде-
лі, відбувається особливо локалізоване негативне зміщення
мозковій полярності. Вважають, що ця відповідь, відома як
NEUROPHYSIOLOGY / НЕЙРОФИЗИОЛОГИЯ.—2013.—T. 45, № 5 483
DOES SIGHT PREDOMINATE SOUND?
негативність розузгодження (НР), є преатентивним механіз-
мом девіантності та детектування, забезпечуючим концен-
трацію прямої уваги на непередбачуваних подіях. Нині іс-
нують суперечливі дані щодо того, що процеси «візуально-»
та «аудіогенерованої» НР безпосередньо взаємодіють або ж
що така взаємодія опосередковується незалежними сенсор-
но специфічними нервовими мережами. Ми подаємо пере-
конливі свідчення про те, що процеси зорового та слухового
НР чітко корелюють. В разі пред’явлення подвійних сенсор-
них «аудіовізуальних» девіантів така синергія здебільшого
диктується унікальною зоровою відповіддю особи. Отрима-
ні нами дані вказують на гальмівну взаємодію процесів НР
у зорових та слухових нейронних мережах. Характеристика
такої кореляції допомагає розтлумачити (та аргументувати)
багато що із суперечливих відомостей, опублікованих нині,
та розв’язати багато складних питань щодо індивідуально-
го сприйняття.
REFERENCES
1. R. Näätänen, A. W. Gaillard, and S. Mäntysalo, “Early
selective-attention effect on evoked potential reinterpreted,”
Acta Psychol., 42, No. 4, 313-329 (1978).
2. T. Kujala, M. Tervaniemi, and E. Schroger, “The mismatch
negativity in cognitive and clinical neuroscience: theoretical
and methodological considerations,” Biol. Psychol., 74, No. 1,
1-19 (2007).
3. R. Näätänen, P. Paavilainen, T. Rinne, and K. Alho, “The
mismatch negativity (MMN) in basic research of central
auditory processing: a review,” Clin. Neurophysiol., 118,
No. 12, 2544-2590 (2007).
4. M. Sams, P. Paavilainen, K. Alho, and R. Näätänen, “Auditory
frequency discrimination and event-related potential,”
Electroencephalogr. Clin. Neurophysiol., 62, No. 6, 437-448
(1985).
5. W. Ritter, D. Deacon, H. Gomes, et al., “The mismatch
negativity of event-related potentials as a probe of transient
auditory memory: a review,” Ear Hear., 16, No. 1, 52-67
(1995).
6. J. Besle, A. Fort, and M. H. Giard, “Is the auditory sensory
memory sensitive to visual information?” Exp. Brain Res., 166,
Nos. 3/4, 337-344 (2005).
7. M. H. Giard, F. Perrin, J. Pernier, and P. Bouchet, “Brain
generators implicated in the processing of auditory stimulus
deviance: a topographic event-related potential study,”
Psychophysiology, 27, No. 6, 627-664 (1990).
8. B. Opitz, T. Rinne, A. Mecklinger, et al., “Differential
contribution of frontal and temporal cortices to auditory
change detection: fMRI and ERP results,” Neuroimage, 15,
No. 1, 167-174 (2002).
9. L. Y. Deouell, A. Parnes, N. Pickard, and R. T. Knight, “Spatial
location is accurately tracked by human auditory sensory
memory: evidence form the mismatch negativity,” Eur. J.
Neurosci., 24, No. 5, 1488-1494 (2006).
10. M. I . Garrido, J . M. Kilner, K. E. Stephan, and
K. J. Friston, “The mismatch negativity: a review of
underlying mechanisms,” Clin. Neurophysiol . , 120 ,
No. 3, 453-463 (2009).
11. P. Pazo-Alvarez, F. Cadaveira, and E. Amenedo, “MMN
in the visual modality: a review,” Biol. Psychol., 63,
No. 3, 199-236 (2003).
12. M. Kimura, E. Schröger, and I. Czigler, “Visual mismatch
negativity and its importance in visual cognitive sciences,”
NeuroReport, 22, No. 14, 669-673 (2011).
13. K. Alho, D. L. Woods, A. Algazi, and R. Näätänen, “Intermodal
selective attention. II. Effects of attentional load on
processing of auditory and visual stimuli in central space,”
Electroencephalogr. Clin. Neurophysiol., 82, No. 5, 356-368
(1992).
14. I. Czigler, L. Balazs, and I. Winkler, “Memory-based detection
of task-irrelevant visual changes,” Psychophysiology, 36,
No. 6, 869-873 (2002).
15. J. L. Kenemans, T. T. G. Jong, and M. N. Verbaten, “Detection
of visual change: mismatch or rareness?” NeuroReport, 14,
No. 9, 1239-1242 (2003).
16. P. Pazo-Alvarez, E. Amenedo, and F. Cadaveira, “Automatic
detection of motion direction changes in the brain,” Eur. J.
Neurosci., 19, 1978-1986 (2004).
17. S. Berti and E. Schrцger, “Distraction effects in vision:
behavioral and event-related potential indices,” NeuroReport,
15, No. 4, 665-669 (2004).
18. C. Stagg, P. Hindley, A. Tales, and S. Butler, “Visual mismatch
negativity: the detection of stimulus change,” NeuroReport,
15, No. 4, 659-663 (2004).
19. I. Czigler, J. Weisz, and I. Winkler, “ERPs and deviance
detection: visual mismatch negativity to repeated visual
stimuli,” Neurosci. Lett., 401, 178-182 (2006).
20. M. Kimura, J. Katayama, H. Ohira, and E. Schroger, “Visual
mismatch negativity: new evidence from the equiprobable
paradigm,” Psychophysiology, 46, No. 2, 402-409 (2009).
21. G. Nyman, K. Alho, P. Laurinen, et al., “Mismatch negativity
(MMN) for sequences of auditory and visual stimuli:
evidence for a mechanism specific to the auditory modality,”
Electroencephalogr. Clin. Neurophysiol., 77, No. 6, 436-444
(1990).
22. R. Näätänen, Attention and Brain Function, Lawrence Erlbaum
Associates, Inc., Hillsdale, NJ (1992).
23. M. Sams, R. Aluanko, M. Hamalainen, et al., “Seeing speech:
visual information from lip movements modifies activity in the
human auditory cortex,” Neurosci. Lett., 127, No. 1, 141-145
(1991).
24. H. McGurk and J. MacDonald, “Hearing lips and seeing
voices,” Nature, 264, No. 5588, 746-748 (1976).
25. R. Möttönen, C. M. Krause, K. Tiippana, and M. Sams,
“Processing of changes in visual speech in the human auditory
cortex,” Cogn. Brain Res., 13, No. 3, 417-425 (2002).
26. C. Colin, M. Radeau, A. Soquet, et al., “Mismatch negativity
evoked by the McGurk-MacDonald effect: a phonetic
representation within short-term memory,” Clin. Neurophysiol.,
113, No. 40, 495-506 (2002).
27. C. Colin, M. Radeau, A. Soquet, and P. Deltenre,
“Generalization of the generation of an MMN by illusory
McGurk percepts: voiceless consonant,” Clin. Neurophysiol.,
115, No. 9, 1989-2000 (2004).
28. D. Saint-Amour, P. De Sanctis, S. Molholm, et al., “Seeing
voices: high-density electrical mapping and source-analysis
of the multisensory mismatch negativity evoked during the
McGurk illusion,” Neuropsychologia, 45, No. 3, 587-597
(2007).
29. C. Colin, M. Redeau, A. Soquet, et al., “Electrophysiology
of spatial scene analysis: the mismatch negativity (MMN) is
NEUROPHYSIOLOGY / НЕЙРОФИЗИОЛОГИЯ.—2013.—T. 45, № 5484
J. C. HORVATH, L. SCHILBERG, and J. THOMSON
sensitive to the ventriloquism illusion,” Clin. Neurophysiol.,
113, No. 4, 507-518 (2002).
30. J. J. Stekelenburg and J. Vroomen, “An event-related potential
investigation of the time-course of temporal ventriloquism,”
NeuroReport, 16, No. 6, 641-644 (2005).
31. A. Alsius, J. Navarra, R. Campbell, and S. Soto-Faraco,
“Audiovisual integration of speech falters under high attention
demands,” Current Biol., 15, No. 9, 839-843 (2005).
32. W. Sit t iprapaporn, “No visual mismatch negativi ty
(MMN) for s imul taneous ly presen ted audiovisua l
stimuli: evidence from human brain processing,” Int.
J. Appl. Biomed. Eng., 2, No. 2, 29-35 (2009).
33. J. J. Stekelenberg and J. Vroomen, “Neural correlates
of audiovisual motion capture,” Exp. Brain Res., 198,
Nos. 2/3, 383-390 (2009).
34. X. Li, Y. Yany, and G. Ren, “Immediate integration of
prosodic information from speech and visual information
from pictures in the absence of focused attention:
a mismatch negat ivi ty s tudy,” Neuroscience, 161 ,
59-66 (2009).
35. A. J. Andres, J. E. Oram Cardy, and M. F. Joanisse,
“Congruency of auditory sounds and visual letters modulates
mismatch negativity and P300 event-related potentials,” Int. J.
Psychophysiol., 79, No. 2, 137-146 (2011).
36. J. Besle, A. Fort, C. Delpuech, and M. H. Giard, “Bimodal
speech: Early suppressive visual effects in human auditory
cortex,” Eur. J. Neurosci., 20, No. 8, 2225-2232 (2004).
37. A. Fort and M. H. Giard, “Multiple electrophysiological
mechanisms of audio-visual integration in human perception,”
in: The Handbook of Multisensory Processes, G. Calvert,
C. Spence, and B. Stein (eds.), MIT Press, Cambridge, MA
(2004).
38. S. Watkins, L. Shams, S. Takana, et al., “Sound alters activity
in human V1 in association with illusory visual perception,”
Neuroimage, 31, 1247-1256 (2006).
39. S. A. Hil lyard, R. F. Hink, V. L. Schwent , and
T. W. Picton, “Electrical signs of selective attention in the
brain,” Science, 182, No. 4108, 177-180 (1973).
40. A. Bendixen, L. SanMiguel, and E. Schroger, “Early
electrophysiological indicators for predictive processing
in audit ion: a review,” Int . J . Psychophysiol . , 83 ,
120-131 (2012).
41. S. Grimm and C. Escera, “Auditory deviance detection
revisited: Evidence for a hierarchical novelty system,” Int. J.
Psychophysiol., 85, 88-92 (2012).
42. E. Schröger and C. Wolff, “Behavioral and electrophysiological
effects of task-irrelevant sound change: A new distraction
paradigm,” Cogn. Brain Res., 7, 71-87 (1998).
43. P. Astikainene, E. Lillstrang, and T. Ruusuvirta, “Visual
mismatch negativity to orientation changes: A memory-based
process,” Psychophysiology, 44, S8 (2007).
44. V. Mazza, M. Turatto, and M. Sarlo, “Rare stimuli or rare
changes: What really matters for the brain?” NeuroReport, 16,
1061-1064 (2005).
|
| id | nasplib_isofts_kiev_ua-123456789-148247 |
| institution | Digital Library of Periodicals of National Academy of Sciences of Ukraine |
| issn | 0028-2561 |
| language | English |
| last_indexed | 2025-12-01T03:38:43Z |
| publishDate | 2013 |
| publisher | Інститут фізіології ім. О.О. Богомольця НАН України |
| record_format | dspace |
| spelling | Horvath, J.C. Schilberg, L. Thomson, J. 2019-02-17T18:49:26Z 2019-02-17T18:49:26Z 2013 Does Sight Predominate Sound? Electrophysiological Evidence for Multisensory Mismatch Negativity Correlation / J.C. Horvath, L. Schilberg, J. Thomson // Нейрофизиология. — 2013. — Т. 45, № 5. — С. 476-484. — Бібліогр.: 44 назв. — англ. 0028-2561 https://nasplib.isofts.kiev.ua/handle/123456789/148247 616-073.7+616.079.4 When being presented with consistent and repetitive sensory stimuli, the human brain creates a predictive “memory trace” against which subsequent stimuli are compared. When later stimuli do not match this predictive model, a highly localized negative shift in the brain polarity occurs. This response, known as the mismatch negativity (MMN), is believed to represent a pre-attentive deviance-detection mechanism that serves to provide direct attention toward unanticipated events. At present, there are conflicting data as to whether visually generated and auditorily generated MMNs interact, or whether they are mediated by independent sensory-specific networks. We present compelling evidence that visual and auditory MMNs are strongly correlated, and that, upon presentation of dual-sensory “audiovisual” deviants, this synergy is heavily dictated by an individual’s unique visual response. This finding is suggestive of inhibitory interaction between the visual and auditory MMN networks. The characterization of this correlation helps one to explain (and explain away) much conflicting data published to date and opens the door to many questions regarding individual perception. Під впливом стійких та повторних сенсорних стимулів мозок людини створює предиктивну енграму, з якою порівнюються наступні подразники. У випадку, коли останні стимули не відповідають створеній предиктивній моделі, відбувається особливо локалізоване негативне зміщення мозковій полярності. Вважають, що ця відповідь, відома як негативність розузгодження (НР), є преатентивним механізмом девіантності та детектування, забезпечуючим концентрацію прямої уваги на непередбачуваних подіях. Нині існують суперечливі дані щодо того, що процеси «візуально-» та «аудіогенерованої» НР безпосередньо взаємодіють або ж що така взаємодія опосередковується незалежними сенсорно специфічними нервовими мережами. Ми подаємо переконливі свідчення про те, що процеси зорового та слухового НР чітко корелюють. В разі пред’явлення подвійних сенсорних «аудіовізуальних» девіантів така синергія здебільшого диктується унікальною зоровою відповіддю особи. Отримані нами дані вказують на гальмівну взаємодію процесів НР у зорових та слухових нейронних мережах. Характеристика такої кореляції допомагає розтлумачити (та аргументувати) багато що із суперечливих відомостей, опублікованих нині, та розв’язати багато складних питань щодо індивідуального сприйняття. en Інститут фізіології ім. О.О. Богомольця НАН України Нейрофизиология Does Sight Predominate Sound? Electrophysiological Evidence for Multisensory Mismatch Negativity Correlation Чи дійсно зір превалює над слухом? електрофізіологічний доказ кореляції мультисенсорної негативності розузгодження Article published earlier |
| spellingShingle | Does Sight Predominate Sound? Electrophysiological Evidence for Multisensory Mismatch Negativity Correlation Horvath, J.C. Schilberg, L. Thomson, J. |
| title | Does Sight Predominate Sound? Electrophysiological Evidence for Multisensory Mismatch Negativity Correlation |
| title_alt | Чи дійсно зір превалює над слухом? електрофізіологічний доказ кореляції мультисенсорної негативності розузгодження |
| title_full | Does Sight Predominate Sound? Electrophysiological Evidence for Multisensory Mismatch Negativity Correlation |
| title_fullStr | Does Sight Predominate Sound? Electrophysiological Evidence for Multisensory Mismatch Negativity Correlation |
| title_full_unstemmed | Does Sight Predominate Sound? Electrophysiological Evidence for Multisensory Mismatch Negativity Correlation |
| title_short | Does Sight Predominate Sound? Electrophysiological Evidence for Multisensory Mismatch Negativity Correlation |
| title_sort | does sight predominate sound? electrophysiological evidence for multisensory mismatch negativity correlation |
| url | https://nasplib.isofts.kiev.ua/handle/123456789/148247 |
| work_keys_str_mv | AT horvathjc doessightpredominatesoundelectrophysiologicalevidenceformultisensorymismatchnegativitycorrelation AT schilbergl doessightpredominatesoundelectrophysiologicalevidenceformultisensorymismatchnegativitycorrelation AT thomsonj doessightpredominatesoundelectrophysiologicalevidenceformultisensorymismatchnegativitycorrelation AT horvathjc čidíisnozírprevalûênadsluhomelektrofízíologíčniidokazkorelâcíímulʹtisensornoínegativnostírozuzgodžennâ AT schilbergl čidíisnozírprevalûênadsluhomelektrofízíologíčniidokazkorelâcíímulʹtisensornoínegativnostírozuzgodžennâ AT thomsonj čidíisnozírprevalûênadsluhomelektrofízíologíčniidokazkorelâcíímulʹtisensornoínegativnostírozuzgodžennâ |