Combining discriminative features to infer complex trajectories

结合数据平衡和注意力机制的CNN+LSTM 的自然语音情感识别①陈　港1, 张石清2, 赵小明1,21(浙江理工大学 机械与自动控制学院, 杭州 310018)2(台州学院 智能信息处理研究所, 台州 318000)通讯作者: 赵小明摘　要: 为了解决语音情感识别中数据集样本分布不平衡的问题, 提出一种结合数据平衡和注意力机制的卷积神经网络(CNN)和长短时记忆单元(LSTM)的语音情感识别方法. 该方法首先对语音情感数据集中的语音样本提取对数梅尔频谱图, 并根据样本分布特点对进行分段处理, 以便实现数据平衡处理, 通过在分段的梅尔频谱数据集中微调预训练好的CNN 模型, 用于学习高层次的片段语音特征. 随后, 考虑到语音中不同片段区域在情感识别作用的差异性, 将学习到的分段CNN 特征输入到带有注意力机制的LSTM 中, 用于学习判别性特征, 并结合LSTM 和Softmax 层从而实现语音情感的分类. 在BAUM-1s 和CHEAVD2.0数据集中的实验结果表明, 本文提出的语音情感识别方法能有效地提高语音情感识别性能.关键词: 卷积神经网络; 长短时记忆单元; 注意力机制; 语音情感识别引用格式: 陈港,张石清,赵小明.结合数据平衡和注意力机制的CNN+LSTM 的自然语音情感识别.计算机系统应用,2021,30(5):269–275./1003-3254/7917.htmlNatural Speech Emotion Recognition by Integrating Data Balance and Attention Mechanism Based on CNN+LSTMCHEN Gang 1, ZHANG Shi-Qing 2, ZHAO Xiao-Ming 1,21(Faculty of Mechanical Engineering & Automation, Zhejiang Sci-Tech University, Hangzhou 310018, China)2(Institute of Intelligent Information Processing, Taizhou University, Taizhou 318000, China)Abstract : In order to solve the problem of unbalanced sample distribution in a dataset in Speech Emotion Recognition (SER), this study proposes a SER method combining a Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) units with data balance and an attention mechanism. This method first extracts the log-Mel spectrogram from the samples in a speech emotion dataset and devides the sample distribution into segments according to sample distribution for balance. Then, this method fine-tunes the pre-trained CNN model in the segmented Mel-spectrum dataset to learn high-level speech segments. Next, given the differences in the emotion recognition of different segments in speech, the learned segmented CNN features are input into the LSTM with an attention mechanism for learning discriminative features, and speech emotions are classified with LSTM and Softmax layers. The experimental results in the BAUM-1s and CHEAVD2.0 datasets show that the method proposed in this study has much better performance than conventional methods.Key words : Convolutional Neural Network (CNN); Long Short-Term Memory (LSTM) unit; attention mechanism; speech emotion recognition计算机系统应用 ISSN 1003-3254, CODEN CSAOBNE-mail: Computer Systems & Applications,2021,30(5):269−275 [doi: 10.15888/ki.csa.007917] ©中国科学院软件研究所版权所有.Tel: +86-10-62661041① 基金项目: 国家自然科学基金 (61976149); 浙江省自然科学基金 (LZ20F020002)Foundation item: National Natural Science Foundation of China (61976149); Natural Science Foundation of Zhejiang Province (LZ20F020002)收稿时间: 2020-09-23; 修改时间: 2020-10-21; 采用时间: 2020-10-28; csa 在线出版时间: 2021-04-28269人类的语言不仅包含了丰富的文本信息, 同时也携带着包含人们情绪表达的音频信息, 如语音的高低、强弱、抑扬顿挫等变化. 如何让计算机从语音信号中自动识别出说话人的情感状态, 即所谓的“语音情感识别”方面的研究, 已成为人工智能、模式识别、情感计算等领域中的一个热点研究课题. 该研究旨在让计算机通过分析说话人的语音信号对用户的情感信息进行获取、识别和响应, 从而实现用户与计算机之间的交互更加和谐与自然. 该研究在智能人机交互、电话客服中心、机器人等方面具有重要的应用价值.目前, 在语音情感识别领域中, 大量的前期工作[1–4]主要是针对模拟情感而进行的, 因为这种模拟情感数据库的建立相对自然情感而言, 要容易得多. 近年来,针对实际环境下的自然语音情感识别方面的研究备受研究者的关注, 因为它更接近实际, 而且比模拟情感的识别要困难得多.语音情感特征提取, 是语音情感识别中的一个关键步骤, 其目的是从情感语音信号中提取能够反映说话人情感表达信息的特征参数. 目前, 大量语音情感识别文献[5–10]采用手工设计的特征用于情感识别, 如韵律特征(基频、振幅、发音持续时间)、音质特征(共振峰、频谱能量分布、谐波噪声比), 谱特征(梅尔频率倒谱系数(Mel Frequency Cepstrum Coefficient, MFCC))等. 近年来, 也出现了一些代表性的包含几千个手工设计特征的声学特征集, 如Interspeech-2010特征集[11], ComParE特征集[12], AVEC-2013特征集[13]以及GeMAPS特征集[14]. 尽管这些手工设计的特征参数已经取得了较好的语音情感识别性能, 但它们是低层次的, 对于情感的判别力还不够高, 与人类理解的情感标签还存在“语义鸿沟”问题.为了解决这个问题, 近年来新出现的深度学习技术[15]可能提供了线索. 近年来, 一些代表性的深度学习方法, 如深度信念网络(Deep Belief Network, DBN)[15]、卷积神经网络(Convolutional Neural Networks, CNN)[16]和长短时记忆单元(Long Short-Term Memory, LSTM)[17]都已经用于语音情感识别. 当使用深度学习方法时, 其输入一般为手工设计的声学特征参数, 或者原始的语音频谱. 例如, 文献[18]采用DBN直接从提取的MFCC 等声学特征参数中提取高层次的属性特征, 然后使用极限学习机(Extreme Learning Machine, ELM)实现情感的分类任务. 文献[19–22]也开始成功使用CNN从原始的语音频谱中提取出合适的特征参数用于语音情感识别. 例如, 文献[19]采用稀疏自动编码器和1层CNN结构的方法从原始的语音频谱中学习情感语音特征.值得指出的是, 文献[19–21]采用样本数量非常有限的情感语音数据集来训练自己的浅层CNN模型(1或2个卷积层). 然而, 在计算机视觉领域, 利用已训练好的深度CNN模型, 如AlexNet[16], 在目标图像数据集进行迁移学习往往取得比浅层CNN模型更好的性能. 主要原因是, 深度CNN模型可以通过采用多层的卷积和池化操作来捕获图像的高层属性特征. 为了充分发挥深度CNN模型的优势, 我们之前的一个工作[22]提出将一维的情感语音信号转换成类似于RGB图像的三通道语音频谱片段作为深度CNN模型的输入, 然后将在ImageNet图像数据集已训练好的AlexNet模型在目标语音情感数据集进行跨模态的迁移学习, 取得了比浅层CNN模型更好的语音情感识别性能. 此外,文献[23]提出一种多尺度的CNN+LSTM的混合深度学习模型, 获得了较好的语音情感性能.然而, 现有基于深度学习方法的语音情感识别研究存在一些问题:(1)没有考虑语音数据集客观存在的情感类别不平衡问题, 即数据集中各种情感类型的样本数量极不均衡. 在这种数据类别不平衡的情况下, 深度学习方法训练时对样本少的情感类型容易出现过拟合的现象.(2)语音信号是一种时间序列信号, 但不同时间上的片段区域对情感识别的作用大小是不一样的. 现有文献大都忽略了一句语音中不同片段区域在情感识别方面作用的差异性, 从而限制了深度学习方法的特征表征能力. 因此, 有必要将人类视觉注意力机制(attention mechanism)与深度学习方法相结合[24,25], 用于语音情感识别的特征学习.针对上述问题, 本文提出一种结合数据平衡和注意力机制的CNN+LSTM的语音情感识别方法, 并且用于自然语音情感类型的识别. 首先, 采用欠采样和过采样的方式实现情感语音数据集中的样本片段数量的类别平衡. 所谓欠采样是在多数类中的语音样本进行部分采样, 而过采样是在少数类中的语音样本进行部分重复采样. 其次, 采用在大规模音频数据集上已训练好的VGGish[26]模型在目标语音情感数据集上进行迁移学习. 最后, 通过对VGGish学习到的一句语音中不计算机系统应用2021 年 第 30 卷 第 5 期270同的片段特征区域, 生成时间分布上的权重, 然后和特征图进行加权求和运算, 从而监督双向LSTM 网络(Bi-LSTM)的学习, 以便给一句语音中不同片段特征区域分配不同的注意力权重值. 这样, 该方法能够聚焦于一句语音中对情感识别起作用的目标片段区域的特征学习, 从而改善深度神经网络的特征表征能力, 进一步提高语音情感识别性能.在自然情感语音数据库BAUM-1s [27]和CHEAVD-2.0[28]的试验结果表明, 本文方法取得的语音情感识别性能优于其它方法.1 提出的方法图1给出了结合数据平衡和注意力机制的CNN+LSTM 的语音情感识别方法的系统流程图. 由图1所示, 该方法包括4个步骤: (1)对数梅尔频谱(Log Mel-spectrogram)的创建和数据平衡(data balance); (2)基于CNN 的深度片段特征学习; (3)基于注意力机制的Bi-LSTM 的情感分类. 图1中每个步骤的实现, 具体如下所述.帧数滤波器组数据注意力层情感类别情感类别全连接层SoftmaxConv1Conv2Pool1Pool2FC Layer 平衡对数梅尔频谱图………VGGish ModelBi-LSTMLSTMLSTM数据平衡生气高兴厌恶悲伤害怕惊奇图1 结合数据平衡和注意力机制的CNN+LSTM 的语音情感识别方法1.1 对数梅尔频谱的创建和数据平衡对原始的一维情感语音信号重采样为16 kHz 的单声道格式, 然后采用帧移为10 ms, 时长为25 ms 的汉宁窗进行短时傅里叶变换, 计算出整句语音信号的声谱图. 将声谱图映射到64阶Mel 滤波器组(Filter banks)中计算出Mel 声谱并取对数, 得到稳定的对数Mel 频谱.考虑到用于后续特征学习的VGGish [26]模型输入的片段大小是96×64, 其时长为0.96 s (含96帧, 每帧0.01 s), 因此需要通过控制时长进行欠采样或重采样, 以便实现训练数据的平衡. 图2和图3分别给出了BAUM-1s 和CHEAVD2.0上的数据平衡示意图.高兴, 悲伤生气, 厌恶害怕, 惊奇时长<0.96 s0.96 s0.96 s0.96 s0.48 s0.24 s0.75 s 0.75 s数据平衡 (BAYM-1s)图2 BAUM-1s 数据集上的数据平衡中性生气, 高兴,悲伤, 担心厌恶, 惊奇焦虑时长<0.96 s0.96 s0.96 s 0.96 s 0.48 s0.24 s0.75 s 0.75 s0.96 s0.96 s 0.96 s数据平衡 (CHEA VD2.0)图3 CHEAVD2.0数据集上的数据平衡如图2, 图3所示, 由于数据集中的每个语音样本的时长是不同的, 然而模型的输入大小是固定的, 因此使用数据平衡方法对输入的语音样本进行规范化处理,数据平衡的实现分为3种情况: (1)对于时长小于0.96 s (时长<0.96 s)的语音样本, 在其后面进行重复补值为零的帧, 从而得到一个时长为0.96 s 的片段, 即含有96帧的对数梅尔频谱图; (2)对于时长大于0.96 s 的语音样本, 会根据其所属类别的样本总数占数据集总样本数的比例的多少进行欠采样或重采样. 例如, 对于BAUM-1s 上类别样本数量较多的高兴(Joy)和悲伤(Sadness)的语音样本采用欠采样方式减少片段数,即只在它们的首尾段处各取一个0.96 s 的片段, 而对于样本数量不多的语音样本采用重采样方式进行片段数的放大. 对于样本数最少的害怕(Fear)和惊奇(Surprise)语音样本, 片段重叠长度为0.24 s, 而样本数较少的生气(Anger)和厌恶(Disgust), 其片段重叠长度为0.48 s. 数据平衡中产生的片段样本的类别等于其所在整句语音的情感类别. 表1和表2分别列出了采用数据平衡前后在BAUM-1s 和CHEAVD2.0训练数据集上的各种情感类型的样本片段数量的对比. 由表1和表2可知, 在数据平衡之前, 每个类别的样本数量差异较大, 如害怕(Fear)和惊奇(Surprise)类别的语音样本数量, 与其他类别的样本数量相差较大, 在数据平衡之后, 数据集中的每个类别的样本数量达到一种近似的数据平衡状态, 从而能训练出更具鲁棒性的网络模型. 作为举例, 由于BAUM-1s2021 年 第 30 卷 第 5 期计算机系统应用271包含30人, 实验时采用5次交叉验证方法, 即所有数据平均分成5组, 使用其中一组用于测试, 剩下的4组用于训练, 共循环5次, 最后取5次的平均结果作为最终的结果. 因此, 表1只列出了第一次交叉验证时的数据平衡前后结果, 其中测试数据包含随机选择的6人(S03、S19、S20、S24、S28、S29).表1 BAUM-1s 数据平衡前后的训练数据样本片段数量比较数据平衡生气厌恶害怕高兴悲伤惊奇平衡前1442689349148783平衡后314522326405345309表2 CHEAVD2.0数据平衡前后的训练数据样本片段数量比较数据平衡生气焦虑厌恶高兴中性悲伤惊奇担心平衡前309617984702597523324334682345平衡后309633521663259727772433158023451.2 基于CNN 的深度片段特征学习目前, 在计算机视觉领域, 文献[29,30]已经证明在目标数据集上对预先训练好的CNN 模型进行微调(Fine-tuning), 是一种有效的减轻数据不足的方法. 具体的实现方式为首先通过一个大型的数据集来训练网络模型, 由于网络模型的参数初始化对于模型的训练是至关重要的, 因此由大型数据集训练完成保存的模型参数可用于初始化目标任务的模型参数. 如文献[30]首先使用Imagenet 数据集训练网络模型, 完成训练后,将模型的最后一个全连接层之前的所有参数迁移到目标任务中的模型, 以此进行目标任务模型的参数初始化. 为此, 采用在大规模音频数据集上已训练好的VGGish [26]模型在目标语音情感数据集上进行迁移学习.VGGish 是由Google 的语音理解技术团队于2017年3月发布的一个在大规模音频数据集训练得到类似于VGG 的模型, 旨在为音频事件检测提供常见的大规模评估任务. 该模型包括6个卷积层(Conv1,Conv2,···, Conv6)、4个最大池化层(Pool1, Pool2,Pool2, Pool3, Pool4)和4个全连接层(FC1, FC2, FC3,FC4). VGGish 的训练数据来自于包括600多个音频事件类的本体的200万个人标记的10 s YouTube 视频音轨组成的数据集. VGGish 能够从原始的语音信号片段96×64中提取高层次的128-D 大小的特征向量.a i 给定第i 个输入片段数据及其对应的情感类别y i , 对预训练好的VGGish 网络(V )进行微调, 则相当于求解下面的最优化问题:ΥV (a i ;θV )V θV W V 式中, 表示网络的最后一个全连接层的输出特征, 表示网络V 的网络参数, 表示网络V 的Softmax 层的权重参数. 损失函数L 则表示为:y j y V j 式中, 表示第j 个语音片段样本真实的类别号, 表示网络V 的Softmax 层所预测的第j 个样本类别号,t 表示情感类别数目.1.3 基于注意力机制的Bi-LSTM 的情感分类为了获取一句语音的长时间动态信息, 采用基于注意力机制的双向长短时记忆网络(Bi-LSTM)对CNN 在不同语音频谱片段上学习到的128-D 特征序列进行时间上的动态信息建模, 并输出整句语音样本的情感识别结果.(p 1,p 2,···,p T )给定一个时间长度为T 的输入序列(x 1, x 2, ···x T ),Bi-LSTM 旨在通过计算网络节点激活函数的输出,将输入序列(x 1, x 2, ···x T )映射到一个输出序列,如下所示:i t f t c t σt h t x t h t t W αβαβW xi x t i t b ασσ(x )=1/(1+e −x )式中, 、、、、分别是LSTM 模型中的输入门、忘记门、细胞存储单元和输出门的激活输出向量.和分别表示第个时间步长的输入向量和隐层向量.表示和之间的权重矩阵. 例如, 是从输入到输入门的权重矩阵. 是的偏置值, 表示Sigmoid 激活函数.为了采用视觉注意力机制获取需要聚焦的重点语音片段区域, 并抑制其他无用的信息, 拟在 Bi-LSTM 的基础上添加一个注意力层(attention layer). 该注意力层首先采用式(8), 计算出不同时间序列片段特征的权重参数, 然后采用式(9)对不同片段特征进行加权求计算机系统应用2021 年 第 30 卷 第 5 期272µ和, 得到整句语音的特征表示, 紧接着就可以采用2 实验与结果分析为了检验所提出方法的自然语音情感识别性能,采用自然情感语音数据集BAUM-1s[27]和CHEAVD2.0[28]进行自然语音情感识别的实验测试.2.1 数据集BAUM-1s[27]数据集是一个在2016年建立的音视频情感数据集. 它包括1222个视频样本, 来自于30个土耳其人. 实验中拟采用6种基本情感进行测试, 即生气(Anger)、厌恶(Disgust)、害怕(Fear)、高兴(Joy)、悲伤(Sadness)和惊奇(Surprise). 这样, 最终收集到520个情感语音样本用于实验测试.CHEAVD2.0[28]是中科院自动化所在2017年为多模态情感识别竞赛时提供的音视频情感数据集. 它包含8种情感类型: 生气(Anger)、厌恶(Disgust)、害怕(Fear)、高兴(Joy)、悲伤(Sadness)、惊奇(Surprise)、担心(Worry)和焦虑(Anxiety). 该数据集共有7030个样本, 分为3部分: 训练集(4917个样本)、验证集(707个样本)和测试集(1406个样本). 本文使用训练集和验证集来检验所提出方法的自然语音情感识别性能, 因为测试集只对参加竞赛者开放和获取.2.2 结果分析表3和表4分别列出了本文方法采用数据平衡前后的试验结果的比较. 从表3和表4可见:(1)不管是否采用数据平衡策略, 本文采用的带注意力机制方法(VGGish+LSTM+Attention)取得的语音情感识别性能, 都要优于未带注意力机制方法(VGGish+ LSTM). 这表明采用注意力机制能够改善LSTM的特征表征能力, 因为注意力机制能够聚焦于一句语音中对情感识别起作用的目标片段区域的特征学习.(2)做数据平衡之后, 所有方法取得的识别性能都要明显高于数据平衡之前的结果. 在数据平衡之前, 本文方法(VGGish+LSTM+Attention)在BAUM-1s和CHEAVD2.0数据集分别取得了44.09%和41.73%. 然而, 在数据平衡之后, 本文方法则分别在BAUM-1s和CHEAVD2.0数据集上提高了3.18%和2.05%. 这说明采用数据平衡策略能够进一步改善深度学习方法的特征学习能力.表3 数据平衡之前的语音情感识别性能(%)方法BAUM-1s CHEAVD2.0 VGGish+LSTM43.1439.89 VGGish+LSTM+注意力44.0941.37表4 数据平衡之后的语音情感识别性能(%)方法BAUM-1s CHEAVD2.0 VGGish+LSTM45.7842.29 VGGish+LSTM+注意力47.2743.42为了进一步给出每种情感类型的具体正确识别率,图4和图5分别列出了本文方法(VGGish+LSTM+ Attention)在数据平衡之后的两个数据集上识别结果的模糊矩阵. 由图4可知, 在数据平衡之后的BAUM-1s数据集上, 高兴(Joy)和悲伤(Sadness)的正确识别率分别达到了64.53%和61.19%, 而其它4种情感类别的识别性能较差, 不足40%. 由图5可知, 在数据平衡之后的CHEAVD2.0数据集上, 生气(Anger)和中性(Neutral)的正确识别率分别达到了65.62%和64.5%,而其它6种情感类型的正确识别率都不足45%. 主要原因可能是这些情感类型学习到的特征表征区分度不高, 导致它们相互之间容易混淆.35.717.505.4111.0510.454.8810.7125.008.116.986.724.881.792.5016.221.742.992.4421.4330.0035.1464.5314.1839.0230.3628.7535.1410.4761.1926.830.006.250.005.234.4821.95预测标签生气生气厌恶害怕高兴悲伤惊奇605040302010图4 本文方法在数据平衡之后的BAUM-1s数据集上识别结果的模糊矩阵为了进一步说明本文方法的优越性, 表5列出了本文方法取得的识别结果与其它现有文献报道结果之间的比较. 从表5可知, 本文方法不仅优于一些采用手工特征的方法, 如MFCC[27], GeMAPS ADDIN EN.2021 年 第 30 卷 第 5 期计算机系统应用273CITE.DATA [28– 31], 也优于一些基于深度卷积神经网络CNN 的方法, 如采用AlexNet 的跨模态迁移方法[22,32]. 以及与含有20层的ResNet 网络的自训练网络方法[33]获得的准确率相近.68.7542.4242.8621.8516.0013.4340.0018.523.1216.670.001.681.001.494.006.170.001.520.000.841.000.04.001.23 3.9112.1223.8136.1311.504.4812.0013.5811.7212.1223.8126.8956.0022.3920.0022.227.811.520.003.362.0043.280.007.41 1.561.524.762.521.502.998.03.703.1212.124.766.7211.0011.9412.027.16生气生气焦虑厌恶高兴中性悲伤惊奇担心厌恶焦虑高兴中性担心悲伤惊奇预测标签6050403020100图5 本文方法在数据平衡之后的CHEAVD2.0数据集上识别结果的模糊矩阵表5 与现有文献报道的结果比较数据集文献特征识别率(%)BAUM-1sZhalehpour S [27]MFCC 29.41Zhang S [22]CNN42.46Ma Y [32]CNN42.38本文方法CNN+LSTM+注意力+数据平衡47.27CHEAVD2.0Li Y [28]GeMAPS 39.90Xi YX[33]CNN(ResNet)43.96Miao HT[31]GeMAPS40.31本文方法CNN+LSTM+注意力+数据平衡43.423 结论与展望本文提出了一种结合数据平衡和注意力机制的卷积神经网络(CNN)和长短时记忆单元(LSTM)的语音情感识别方法, 采用数据平衡方法对语音情感数据集中每个类别的样本进行相应的预处理, 再采用预训练好的VGGish 网络在目标数据集上进行微调, 从而学习出分段语音特征, 再通过带有注意力机制的LSTM 从分段语音特征中学习对应的高阶判别性特征, 最后通过Softmax 层进行情感分类. 本文在两个自然语音情感数据集BAUM-1s 和CHEAVD2.0中的实验证明了提出的语音情感识别方法有较好的识别性能. 由于本文提出的网络模型的训练方式并不是端到端的训练方式, 因此, 在未来的工作中, 希望能以端到端的方式训练网络模型; 由于高阶注意力生成的全局性特征更具表达力, 并能辅助网络模型对语音信息长范围的相关性进行建模, 因此, 在未来的工作中, 希望能将二阶注意力机制或更高阶的注意力机制与现有的网络模型进行结合, 以学习出更具判别性的语音情感特征.参考文献韩文静, 李海峰, 阮华斌, 等. 语音情感识别研究进展综述. 软件学报, 2014, 25(1): 37–50. [doi: 10.13328/ki.jos.004497]1张石清, 李乐民, 赵知劲. 人机交互中的语音情感识别研究进展. 电路与系统学报, 2013, 18(2): 440–451, 434.2Schuller BW. Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends. Communications of the ACM, 2018, 61(5): 90–99. [doi: 10.1145/3129340]3Akçay MB, Oğuz K. Speech emotion recognition: Emotional models, databases, features, preprocessing methods,supporting modalities, and classifiers. Speech Communication, 2020, 116: 56–76. [doi: 10.1016/j.specom.2019.12.001]4Song P. Transfer linear subspace learning for cross-corpusspeech emotion recognition. IEEE Transactions on Affective Computing, 2019, 10(2): 265–275. [doi: 10.1109/TAFFC.2017.2705696]5Demircan S, Kahramanli H. Application of fuzzy C-means clustering algorithm to spectral features for emotion classification from speech. Neural Computing and Applications, 2018, 29(8): 59–66. [doi: 10.1007/s00521-016-2712-y ]6Zhao XM, Zhang SQ. Spoken emotion recognition via locality-constrained kernel sparse representation. Neural Computing and Applications, 2015, 26(3): 735–744. [doi: 10.1007/s00521-014-1755-1]7Gharavian D, Sheikhan M, Nazerieh A, et al . Speech emotion recognition using FCBF feature selection method and GA-optimized fuzzy ARTMAP neural network. Neural Computing and Applications, 2012, 21(8): 2115–2126.8Zhang ZX, Coutinho E, Deng J, et al . Cooperative learning and its application to emotion recognition from speech.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2015, 23(1): 115–126.9朱菊霞, 吴小培, 吕钊. 基于SVM 的语音情感识别算法. 计算机系统应用, 2011, 20(5): 87–91. [doi: 10.3969/j.issn.1003-3254.2011.05.019]10Kayaoglu M, Eroglu Erdem C. Affect recognition using key frame selection based on minimum sparse reconstruction.Proceedings of the 2015 ACM on International Conference11计算机系统应用2021 年 第 30 卷 第 5 期274on Multimodal Interaction. Seattle, WA, USA. 2015.519–524.Schuller B, Steidl S, Batliner A, et al . The INTERSPEECH 2013 computational paralinguistics challenge: Social signals,conflict, emotion, autism. INTERSPEECH 2013: 14th Annual Conference of the International Speech Communication Association. Lyon, France. 2013. 148–152.12Valstar M, Schuller B, Smith K, et al . AVEC 2013: The continuous audio/visual emotion and depression recognition challenge. Proceedings of the 3rd ACM International Workshop on Audio/Visual Emotion Challenge. Barcelona,Spain. 2013. 3–10.13Eyben F, Scherer KR, Schuller BW, et al . The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing. IEEE Transactions on Affective Computing, 2016, 7(2): 190–202. [doi: 10.1109/TAFFC.2015.2457417]14Hinton GE, Salakhutdinov RR. Reducing the dimensionality of data with neural networks. Science, 2006, 313(5786):504–507. [doi: 10.1126/science.1127647]15Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks.Proceedings of the 25th International Conference on Neural Information Processing Systems. Siem Reap, Cambodia.2012. 1097–1105.16Hochreiter S, Schmidhuber J. Long short-term memory.Neural Computation, 1997, 9(8): 1735–1780. [doi: 10.1162/neco.1997.9.8.1735]17Han K, Yu D, Tashev I. Speech emotion recognition using deep neural network and extreme learning machine. 15th Annual Conference of the International Speech Communication Association. Singapore. 2014. 223–227.18Mao QR, Dong M, Huang ZW, et al . Learning salient features for speech emotion recognition using convolutional neural networks. IEEE Transactions on Multimedia, 2014,16(8): 2203–2213. [doi: 10.1109/TMM.2014.2360798]19Trigeorgis G, Ringeval F, Brueckner R, et al . Adieu features?End-to-end speech emotion recognition using a deep convolutional recurrent network. 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Shanghai, China. 2016. 5200–5204.20Huang CW, Narayanan SS. Deep convolutional recurrent neural network with attention mechanism for robust speech emotion recognition. 2017 IEEE International Conference on Multimedia and Expo (ICME). Hong Kong, China. 2017.583–588.21Zhang SQ, Zhang SL, Huang TJ, et al . Learning affective features with a hybrid deep model for audio-visual emotion22recognition. IEEE Transactions on Circuits and Systems for Video Technology, 2018, 28(10): 3030–3043. [doi: 10.1109/TCSVT.2017.2719043]Zhang SQ, Zhao XM, Tian Q. Spontaneous speech emotion recognition using multiscale deep convolutional LSTM.IEEE Transactions on Affective Computing, 2019. [doi:10.1109/TAFFC.2019.2947464]23黎万义, 王鹏, 乔红. 引入视觉注意机制的目标跟踪方法综述. 自动化学报, 2014, 40(4): 561–576.24孙小婉, 王英, 王鑫, 等. 面向双注意力网络的特定方面情感分析模型. 计算机研究与发展, 2019, 56(11): 2384–2395.[doi: 10.7544/issn1000-1239.2019.20180823]25Hershey S, Chaudhuri S, Ellis DP, et al . CNN architectures for large-scale audio classification. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). New Orleans, LA, USA. 2017. 131–135.26Zhalehpour S, Onder O, Akhtar Z, et al . BAUM-1: A spontaneous audio-visual face database of affective and mental states. IEEE Transactions on Affective Computing,2017, 8(3): 300–313. [doi: 10.1109/TAFFC.2016.2553038]27Li Y, Tao JH, Schuller B, et al . MEC 2017: Multimodal emotion recognition challenge. 2018 First Asian Conference on Affective Computing and Intelligent Interaction (AC ⅡAsia). Beijing, China. 2018. 1–5.28Campos V, Jou B, Giró-i-Nieto X. From pixels to sentiment:Fine-tuning CNNs for visual sentiment prediction. Image and Vision Computing, 2017, 65: 15–22. [doi: 10.1016/j.imavis.2017.01.011]29Oquab M, Bottou L, Laptev I, et al . Learning and transferring mid-level image representations using convolutional neural networks. Proceedings of the 2014IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH, USA. 2014. 1717–1724.30Miao HT, Zhang YF, Li WP, et al . Chinese multimodal emotion recognition in deep and traditional machine leaming approaches. 2018 First Asian Conference on Affective Computing and Intelligent Interaction (AC Ⅱ Asia). Beijing,China. 2018. 1–6.31Ma YX, Hao YX, Chen M, et al . Audio-visual emotion fusion (AVEF): A deep efficient weighted rmation Fusion, 2019, 46: 184–192. [doi: 10.1016/j.inffus.2018.06.003]32Xi YX, Li PC, Song Y, et al . Speaker to emotion: Domain adaptation for speech emotion recognition with residual adapters. 2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). Lanzhou, China. 2019. 513–518.332021 年 第 30 卷 第 5 期计算机系统应用275。

Fast Compressive TrackingKaihua Zhang,Lei Zhang,Member,IEEE,and Ming-Hsuan Yang,Senior Member,IEEE Abstract—It is a challenging task to develop effective and efficient appearance models for robust object tracking due to factors such as pose variation,illumination change,occlusion,and motion blur.Existing online tracking algorithms often update models with samples from observations in recent frames.Despite much success has been demonstrated,numerous issues remain to be addressed.First, while these adaptive appearance models are data-dependent,there does not exist sufficient amount of data for online algorithms to learn at the outset.Second,online tracking algorithms often encounter the drift problems.As a result of self-taught learning,misaligned samples are likely to be added and degrade the appearance models.In this paper,we propose a simple yet effective and efficient tracking algorithm with an appearance model based on features extracted from a multiscale image feature space with data-independent basis.The proposed appearance model employs non-adaptive random projections that preserve the structure of the image feature space of objects.A very sparse measurement matrix is constructed to efficiently extract the features for the appearance model.We compress sample images of the foreground target and the background using the same sparse measurement matrix.The tracking task is formulated as a binary classification via a naive Bayes classifier with online update in the compressed domain.A coarse-to-fine search strategy is adopted to further reduce the computational complexity in the detection procedure.The proposed compressive tracking algorithm runs in real-time and performs favorably against state-of-the-art methods on challenging sequences in terms of efficiency,accuracy and robustness.Index Terms—Visual tracking,random projection,compressive sensingÇ1I NTRODUCTIOND ESPITE that numerous algorithms have been proposedin the literature,object tracking remains a challenging problem due to appearance change caused by pose,illumi-nation,occlusion,and motion,among others.An effective appearance model is of prime importance for the success of a tracking algorithm that has attracted much attention in recent years[2],[3],[4],[5],[6],[7],[8],[9],[10],[11],[12], [13],[14],[15],[16].Numerous effective representation schemes have been proposed for robust object tracking in recent years.One commonly adopted approach is to learn a low-dimensional subspace(e.g.,eigenspace[7],[17]),which can adapt online to object appearance change.Since this approach is data-dependent,the computational complexity is likely to increase significantly because it needs eigen-decomposi-tions.Moreover,the noisy or misaligned samples are likely to degrade the subspace basis,thereby causing these algo-rithms to drift away the target objects gradually.Another successful approach is to extract discriminative features from a high-dimensional space.Since object tracking can be posed as a binary classification task which separates object from its local background,a discriminative appearance model plays an important role for its success.Online boost-ing methods[6],[10]have been proposed to extract discrim-inative features for object tracking.Alternatively,high-dimensional features can be projected to a low-dimensional space from which a classifier can be constructed.The compressive sensing(CS)theory[18],[19]shows that if the dimension of the feature space is sufficiently high, these features can be projected to a randomly chosen low-dimensional space which contains enough information to reconstruct the original high-dimensional features.The dimensionality reduction method via random projection (RP)[20],[21]is data-independent,non-adaptive and infor-mation-preserving.In this paper,we propose an effective and efficient tracking algorithm with an appearance model based on features extracted in the compressed domain[1]. The main components of the proposed compressive track-ing algorithm are shown by Fig.1.We use a very sparse measurement matrix that asymptotically satisfies the restricted isometry property(RIP)in compressive sensing theory[18],thereby facilitating efficient projection from the image feature space to a low-dimensional compressed sub-space.For tracking,the positive and negative samples are projected(i.e.,compressed)with the same sparse measure-ment matrix and discriminated by a simple naive Bayes classifier learned online.The proposed compressive track-ing algorithm runs at real-time and performs favorably against state-of-the-art trackers on challenging sequences in terms of efficiency,accuracy and robustness.The rest of this paper is organized as follows.Wefirst review the most relevant work on online object tracking in Section2.The preliminaries of compressive sensing and random projection are introduced in Section3.The proposed algorithm is detailed in Section4,and the experimental results are presented in Section5withK.Zhang is with the School of Information and Control,NanjingUniversity of Information Science&Technology,Nanjing,China.E-mail:zhkhua@.L.Zhang is with the Department of Computing,the Hong Kong PolytechnicUniversity,Hong Kong.E-mail:cslzhang@.hk.M.-H.Yang is with the Department of Electrical Engineering and Com-puter Science(EECS),School of Engineering,University of California atMerced,5200North Lake Road,Merced,CA95344.E-mail:mhyang@.Manuscript received1Feb.2013;revised11Jan.2014;accepted16Mar.2014.Date of publication6Apr.2014;date of current version10Sept.2014.Recommended for acceptance by S.Avidan.For information on obtaining reprints of this article,please send e-mail to:reprints@,and reference the Digital Object Identifier below.Digital Object Identifier no.10.1109/TPAMI.2014.23158080162-8828ß2014IEEE.Personal use is permitted,but republication/redistribution requires IEEE permission.See /publications_standards/publications/rights/index.html for more information.comparisons to state-of-the-art methods on challenging sequences.We conclude with remarks on our future work in Section6.2R ELATED W ORKRecent surveys of object tracking can be found in[22],[23], [24].In this section,we briefly review the most relevant liter-ature of online object tracking.In general,tracking algo-rithms can be categorized as either generative[2],[3][7],[9], [11],[12],[25],[26],[27],[28],[29]or discriminative[4],[5], [6],[8],[10],[30],[13],[16]based on their appearance models.Generative tracking algorithms typically learn a model to represent the target object and then use it to search for the image region with minimal reconstruction error.Black and Jepson[2]learn an offline subspace model to represent the object of interest for tracking.Reference templates based on color histogram[31],[32],integral histogram[25]have been used for tracking.In[3]Jepson et al.present a Gaussian mix-ture model with an online expectation maximization algo-rithm to handle object appearance variations during tracking.Ho et al.[17]propose a tracking method using a set of learned subspace model to deal with appearance change. Instead of using pre-trained subspace,the IVT method[7] learns an appearance model online to adapt appearance change.Kwon and Lee[9]combine multiple observation and motion models in a modified particlefiltering frame-work to handle large appearance and motion variation. Recently,sparse representation has been used in the ‘1-tracker where an object is modeled by a sparse linear com-bination of target and trivial templates[12].However,the computational complexity of the‘1-tracker is rather high, thereby limiting its applications in real-time scenarios.Li et al.[11]further extend it by using the orthogonal matching pursuit algorithm for solving the optimization problems effi-ciently,and Bao et al.[27]improve the efficiency via acceler-ated proximal gradient approach.A representation based on distribution of pixels at multiple layers is proposed to describe object appearance for tracking[29].Oron et al.[28] propose a joint model of appearance and spatial configura-tion of pixels which estimates the amount of local distortion of the target object,thereby well handling rigid and nonrigid deformations.Recently,Zhang et al.[26]propose a multi-task approach to jointly learn the particle representations for robust object tracking.Despite much demonstrated success of these online generative tracking algorithms,several prob-lems remain to be solved.First,numerous training samples cropped from consecutive frames are required in order to learn an appearance model online.Since there are only a few samples at the outset,most tracking algorithms often assume that the target appearance does not change much during this period.However,if the appearance of the target changes significantly,the drift problem is likely to occur. Second,these generative algorithms do not use the back-ground information which is likely to improve tracking sta-bility and accuracy.Discriminative algorithms pose the tracking problem as a binary classification task with local search and determine the decision boundary for separating the target object from the background.Avidan[4]extends the opticalflow approach with a support vector machine(SVM)classifier for object tracking,and Collins et al.[5]demonstrate that the most dis-criminative features can be learned online to separate the tar-get object from the background.In[6]Grabner et al.propose an online boosting algorithm to select features for tracking. However,these trackers[4],[5],[6]use one positive sample (i.e.,the current tracker location)and a few negative samples when updating the classifier.As the appearance model is updated with noisy and potentially misaligned examples, this often leads to the tracking drift problem.An online semi-supervised boosting method is proposed by Grabner et al.[8]to alleviate the drift problem in which only the samples in thefirst frame are labeled and all the other samples are unlabeled.Babenko et al.[10]formulate online tracking within the multiple instance learning framework where sam-ples are considered within positive and negative bags orsets. Fig.1.Main components of the proposed compressive tracking algorithm.A semi-supervised learning approach[33]is developed in which positive and negative samples are selected via an online classifier with structural constraints.Wang et al.[30] present a discriminative appearance model based on super-pixels which is able to handle heavy occlusions and recovery from drift.In[13],Hare et e an online structured output support vector machine for robust tracking which can mitigate the effect of wrong labeling samples.Recently, Henriques et al.[16]introduce a fast tracking algorithm which exploits the circulant structure of the kernel matrix in SVM classifier that can be efficiently computed by the fast Fourier transform algorithm.3P RELIMINARIESWe present some preliminaries of compressive sensing which are used in the proposed tracking algorithm.3.1Random Projection and Compressive Sensing In random projection,a random matrix R2R nÂm whose rows have unit length projects data from the high-dimensional feature space x2R m to a lower-dimensional space v2R nv¼Rx;(1) where n(m.Each projection v is essentially equivalent to a compressive measurement in the compressive sensing encoding stage.The compressive sensing theory[19],[34] states that if a signal is K-sparse(i.e.,the signal is a linear combination of only K basis[35]),it is possible to near per-fectly reconstruct the signal from a small number of random measurements.The encoder in compressive sensing(using (1))correlates signal with noise(using random matrix R) [19],thereby it is a universal encoding which requires no prior knowledge of the signal structure.In this paper,we adopt this encoder to construct the appearance model for visual tracking.Ideally,we expect R provides a stable embedding that approximately preserves the salient information in any K-sparse signal when projecting from x2R m to v2R n.A necessary and sufficient condition for this stable embedding is that it approximately preserves distances between any pairs of K-sparse signals that share the same K basis.That is,for any two K-sparse vectors x1and x2sharing the same K basis,ð1À Þk x1Àx2k2‘2k Rx1ÀRx2k2‘2ð1þ Þk x1Àx2k2‘2:(2)The restricted isometry property[18],[19]in compressive sensing shows the above results.This property is achieved with high probability for some types of random matrix R whose entries are identically and independently sampled from a standard normal distribution,symmetric Bernoulli distribution or Fourier matrix.Furthermore,the above result can be directly obtained from the Johnson-Linden-strauss(JL)lemma[20].Lemma1.(Johnson-Lindenstrauss lemma)[20]:Let Q be afinite collection of d points in R m.Given0< <1and b>0,let n be a positive integer such thatn!4þ2b2=2À 3=3lnðdÞ:(3) Let R2R nÂm be a random matrix with Rði;jÞ¼r ij,wherer ij¼þ1;with probability12;À1;with probability12;&(4)orr ij¼ffiffiffi3pÂþ1;with probability16;0;with probability23;À1;with probability16:8<:(5)Then,with probability exceeding1ÀdÀb,the following state-ment holds:For every x1;x22Q,ð1À Þk x1Àx2k2‘21ffiffiffinp k Rx1ÀRx2k2‘2ð1þ Þk x1Àx2k2‘2:(6)Baraniuk et al.[36]prove that any random matrix sat-isfying the Johnson-Lindenstrauss lemma also holds true for the restricted isometry property in compressive sens-ing.Therefore,if the random matrix R in(1)satisfies the JL lemma,x can be reconstructed with minimum error from v with high probability if x is K-sparse(e.g.,audio or image signals).This strong theoretical support moti-vates us to analyze the high-dimensional signals via their low-dimensional random projections.In the proposed algorithm,a very sparse matrix is adopted that not only asymptotically satisfies the JL lemma,but also can be effi-ciently computed for real-time tracking.3.2Very Sparse Random Measurement MatrixA typical measurement matrix satisfying the restricted isom-etry property is the random Gaussian matrix R2R nÂm where r ij$Nð0;1Þ(i.e.,zero mean and unit variance),as used in recent work[11],[37],[38].However,as the matrix is dense,the memory and computational loads are very expen-sive when m is large.In this paper,we adopt a very sparse random measurement matrix with entries defined asr ij¼ffiffiffirpÂ1;with probability12r;0;with probability1À1r;À1;with probability12r:8><>:(7)Achlioptas[20]proves that this type of matrix with r¼1or 3satisfies the Johnson-Lindenstrauss lemma(i.e.,(4)and (5)).This matrix is easy to compute which requires only a uniform random generator.More importantly,when r¼3, it is sparse where two thirds of the computation can be avoided.In addition,Li et al.[39]show that for r¼oðmÞ(x2R m),the random projections are almost as accurate as the conventional random projections where r ij$Nð0;1Þ. Therefore,the random matrix(7)with r¼oðmÞasymptoti-cally satisfies the JL lemma.In this work,we set r¼oðmÞ¼m=ða log10ðmÞÞ¼m=ð10aÞ$m=ð6aÞwith afixed constant a because the dimensionality m is typically in the order of106 to1010.For each row of R,only about c¼ð12rþ12rÞÂm¼a log10ðmÞ10a nonzero entries need to be computed.We observe that good results can be obtained by fixing a ¼0:4in our experiments.Therefore,the computational complexity is only o ðcn Þ(n ¼100in this work)which is very low.Furthermore,only the nonzero entries of R need to be stored which makes the memory requirement also very light.4P ROPOSED A LGORITHMIn this section,we present the proposed compressive track-ing algorithm in details.The tracking problem is formulated as a detection task and the main steps of the proposed algo-rithm are shown in Fig.1.We assume that the tracking win-dow in the first frame is given by a detector or manual label.At each frame,we sample some positive samples near the current target location and negative samples away from the object center to update the classifier.To predict the object location in the next frame,we draw some samples around the current target location and determine the one with the maximal classification score.4.1Image RepresentationTo account for large scale change of object appearance,a multiscale image representation is often formed by convolv-ing the input image with a Gaussian filter of different spa-tial variances [40].The Gaussian filters in practice have to be truncated which can be replaced by rectangle filters.Bay et al.[41]show that this replacement does not affect the per-formance of the interest point detectors but can significantly speed up the detectors via integral image method [42].For each sample Z 2R w Âh ,its multiscale representation (as illustrated in Fig.2)is constructed by convolving Z with a set of rectangle filters at multiple scales f F 1;1;...;F w;h g defined byF w;h ðx;y Þ¼1wh Â1;1 x w ,1 y h;0;otherwise ;&(8)where w and h are the width and height of a rectangle fil-ter,respectively.Then,we represent each filtered image as a column vector in R wh and concatenate these vectors as a very high-dimensional multiscale image feature vector x ¼ðx 1;...;x m Þ>2R m where m ¼ðwh Þ2.The dimensionality m is typically in the order of 10to 1010.We adopt asparse random matrix R in (7)to project x onto a vector v 2R n in a low-dimensional space.The random matrix R needs to be computed only once offline and remains fixed throughout the tracking process.For the sparse matrix R in (7),the computational load is very light.As shown in Fig.3,we only need to store the nonzero entries in R and the positions of rectangle filters in an input image corre-sponding to the nonzero entries in each row of R .Then,v can be efficiently computed by using R to sparsely mea-sure the rectangular features which can be efficiently computed using the integral image method [42].4.2Analysis of Compressive Features 4.2.1Relationship to the Haar-Like FeaturesAs shown in Fig.3,each element v i in the low-dimensional feature v 2R n is a linear combination of spatially distrib-uted rectangle features at different scales.Since the coeffi-cients in the measurement matrix can be positive or negative (via (7)),the compressive features compute the rel-ative intensity difference in a way similar to the generalized Haar-like features [10](See Fig.3).The Haar-like features have been widely used for object detection with demon-strated success [10],[42],[43].The basic types of these Haar-like features are typically designed for different tasks [42],[43].There often exist a very large number of Haar-like fea-tures which make the computational load very heavy.This problem is alleviated by boosting algorithms for selecting important features [42],[43].Recently,Babenko et al.[10]adopt the generalized Haar-like features where each one is a linear combination of randomly generated rectangle fea-tures,and use online boosting to select a small set of them for object tracking.In this work,the large set of Haar-like features are compressively sensed with a very sparse mea-surement matrix.The compressive sensing theories ensure that the extracted features of our algorithm preserve almost all the information of the original image,and hence is able to correctly classify any test image because the dimension of the feature space is sufficiently large (106to 1010)[37].Therefore,the projected features can be classified in the compressed domain efficiently and effectively without the curse of dimensionality.4.2.2Scale Invariant PropertyIt is easy to show that the low-dimensional feature v is scale invariant.As shown in Fig.3,each feature in v is a linear combination of some rectangle filters convolving the input image at different positions.Therefore,withoutFig.2.Illustration of multiscale imagerepresentation.Fig.3.Graphical representation of compressing a high-dimensional vec-tor x to a low-dimensional vector v .In the matrix R ,dark,gray and white rectangles represent negative,positive,and zero entries,respectively.The blue arrows illustrate that one of nonzero entries of one row of R sensing an element in x is equivalent to a rectangle filter convolving the intensity at a fixed position of an input image.loss of generality,we only need to show that the j th rect-angle feature x j in the i th feature v i in v is scale invariant.From Fig.4,we havex j ðs y Þ¼F sw j ;sh j ðs y Þ Z ðs y Þ¼F sw j ;sh j ða Þ Z ða Þj a ¼s y¼1s 2w i h i Z u 2V sZ ða Àu Þd u ¼1s 2w i h i Z u 2VZ ðy Àu Þj s 2j d u ¼1w i h i Z u 2VZ ðy Àu Þd u ¼F w j ;h j ðy Þ Z ðy Þ¼x j ðy Þ;where V ¼fðu 1;u 2Þj 1 u 1 w i ;1 u 2 h i g and V s ¼fðu 1;u 2Þj 1 u 1 sw i ;1 u 2 sh i g .4.3Classifier Construction and UpdateWe assume all elements in v are independently distributed and model them with a naive Bayes classifier [44],H ðv Þ¼log Q n i ¼1p ðv i j y ¼1Þp ðy ¼1ÞQ ni ¼1p ðv i j y ¼0Þp ðy ¼0Þ¼X n i ¼1log p ðv i j y ¼1Þp ðv i j y ¼0Þ ;(9)where we assume uniform prior,p ðy ¼1Þ¼p ðy ¼0Þ,andy 2f 0;1g is a binary variable which represents the sample label.Diaconis and Freedman [45]show that random projec-tions of high dimensional random vectors are almost always Gaussian.Thus,the conditional distributions p ðv i j y ¼1Þand p ðv i j y ¼0Þin the classifier H ðv Þare assumed to beGaussian distributed with four parameters ðm 1i ;s 1i ;m 0i ;s 0i Þ,p ðv i j y ¼1Þ$N Àm 1i ;s 1iÁ;p ðv i j y ¼0Þ$N Àm 0i ;s 0i Á;(10)where m 1i (m 0i )and s 1i (s 0i )are mean and standard deviation of the positive (negative)class.The scalar parameters in (10)are incrementally updated bym 1i m 1i þð1À Þm1s 1i ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi Às 1i Á2þð1À Þðs 1Þ2þ ð1À ÞÀm 1i Àm 1Á2q ;(11)where >0is a learning parameter,s 1¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi1n P n À1k ¼0j y ¼1ðv i ðk ÞÀm 1Þ2q and m 1¼1n P n À1k ¼0j y ¼1v i ðk Þ.Parameters m 0i and s 0i are updated with similar rules.The above equations can be easily derived by maximum likelihood estimation [46].Fig.5shows the probability distributions for three different fea-tures of the positive and negative samples cropped from a few frames of a sequence for clarity of presentation.It shows that a Gaussian distribution with online update using (11)is a good approximation of the features in the projected space where samples can be easily separated.Because the variables are assumed to be independent in our classifier,the n -dimensional multivariate problem is reduced to the n univariate estimation problem.Thus,it requires fewer training samples to obtain accurate estima-tion than estimating the covariance matrix in the multivari-ate estimation.Furthermore,several densely sampled positive samples surrounding the current tracking result are used to update the distribution parameters,which is able to obtain robust estimation even when the tracking result has some drift.In addition,the useful information from the former accurate samples is also used to update the parameter distributions,thereby facilitating the proposed algorithm to be robust to misaligned samples.Thus,our classifier performs robustly even when the misaligned or the insufficient number of training samples are used.4.4Fast Compressive TrackingThe aforementioned classifier is used for local search.To reduce the computational complexity,a coarse-to-fine slid-ing window search strategy is adopted (See Fig.6).The main steps of our algorithm are summarized in Algorithm 1.First,we search the object location based on the previous object location by shifting the window with a large numberofFig.4.Illustration of scale invariant property of low-dimensional features.From the left figure to the right one,the ratio is s .Red rectangle repre-sents the j th rectangle feature at position y.Fig.5.Probability distributions of three different features in a low-dimen-sional space.The red stair represents the histogram of positive samples while the blue one represents the histogram of negative samples.The red and blue lines denote the corresponding estimated distributions by the proposed incremental updatemethod.Fig.6.Coarse-to-fine search for new object location.Left:object center location (denoted by red solid circle)at the t th frame.Middle:coarse-grained search with a large radius g c and search step D c based on the previous object location.Right:fine-grained search with a small radius g f <g c and search step D f <D c based on the coarse-grained search location (denoted by green solid circle).The final object location is denoted by blue solid circle.pixels D c within a large search radius g c.This generates fewer windows than locally exhaustive search method(e.g., [10])but the detected object location may be slightly inaccu-rate but close to the accurate object location.Based on the coarse-grained detected location,fine-grained search is car-ried out with a small number of pixels D f within a small search radius g f.For example,we set g c¼25,D c¼4,and g f¼10,D f¼1in all the experiments.If we use thefine-grained locally exhaustive method with g c¼25and D f¼1, the total number of search windows is about1,962(i.e.,pg2c). However,using this coarse-to-fine search strategy,the total number of search windows is about436(i.e.,pg2c=16þpg2f), thereby significantly reducing computationalcost.4.4.1Multiscale Fast Compressive TrackingAt each location in the search region,three image patches are cropped at different scale s:current(s¼1),small (s¼1Àd)and large scale(s¼1þd),to account for appear-ance variation due to fast scale change.The template of each rectangle feature for patch with scale s is multiplied by ratio s(See Fig.4).Therefore,the feature v s for each patch with scale s can be efficiently extracted by using the integral image method[42].Since the low-dimensional features for each image patch are scale invariant,we have v s t¼arg max v2F HðvÞ%v tÀ1,where v tÀ1is the low-dimensional feature vector that represents the object in the (tÀ1)th frame,and F is the set of low-dimensional features extracted from image patches at different scales.The classi-fier is updated with cropped positive and negative samples based on the new object location and scale.The above pro-cedures can be easily integrated into Algorithm1:the scale is updated everyfifth frame in thefine-grained search pro-cedure(See Step4in Algorithm1),which is a tradeoff between computational efficiency and effectiveness of han-dling appearance change caused by fast scale change.4.5DiscussionWe note that simplicity is the prime characteristic of the proposed algorithm in which the proposed sparse measure-ment matrix R is independent of training samples,thereby resulting in an efficient method.In addition,the proposed algorithm achieves robust performance as discussed below.Difference with related work.It should be noted that the proposed algorithm is different from recent work based on sparse representation[12]and compressive sensing [11].First,both algorithms are generative models that encode an object sample by sparse representation of tem-plates using‘1-minimization.Thus the training samples cropped from the previous frames are stored and updated,but this is not required in the proposed algo-rithm due to the use of a data-independent measurement matrix.Second,the proposed algorithm extracts a linear combination of generalized Haar-like features and other trackers[12],[11]use sparse representations of holistic templates which are less robust as demonstrated in the experiments.Third,both tracking algorithms[12],[11] need to solve numerous time-consuming‘1-minimization problems although one method has been recently pro-posed to alleviate the problem of high computational complexity[27].In contrast,the proposed algorithm is efficient as only matrix multiplications are required.The proposed method is different from the MIL tracker [10]as itfirst constructs a feature pool in which each feature is randomly generated as a weighted sum of pixels in two to four rectangles.A subset of most discriminative features are then selected via an MIL boosting method to construct thefinal strong classifier.However,as the adopted mea-surement matrix of the proposed algorithm satisfies the JL lemma,the compressive features can preserve the‘2dis-tance of the original high-dimensional features.Since each feature that represents a target or background sample is assumed to be independently distributed with a Gaussian distribution,the feature vector for each sample is modeled as a mixture of Gaussian(MoG)distribution.The MoG dis-tribution is essentially a mixture of weighted‘2distances of Gaussian distributions.Thus,the‘2distance between the target and background distributions is preserved in the compressive feature space,and the proposed algorithm can obtain favorable results without further learning the dis-criminative features from the compressive feature space.Discussion with the online AdaBoost method(OAB)[6].The reasons that our method performs better than the OAB method can be attributed to the following factors.First,to reduce the computational complexity,the feature pool size designed by the OAB method is small(less than250accord-ing to the default setting in[6]which may contain insuffi-cient discriminative features.However,our compressive features can preserve the intrinsic discriminative strength of the original high-dimensional multiscale features,i.e.,large (between106and1010)feature space.Therefore,our com-pressive features have better discriminative capability than the Haar-like features used by the OAB method.Second, the proposed method uses several positive samples(patches close to the tracking result at any frame)for online update of the appearance model which alleviates the errors intro-duced by inaccurate tracking locations,whereas the OAB method only uses one positive sample(i.e.,the tracking。

Computer Science and Application 计算机科学与应用, 2023, 13(7), 1473-1484 Published Online July 2023 in Hans. https:///journal/csa https:///10.12677/csa.2023.137146基于孪生网络的心电信号智能诊断模型马 帅，刘元昊延安大学数学与计算机科学学院，陕西 延安收稿日期：2023年6月20日；录用日期：2023年7月20日；发布日期：2023年7月28日摘要在构建医学心电信号分类模型时，现有数据库中正样本和负样本存在类别不平衡的问题，即正常心电数据往往多于异常心电数据，从而导致所构建的模型会产生一定的偏差，影响模型最终的分类结果。

利用现有12导联心电诊断数据集构建心电信号分类模型，结合对比学习的方法，在传统网络基础上利用孪生网络结构对模型进行预训练，每条心电信号分别与其对应的正例和反例进行匹配得到样本对，并将孪生网络提取的判别特征用于下游任务进行分类。

实验表明，基于对比学习预训练范式能够更好地提取心电信号特征，对于克服心电数据不平衡问题有显著效果，该方法可有效地利用了大量正样本，因此基于孪生网络结构的预训练模型所提取的特征判别性能优于传统特征提取模型。

测试集上的准确率达到了96.55%，假阳性率6.91%，漏诊率仅2%。

关键词心电信号分类，孪生网络，对比损失，1D-CNN ，LSTMIntelligent Diagnosis Model for ECG Signals Based on Siamese NetworksShuai Ma, Yuanhao LiuCollege of Mathematics and Computer Science, Yan’an University, Yan’an ShaanxiReceived: Jun. 20th , 2023; accepted: Jul. 20th , 2023; published: Jul. 28th, 2023AbstractIn the field of medical ECG signal classification, the issue of class imbalance arises when the exist-ing database contains a higher number of normal ECG data compared to abnormal ECG data. This imbalance can lead to biased models and affect the classification results. To address this problem, we propose a method combining contrastive learning with a siamese network structure to pre-train a model for ECG signal classification. In this study, we utilized a dataset of 12-lead ECG diag-马帅，刘元昊nostic data to construct our model. The siamese network structure was employed to pretrain the model, where each ECG signal was matched with its corresponding positive and negative samples to form pairs. The discriminative features extracted by the siamese network were then utilized for downstream classification tasks. Experimental results demonstrate that the pretraining paradigm based on contrastive learning effectively extracts discriminative features from ECG signals and sig-nificantly overcomes the issue of data imbalance. This method successfully utilizes a large number of positive samples, resulting in superior discriminative performance of the feature extraction model compared to traditional methods. The accuracy of the test set reached 96.55%, with a false positive rate of 6.91% and a false negative rate of only 2%. Overall, our study demonstrates that utilizing a siamese network structure for pretraining, combined with contrastive learning, im-proves the feature extraction process for ECG signal classification. This approach effectively ad-dresses the issue of data imbalance, resulting in highly accurate classification results.KeywordsECG Signal Classification, Siamese Network, Contrastive Loss, 1D-CNN, LSTM Array Copyright © 2023 by author(s) and Hans Publishers Inc.This work is licensed under the Creative Commons Attribution International License (CC BY 4.0)./licenses/by/4.0/1. 引言根据国家心血管病中心2021年发布的中国心血管健康与疾病年报显示，我国目前心血管疾病患者达到3.3亿，心血管疾病已经成为我国慢性病的“主力军”，严重危害国民健康安全，我国2019年农村、城市心血管病分别占死因的46.74%和44.26%，并且呈年轻化趋势。

837E-mail:***********.cn Website: Tel: ************©中国图象图形学报版权所有中国图象图形学报JOURNAL OF IMAGE AND GRAPHICS中图法分类号:TP391.41文献标识码：A文章编号：1006-8961(2021 )04-0837-1()论文引用格式：Cai Z D，Ying N ，Guo C S ，Guo R and Yang P. 2021. Research on multiperson pose estimation combined with Y0L0v3 pruning model.Journd of Image and GraphicS ，26(04) :0837-0846(蔡哲栋，应娜，郭春生，郭锐，杨鹏.2021. Y 0U )v3剪枝模型的多人姿态估计.中国图象图形学报，26 (04) :0837-0846) [ D01:10•丨 1834/jig. 200138 ]YOLOV 3剪枝模型的多人姿态估计蔡哲栋，应娜％郭春生，郭锐，杨鹏杭州电子科技大学通信工程学院，杭州310018摘要：目的为了解决复杂环境中多人姿态估计存在的定位和识别等问题，提高多人姿态估计的准确率，减少算 法存在的大tt冗余参数，提高姿态估计的运行速率，提出了基于批量归一化层（batch normalization, B N )通道剪枝的多 人姿态估计算法（Y 0L0v3 prune pose estimator, Y L P P E )。

方法以目标检测算法 Y0L0v3 ( you only look once v3 )和 堆叠沙漏网络（stacked hourglass network. S H M )算法为基础，通过重叠度K-means算法修改Y 0L 0v 3网络描框以更 适应行人目标检测，并训练得到Trimming-YOL〇V3网络；利用批量归一化层的缩放因子对Trimming-YOL〇V3网络 进行循环迭代式通道剪枝，设置剪枝阈值与缩放因子，实现较为有效的模型剪枝效果，训练得到Trim-Prune- Y 0L 0v 3网络；为了结合单人姿态估计网络，重定义图像尺寸为256 x 256像素（非正方形图像通过补零实现）；再级 联4个Hourglass子网络得到堆叠沙漏网络,从而提升整体姿态估计精度。