o *i@s2ddlZddlmZmZmZddlZdgZdejdejfddZ d#dejd ejd ejdejd ejd eejdeejfd dZ de dej j fddZ dee dedeeefddZdeedeededejdejf ddZGdddej j ZGdddej j ZGd d!d!ej j ZGd"ddeZdS)$N)ListOptionalTupleEmformerlengthsreturncCsF|jd}tt|}tj||j|jd||| dk}|S)Nr)devicedtype) shapeinttorchmaxitemZarangerr expand unsqueeze)r batch_size max_length padding_maskri/var/www/html/eduruby.in/lip-sync/lip-sync-env/lib/python3.10/site-packages/torchaudio/models/emformer.py_lengths_to_padding_mask s r utterance right_contextsummarymemsleft_context_keyc Cs|d|d|d}|d}|dkrd}|S|t||d} |dur3|dnd} ||d| | } t| d}|S)Nrr )r)sizer rr r) rrrrrrTBrright_context_blocks_lengthZleft_context_blocks_lengthZklengthsrrr_gen_padding_masks  r! activationcCsD|dkr tjS|dkrtjS|dkrtjStd|)NreluZgeluZsiluzUnsupported activation )r nnZReLUZGELUZSiLU ValueError)r"rrr_get_activation_module's   r&weight_init_scale_strategy num_layerscCs\|dur ddt|DS|dkrddt|DS|dkr'ddt|DStd|)NcSsg|]}dqSNr).0_rrr 4sz*_get_weight_init_gains.. depthwisecSsg|] }dt|dqS)?r mathsqrtr* layer_idxrrrr,6sZconstantcSsg|] }dtdqS)r.r/r2rrrr,8sz-Unsupported weight_init_scale_strategy value )ranger%)r'r(rrr_get_weight_init_gains2sr6 col_widthscol_masknum_rowsrcs@t|t|kr tdfddt||D}tj|ddS)Nz0Length of col_widths must match that of col_maskcs4g|]\}}|rtj|dntj|dqS)r)r Zoneszeros)r*Z col_widthZ is_ones_colrr9rrr,Cs z-_gen_attention_mask_block..r dim)lenr%zipr cat)r7r8r9rZ mask_blockrr<r_gen_attention_mask_block=s  rBcseZdZdZ    d$dededed eed ed ef fd d Zde j de j de e j e j ffddZ de j de j dee j de j fddZ  d%de j de j de j de j de j de j dee j dee j de e j e j e j e j ffddZde j de j de j de j de j de j de e j e j ffd d!Ze jjde j de j de j de j de j de j de j de e j e j e j e j ffd"d#ZZS)&_EmformerAttentiona_Emformer layer attention module. Args: input_dim (int): input dimension. num_heads (int): number of attention heads in each Emformer layer. dropout (float, optional): dropout probability. (Default: 0.0) weight_init_gain (float or None, optional): scale factor to apply when initializing attention module parameters. (Default: ``None``) tanh_on_mem (bool, optional): if ``True``, applies tanh to memory elements. (Default: ``False``) negative_inf (float, optional): value to use for negative infinity in attention weights. (Default: -1e8) NFח input_dim num_headsdropoutweight_init_gain tanh_on_mem negative_infcst||dkrtd|d|d||_||_||_||_||_|j|jd|_t j j |d|dd|_ t j j ||dd|_ t j j ||dd|_|rht j jj|j j|d t j jj|j j|d dSdS) Nrz input_dim (z") is not a multiple of num_heads (z).gr4T)Zbias)Zgain)super__init__r%rFrGrHrJrKscalingr r$Linearemb_to_key_value emb_to_queryout_projinitZxavier_uniform_weight)selfrFrGrHrIrJrK __class__rrrMYs z_EmformerAttention.__init__inputrrc CsX|j\}}}|dd}|d||}t||g}||jddd\}} || fS)Nrr r4chunksr>)r rr rArPchunk) rUrXrrr+summary_lengthZright_ctx_utterance_blockZmems_right_ctx_utterance_blockkeyvaluerrr_gen_key_valuews z!_EmformerAttention._gen_key_valueattention_weightsattention_maskrcCs|}||d|j}|d}|d|j}|durC|||j|d}||ddtj |j}|||j|d}tj j j |dd |}tj j j|t|j|jdS)Nrr r4r=)ptraining)floatZ masked_fillrrKrrGviewtor boolr$Z functionalZsoftmaxZtype_asrHrd)rUr`rarZattention_weights_floatrrattention_probsrrr_gen_attention_probss z'_EmformerAttention._gen_attention_probsrrrrrleft_context_valc s|d|d|d|d} t|||g} t|||gjddd\} } |dur{|dur{| t||d} t| d|d| || |d| dg} t| d|d| || |d| dg} fdd| | | fD\}}}t|j | dd}t ||||||} |||}t||}|j j| jjfkrtd| dd| j}|}|d}|d| |}|| |d}jrt|}ntj|dd d }||| | fS) Nr rr4rYcs4g|]}|djjjddqS)rbrr ) contiguousrfrGrF transpose)r*ZtensorrrUrrr,s&z4_EmformerAttention._forward_impl..z+Computed attention has incorrect dimensionsi )minr)rrQr rArPr[rr ZbmmrNrmr!rjr rGrFAssertionErrorrlrfrRrJtanhclamp)rUrrrrrrarrkrqueryr]r^r Zreshaped_queryZ reshaped_keyZreshaped_valuer`rri attentionZoutput_right_context_memsr\output_right_context output_memsrrnr _forward_implsP $        z _EmformerAttention._forward_implc Cs,|||||||\}}} } ||ddfS)acForward pass for training. B: batch size; D: feature dimension of each frame; T: number of utterance frames; R: number of right context frames; S: number of summary elements; M: number of memory elements. Args: utterance (torch.Tensor): utterance frames, with shape `(T, B, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``utterance``. right_context (torch.Tensor): right context frames, with shape `(R, B, D)`. summary (torch.Tensor): summary elements, with shape `(S, B, D)`. mems (torch.Tensor): memory elements, with shape `(M, B, D)`. attention_mask (torch.Tensor): attention mask for underlying attention module. Returns: (Tensor, Tensor): Tensor output frames corresponding to utterance and right_context, with shape `(T + R, B, D)`. Tensor updated memory elements, with shape `(M, B, D)`. Nrb)rx) rUrrrrrraoutputrwr+rrrforwards"z_EmformerAttention.forwardc Cs|d|d|d}|d|d|d|d} t|| jtj|jd} d| dd|df<|j|||||| ||d\} } } }| | | |d|dd||d|ddfS)aForward pass for inference. B: batch size; D: feature dimension of each frame; T: number of utterance frames; R: number of right context frames; S: number of summary elements; M: number of memory elements. Args: utterance (torch.Tensor): utterance frames, with shape `(T, B, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``utterance``. right_context (torch.Tensor): right context frames, with shape `(R, B, D)`. summary (torch.Tensor): summary elements, with shape `(S, B, D)`. mems (torch.Tensor): memory elements, with shape `(M, B, D)`. left_context_key (torch.Tensor): left context attention key computed from preceding invocation. left_context_val (torch.Tensor): left context attention value computed from preceding invocation. Returns: (Tensor, Tensor, Tensor, and Tensor): Tensor output frames corresponding to utterance and right_context, with shape `(T + R, B, D)`. Tensor updated memory elements, with shape `(M, B, D)`. Tensor attention key computed for left context and utterance. Tensor attention value computed for left context and utterance. rr rTrbN)rrk)rr r;rgrhrrx)rUrrrrrrrkZ query_dimZkey_dimraryrwr]r^rrrinfers&)( z_EmformerAttention.infer)rDNFrE)NN)__name__ __module__ __qualname____doc__r rerrhrMr Tensorrr_rjrxrzjitexportr| __classcell__rrrVrrCLs&   I % rCcsheZdZdZ       d6ded ed ed ed ed edededeededeffdd Z dedee j de e j fddZde e j dee j e j e j ffddZde j de j ded e j de e j de e j f d!d"Zd#e j d$e j d%e j de j fd&d'Zd$e j d%e j dee j e j ffd(d)Zd#e j d$e j d%e j dee j e j ffd*d+Zd$e j d,e j d%e j d e j d-ee j dee j e j ff d.d/Zd$e j d,e j d%e j d e j dee e j dee j e j e e j ff d0d1Zd$e j d,e j d%e j d e j d-e j dee j e j e j ff d2d3Ze jjd$e j d,e j d%e j dee e j d e j dee j e j e e j e j ff d4d5ZZS)7_EmformerLayera$Emformer layer that constitutes Emformer. Args: input_dim (int): input dimension. num_heads (int): number of attention heads. ffn_dim: (int): hidden layer dimension of feedforward network. segment_length (int): length of each input segment. dropout (float, optional): dropout probability. (Default: 0.0) activation (str, optional): activation function to use in feedforward network. Must be one of ("relu", "gelu", "silu"). (Default: "relu") left_context_length (int, optional): length of left context. (Default: 0) max_memory_size (int, optional): maximum number of memory elements to use. (Default: 0) weight_init_gain (float or None, optional): scale factor to apply when initializing attention module parameters. (Default: ``None``) tanh_on_mem (bool, optional): if ``True``, applies tanh to memory elements. (Default: ``False``) negative_inf (float, optional): value to use for negative infinity in attention weights. (Default: -1e8) rDr#rNFrErFrGffn_dimsegment_lengthrHr"left_context_lengthmax_memory_sizerIrJrKc stt|||| | | d|_tj||_tjj||dd|_ t |} tj tj |tj ||| tj|tj ||tj||_tj ||_tj ||_||_||_||_||_|dk|_dS)N)rFrGrHrIrJrKTZ kernel_sizeZstrideZ ceil_moder)rLrMrCrur r$ZDropoutrH AvgPool1d memory_opr&Z SequentialZ LayerNormrOpos_fflayer_norm_inputlayer_norm_outputrrrrFuse_mem) rUrFrGrrrHr"rrrIrJrKZactivation_modulerVrrrMRs6      z_EmformerLayer.__init__rrrcCsbtj|j||j|d}tj|j||j|d}tj|j||j|d}tjd|tj|d}||||gS)Nr:r r{)r r;rrFrZint32)rUrrZ empty_memoryrrk past_lengthrrr _init_states  z_EmformerLayer._init_statestatecCs|ddd}t|j|}t|jt||j}|d|j|d}|d|j|d}|d|j|d}|||fS)Nrr r4)rrprrr0ceilr)rUrrZpast_left_context_lengthZpast_mem_lengthpre_memslc_keylc_valrrr _unpack_states  z_EmformerLayer._unpack_statenext_knext_v update_lengthrcCst|d|g}t|d|g}t|d|g|j d|d<||jd|jd|d<||jd|jd|d<|d||d<|S)Nr r4rr)r rArr r)rUrrrrrZnew_knew_vrrr _pack_states"z_EmformerLayer._pack_state rc_outputrrcCs4||t||g}|||}||}|Sr))rHr rArr)rUrrrresultrrr_process_attention_outputs z(_EmformerLayer._process_attention_outputcCs8|t||g}||dd|d|dfSNr)rr rAr)rUrrrrrr_apply_pre_attention_layer_normsz._EmformerLayer._apply_pre_attention_layer_normcCs2||||}||dd|d|dfSr)rr)rUrrrrrr_apply_post_attention_ffns$z(_EmformerLayer._apply_post_attention_ffnrrac Csp|durtd|jr||dddddd}n tdj|j|jd}|j ||||||d\}}||fS)Nz;attention_mask must be not None when for_inference is Falser r4rr{)rrrrrra) r%rrpermuter emptyrgr rru) rUrrrrrarrnext_mrrr_apply_attention_forwards  z'_EmformerLayer._apply_attention_forwardc Cs|dur|j|d|jd}||\}}}|jr0||dddddd} | dd} n tdj |j |jd} |j j |||| |||d\} } } } | | | |d||}| | |fS)Nr r:r4rr{)rrrrrrrk)rrrrrrrr rrgr rur|r)rUrrrrrrrrrrrrrrrr_apply_attention_infers$ z%_EmformerLayer._apply_attention_inferc CsB|||\}}||||||\}} ||||\} } | | | fS)a1Forward pass for training. B: batch size; D: feature dimension of each frame; T: number of utterance frames; R: number of right context frames; M: number of memory elements. Args: utterance (torch.Tensor): utterance frames, with shape `(T, B, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``utterance``. right_context (torch.Tensor): right context frames, with shape `(R, B, D)`. mems (torch.Tensor): memory elements, with shape `(M, B, D)`. attention_mask (torch.Tensor): attention mask for underlying attention module. Returns: (Tensor, Tensor, Tensor): Tensor encoded utterance frames, with shape `(T, B, D)`. Tensor updated right context frames, with shape `(R, B, D)`. Tensor updated memory elements, with shape `(M, B, D)`. )rrr) rUrrrrralayer_norm_utterancelayer_norm_right_contextrrwoutput_utterancervrrrrzs $ z_EmformerLayer.forwardc CsF|||\}}||||||\}} } ||||\} } | | | | fS)a2Forward pass for inference. B: batch size; D: feature dimension of each frame; T: number of utterance frames; R: number of right context frames; M: number of memory elements. Args: utterance (torch.Tensor): utterance frames, with shape `(T, B, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``utterance``. right_context (torch.Tensor): right context frames, with shape `(R, B, D)`. state (List[torch.Tensor] or None): list of tensors representing layer internal state generated in preceding invocation of ``infer``. mems (torch.Tensor): memory elements, with shape `(M, B, D)`. Returns: (Tensor, Tensor, List[torch.Tensor], Tensor): Tensor encoded utterance frames, with shape `(T, B, D)`. Tensor updated right context frames, with shape `(R, B, D)`. List[Tensor] list of tensors representing layer internal state generated in current invocation of ``infer``. Tensor updated memory elements, with shape `(M, B, D)`. )rrr) rUrrrrrrrrrw output_staterrvrrrr|s )   z_EmformerLayer.infer)rDr#rrNFrE)r}r~rrr restrrrhrMr rrrrrrrrrrrrrzrrr|rrrrVrr?s    ".(           / rcseZdZ   ddejjdedededef fdd Zd ejd ejfd d Z d eded e efddZ d ejd ejfddZ d ejdejd e ejejffddZejj dd ejdejdee e ejd e ejeje e ejffddZZS) _EmformerImplremformer_layersrrright_context_lengthrcsJt|dk|_tjj||dd|_||_||_||_ ||_ ||_ dS)NrTr) rLrMrr r$rrrrrrr)rUrrrrrrVrrrMPs   z_EmformerImpl.__init__rXrcCs|jd}t||j|j}g}t|dD]}|d|j}||j}||||q||||jdt|SNrr ) r r0rrrr5appendr rA)rUrXrnum_segsZright_context_blocksseg_idxstartendrrr_gen_right_contextfs   z _EmformerImpl._gen_right_contextrutterance_lengthc Cst||j}|j}|j}||}||}t||j|d}t|d|j|} |j|} |jrUt||jd} |d} | || | |||| ||| ||| g } | S||| ||| ||| g} | Sr) r0rrrrrrprr)rUrrrrclcZrc_startZrc_endZ seg_startZseg_endZ rc_lengthZm_startZ mem_lengthr7rrr_gen_attention_mask_col_widthsqs<  z,_EmformerImpl._gen_attention_mask_col_widthsc Cs*|d}t||j}g}g}g}|jr0d}ddt|D}ddt|D} |||g} nd}ddt|D}d} ||g} t|D]=} || |} t| ||j|j } | | t| |t |j|| |j|j }| || durt| | d|j }| |qEdt d d| Dt j}|S) Nr cSg|]}|dvqS))r rr*idxrrrr,z5_EmformerImpl._gen_attention_mask..cSr))rrrrrrrr,rcSr))r rrrrrrr,rr cSsg|]}t|qSr)r rA)r*maskrrrr,s)rr0rrrr5rrBrrrrpr rArgrh)rUrXrrZrc_maskZ query_maskZ summary_maskZnum_colsZrc_q_cols_maskZ s_cols_maskZmasks_to_concatrr7Z rc_mask_blockZquery_mask_blockZsummary_mask_blockrarrr_gen_attention_masksH          z!_EmformerImpl._gen_attention_maskrc Cs|ddd}||}|d|d|j}||}|jr3||ddddddddn tdj |j |j d}|}|j D] }||||||\}}}qD|ddd|fS)aGForward pass for training and non-streaming inference. B: batch size; T: max number of input frames in batch; D: feature dimension of each frame. Args: input (torch.Tensor): utterance frames right-padded with right context frames, with shape `(B, T + right_context_length, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid utterance frames for i-th batch element in ``input``. Returns: (Tensor, Tensor): Tensor output frames, with shape `(B, T, D)`. Tensor output lengths, with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in output frames. r rr4Nrbr{) rrrrrrrr rrgr rr) rUrXrrrrarrylayerrrrrzs  ( z_EmformerImpl.forwardNstatescCs$|d|j|jkrtd|j|jd|dd|ddd}|d|j}||d}|d|}tj||jdd}|jrT||ddddddn t dj |j |j d }|} g} t |jD]\} } | | |||durxdn|| |\} }} }| | qi| ddd|| fS) aForward pass for streaming inference. B: batch size; D: feature dimension of each frame. Args: input (torch.Tensor): utterance frames right-padded with right context frames, with shape `(B, segment_length + right_context_length, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``input``. states (List[List[torch.Tensor]] or None, optional): list of lists of tensors representing internal state generated in preceding invocation of ``infer``. (Default: ``None``) Returns: (Tensor, Tensor, List[List[Tensor]]): Tensor output frames, with shape `(B, segment_length, D)`. Tensor output lengths, with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in output frames. List[List[Tensor]] output states; list of lists of tensors representing internal state generated in current invocation of ``infer``. r zIPer configured segment_length and right_context_length, expected size of z# for dimension 1 of input, but got .rr4N)rpr{)rrrr%rr rsrrrrgr r enumeraterr|r)rUrXrrZright_context_start_idxrrZoutput_lengthsrryZ output_statesr3rrrrrr|s:     z_EmformerImpl.infer)rrrr))r}r~rr r$ ModuleListr rMrrrrrrrzrrrr|rrrrVrrOs< $&0#rcsleZdZdZ        dded ed ed ed ed edededededeededeffdd Z Z S)ra_Emformer architecture introduced in *Emformer: Efficient Memory Transformer Based Acoustic Model for Low Latency Streaming Speech Recognition* :cite:`shi2021emformer`. See Also: * :func:`~torchaudio.models.emformer_rnnt_model`, :func:`~torchaudio.models.emformer_rnnt_base`: factory functions. * :class:`torchaudio.pipelines.RNNTBundle`: ASR pipelines with pretrained model. Args: input_dim (int): input dimension. num_heads (int): number of attention heads in each Emformer layer. ffn_dim (int): hidden layer dimension of each Emformer layer's feedforward network. num_layers (int): number of Emformer layers to instantiate. segment_length (int): length of each input segment. dropout (float, optional): dropout probability. (Default: 0.0) activation (str, optional): activation function to use in each Emformer layer's feedforward network. Must be one of ("relu", "gelu", "silu"). (Default: "relu") left_context_length (int, optional): length of left context. (Default: 0) right_context_length (int, optional): length of right context. (Default: 0) max_memory_size (int, optional): maximum number of memory elements to use. (Default: 0) weight_init_scale_strategy (str or None, optional): per-layer weight initialization scaling strategy. Must be one of ("depthwise", "constant", ``None``). (Default: "depthwise") tanh_on_mem (bool, optional): if ``True``, applies tanh to memory elements. (Default: ``False``) negative_inf (float, optional): value to use for negative infinity in attention weights. (Default: -1e8) Examples: >>> emformer = Emformer(512, 8, 2048, 20, 4, right_context_length=1) >>> input = torch.rand(128, 400, 512) # batch, num_frames, feature_dim >>> lengths = torch.randint(1, 200, (128,)) # batch >>> output, lengths = emformer(input, lengths) >>> input = torch.rand(128, 5, 512) >>> lengths = torch.ones(128) * 5 >>> output, lengths, states = emformer.infer(input, lengths, None) rDr#rr-FrErFrGrr(rrHr"rrrr'rJrKc sVt| | tj f ddt|D}tj|| ddS)Ncs.g|]}t | d qS))rHr"rrrIrJrK)rr2 r"rHrrFrrrKrGrrJZweight_init_gainsrrr,]s z%Emformer.__init__..)rrr)r6r r$rr5rLrM)rUrFrGrr(rrHr"rrrr'rJrKrrVrrrMKs  zEmformer.__init__)rDr#rrrr-FrE) r}r~rrr rerrrhrMrrrrVrr&sJ+     r))r0typingrrrr __all__rrr!rr$Moduler&r rer6rhrrBrCrrrrrrrsT "   tX