o *i5@s`ddlZddlZddlmZmZddlmZddlmZddl m Z m Z m Z ddl Z ddlZddlmZddlmZmZmZgZdee e jjZed d eZd d ZGd dde jjZ Gddde jjZ!GdddeZ"GdddeZ#Gddde jje"Z$Gddde#Z%eGdddZ&e&deeddddd d!d"d#d$d%d&d d' Z'd(e'_(dS))N)ABCabstractmethod) dataclass)partial)CallableListTuple) module_utils)emformer_rnnt_baseRNNTRNNTBeamSearch( g?cCs@t||tjk||tjk<||tjktj||tjk<|SN)torchlogmathexrq/var/www/html/eduruby.in/lip-sync/lip-sync-env/lib/python3.10/site-packages/torchaudio/pipelines/rnnt_pipeline.py_piecewise_linear_logsrc$eZdZfddZddZZS)_FunctionalModulect||_dSr)super__init__ functional)selfr __class__rrr  z_FunctionalModule.__init__cCs ||Sr)rrinputrrrforwards z_FunctionalModule.forward__name__ __module__ __qualname__rr% __classcell__rrr rrs rcr)_GlobalStatsNormalizationcsptt|}t|}Wdn1swY|dt|d|dt|ddS)Nmean invstddev) rropenjsonloadsreadZregister_bufferrtensor)rZglobal_stats_pathfZblobr rrr$s  z"_GlobalStatsNormalization.__init__cCs||j|jSr)r,r-r#rrrr%-sz!_GlobalStatsNormalization.forwardr&rrr rr+#s  r+c@s0eZdZedejdeejejffddZdS)_FeatureExtractorr$returncCdS)XGenerates features and length output from the given input tensor. Args: input (torch.Tensor): input tensor. Returns: (torch.Tensor, torch.Tensor): torch.Tensor: Features, with shape `(length, *)`. torch.Tensor: Length, with shape `(1,)`. Nrr#rrr__call__2z_FeatureExtractor.__call__N)r'r(r)rrTensorrr8rrrrr41s&r4c@s&eZdZedeedefddZdS)_TokenProcessortokensr5cKr6)zDecodes given list of tokens to text sequence. Args: tokens (List[int]): list of tokens to decode. Returns: str: Decoded text sequence. Nr)rr<kwargsrrrr8Cr9z_TokenProcessor.__call__N)r'r(r)rrintstrr8rrrrr;Bsr;csNeZdZdZdejjddffdd Zdejde ejejffdd Z Z S) _ModuleFeatureExtractorz``torch.nn.Module``-based feature extraction pipeline. Args: pipeline (torch.nn.Module): module that implements feature extraction logic. pipeliner5Ncrr)rrrA)rrAr rrrWr"z _ModuleFeatureExtractor.__init__r$cCs$||}t|jdg}||fS)r7r)rArr2shape)rr$featureslengthrrrr%[s z_ModuleFeatureExtractor.forward) r'r(r)__doc__rnnModulerr:rr%r*rrr rr@Ps(r@c@s>eZdZdZdeddfddZd deed edefd d Z dS) _SentencePieceTokenProcessorztSentencePiece-model-based token processor. Args: sp_model_path (str): path to SentencePiece model. sp_model_pathr5NcCsJtds tdddl}|j|d|_|j|j|jh|_ dS)N sentencepiecez2SentencePiece is not available. Please install it.r)Z model_file) r Zis_module_available RuntimeErrorrJZSentencePieceProcessorsp_modelZunk_idZeos_idZpad_idpost_process_remove_list)rrIZspmrrrrts  z%_SentencePieceTokenProcessor.__init__Tr<lstripcsDfdd|ddD}dj|dd}|r |S|S)aXDecodes given list of tokens to text sequence. Args: tokens (List[int]): list of tokens to decode. lstrip (bool, optional): if ``True``, returns text sequence with leading whitespace removed. (Default: ``True``). Returns: str: Decoded text sequence. csg|] }|jvr|qSr)rM).0Z token_indexrrr sz9_SentencePieceTokenProcessor.__call__..Nu▁ )joinrLZ id_to_piecereplacerN)rr<rNZfiltered_hypo_tokensZ output_stringrrPrr8s z%_SentencePieceTokenProcessor.__call__)T) r'r(r)rEr?rrr>boolr8rrrrrHms rHc@sLeZdZUdZGdddeZGdddeZee d<e ge fe d<ee d<ee d <e e d <e e d <e e d <e e d <e e d<e e d<e e d<e e d<de fddZ ede fddZede fddZede fddZede fddZede fddZede fdd Zdefd!d"Zdefd#d$Zdefd%d&Zdefd'd(Zd)S)* RNNTBundleuDataclass that bundles components for performing automatic speech recognition (ASR, speech-to-text) inference with an RNN-T model. More specifically, the class provides methods that produce the featurization pipeline, decoder wrapping the specified RNN-T model, and output token post-processor that together constitute a complete end-to-end ASR inference pipeline that produces a text sequence given a raw waveform. It can support non-streaming (full-context) inference as well as streaming inference. Users should not directly instantiate objects of this class; rather, users should use the instances (representing pre-trained models) that exist within the module, e.g. :data:`torchaudio.pipelines.EMFORMER_RNNT_BASE_LIBRISPEECH`. Example >>> import torchaudio >>> from torchaudio.pipelines import EMFORMER_RNNT_BASE_LIBRISPEECH >>> import torch >>> >>> # Non-streaming inference. >>> # Build feature extractor, decoder with RNN-T model, and token processor. >>> feature_extractor = EMFORMER_RNNT_BASE_LIBRISPEECH.get_feature_extractor() 100%|███████████████████████████████| 3.81k/3.81k [00:00<00:00, 4.22MB/s] >>> decoder = EMFORMER_RNNT_BASE_LIBRISPEECH.get_decoder() Downloading: "https://download.pytorch.org/torchaudio/models/emformer_rnnt_base_librispeech.pt" 100%|███████████████████████████████| 293M/293M [00:07<00:00, 42.1MB/s] >>> token_processor = EMFORMER_RNNT_BASE_LIBRISPEECH.get_token_processor() 100%|███████████████████████████████| 295k/295k [00:00<00:00, 25.4MB/s] >>> >>> # Instantiate LibriSpeech dataset; retrieve waveform for first sample. >>> dataset = torchaudio.datasets.LIBRISPEECH("/home/librispeech", url="test-clean") >>> waveform = next(iter(dataset))[0].squeeze() >>> >>> with torch.no_grad(): >>> # Produce mel-scale spectrogram features. >>> features, length = feature_extractor(waveform) >>> >>> # Generate top-10 hypotheses. >>> hypotheses = decoder(features, length, 10) >>> >>> # For top hypothesis, convert predicted tokens to text. >>> text = token_processor(hypotheses[0][0]) >>> print(text) he hoped there would be stew for dinner turnips and carrots and bruised potatoes and fat mutton pieces to [...] >>> >>> >>> # Streaming inference. >>> hop_length = EMFORMER_RNNT_BASE_LIBRISPEECH.hop_length >>> num_samples_segment = EMFORMER_RNNT_BASE_LIBRISPEECH.segment_length * hop_length >>> num_samples_segment_right_context = ( >>> num_samples_segment + EMFORMER_RNNT_BASE_LIBRISPEECH.right_context_length * hop_length >>> ) >>> >>> # Build streaming inference feature extractor. >>> streaming_feature_extractor = EMFORMER_RNNT_BASE_LIBRISPEECH.get_streaming_feature_extractor() >>> >>> # Process same waveform as before, this time sequentially across overlapping segments >>> # to simulate streaming inference. Note the usage of ``streaming_feature_extractor`` and ``decoder.infer``. >>> state, hypothesis = None, None >>> for idx in range(0, len(waveform), num_samples_segment): >>> segment = waveform[idx: idx + num_samples_segment_right_context] >>> segment = torch.nn.functional.pad(segment, (0, num_samples_segment_right_context - len(segment))) >>> with torch.no_grad(): >>> features, length = streaming_feature_extractor(segment) >>> hypotheses, state = decoder.infer(features, length, 10, state=state, hypothesis=hypothesis) >>> hypothesis = hypotheses[0] >>> transcript = token_processor(hypothesis[0]) >>> if transcript: >>> print(transcript, end=" ", flush=True) he hoped there would be stew for dinner turn ips and car rots and bru 'd oes and fat mut ton pieces to [...] c@eZdZdZdS)zRNNTBundle.FeatureExtractorz:Interface of the feature extraction part of RNN-T pipelineNr'r(r)rErrrrFeatureExtractorr[c@rY)zRNNTBundle.TokenProcessorz7Interface of the token processor part of RNN-T pipelineNrZrrrrTokenProcessorr\r] _rnnt_path_rnnt_factory_func_global_stats_path_sp_model_path_right_padding_blank _sample_rate_n_fft_n_mels _hop_length_segment_length_right_context_lengthr5cCs6|}tj|j}t|}||||Sr) r_ torchaudioutilsdownload_assetr^rloadZload_state_dicteval)rmodelpathZ state_dictrrr _get_models   zRNNTBundle._get_modelcC|jS)zSSample rate (in cycles per second) of input waveforms. :type: int )rdrPrrr sample_ratezRNNTBundle.sample_ratecCrr)z7Size of FFT window to use. :type: int )rerPrrrn_fftrtzRNNTBundle.n_fftcCrr)z`Number of mel spectrogram features to extract from input waveforms. :type: int )rfrPrrrn_mels rtzRNNTBundle.n_melscCrr)zdNumber of samples between successive frames in input expected by model. :type: int )rgrPrrr hop_lengthrtzRNNTBundle.hop_lengthcCrr)zTNumber of frames in segment in input expected by model. :type: int )rhrPrrrsegment_lengthrtzRNNTBundle.segment_lengthcCrr)zcNumber of frames in right contextual block in input expected by model. :type: int )rirPrrrright_context_length%rtzRNNTBundle.right_context_lengthcCs|}t||jS)zOConstructs RNN-T decoder. Returns: RNNTBeamSearch )rqr rc)rrorrr get_decoder-s zRNNTBundle.get_decoderc s^tjj}ttjtjj j j j j dtddtddt|tfddS)zzConstructs feature extractor for non-streaming (full-context) ASR. Returns: FeatureExtractor rsrurvrwcS |ddSNrRrZ transposerrrrB z2RNNTBundle.get_feature_extractor..cS t|tSrr_gainrrrrrCrcstjj|dddjfS)Nr)rrFrpadrbrrPrrrEsrjrkrlr`r@rrFZ SequentialZ transformsZMelSpectrogramrsrurvrwrr+rZ local_pathrrPrget_feature_extractor6s  z RNNTBundle.get_feature_extractorc CsPtj|j}ttjtjj |j |j |j |j dtddtddt|S)zvConstructs feature extractor for streaming (simultaneous) ASR. Returns: FeatureExtractor r{cSr|r}r~rrrrrUrz.cSrrrrrrrrVrrrrrrget_streaming_feature_extractorIs  z*RNNTBundle.get_streaming_feature_extractorcCstj|j}t|S)zQConstructs token processor. Returns: TokenProcessor )rjrkrlrarHrrrrget_token_processor[szRNNTBundle.get_token_processorN)r'r(r)rEr4r[r;r]r?__annotations__rr r>rqpropertyrsrurvrwrxryr rzrrrrrrrrXsB H rXz(models/emformer_rnnt_base_librispeech.pti)Z num_symbolsz2pipeline-assets/global_stats_rnnt_librispeech.jsonz.pipeline-assets/spm_bpe_4096_librispeech.modelii>iP) r^r_r`rarbrcrdrerfrgrhriaASR pipeline based on Emformer-RNNT, pretrained on *LibriSpeech* dataset :cite:`7178964`, capable of performing both streaming and non-streaming inference. The underlying model is constructed by :py:func:`torchaudio.models.emformer_rnnt_base` and utilizes weights trained on LibriSpeech using training script ``train.py`` `here `__ with default arguments. Please refer to :py:class:`RNNTBundle` for usage instructions. ))r/rabcrr dataclassesr functoolsrtypingrrrrrjZtorchaudio._internalr Ztorchaudio.modelsr r r __all__log10ZiinfoZint16maxZ_decibelpowrrrFrGrr+r4r;r@rHrXZEMFORMER_RNNT_BASE_LIBRISPEECHrErrrrsL    +M