h ZdZddlZddlmZddlmZmZddlZddlm Z ddl m cm Z ddl mZddlmZmZdd lmZd d lmZeGd d eZeGddeZeGddeZGdde j2ZGdde j2ZGdde j2ZGdde j2ZGdde j2ZGdde j2ZGdde j2Z Gd d!e j2Z!eGd"d#eZ"ed$%Gd&d'e"Z#d'd#gZ$y)(zTransformers Xcodec model.N) dataclass)OptionalUnion)PreTrainedAudioTokenizerBase) ModelOutputauto_docstring) AutoModel) XcodecConfigcbeZdZUdZdZeejed<dZ eejed<y) XcodecOutputao Args: audio_codes (`torch.LongTensor` of shape `(batch_size, num_quantizers, codes_length)`, *optional*): Discrete code indices computed using `model.encode`. audio_values (`torch.FloatTensor` of shape `(batch_size, channels, num_samples)`, *optional*) Decoded audio values obtained using the decoder part of Xcodec. N audio_codes audio_values) __name__ __module__ __qualname____doc__rrtorch LongTensor__annotations__r FloatTensorh/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/transformers/models/xcodec/modeling_xcodec.pyrrs3/3K%**+204L(5,,-4rrc:eZdZUdZdZeejed<y)XcodecEncoderOutputz Args: audio_codes (`torch.LongTensor` of shape `(batch_size, num_quantizers, codes_length)`, *optional*): Discrete code indices computed using `model.encode`. Nr) rrrrrrrrrrrrrr-s /3K%**+2rrc:eZdZUdZdZeejed<y)XcodecDecoderOutputz Args: audio_values (`torch.FloatTensor` of shape `(batch_size, channels, num_samples)`, *optional*): Decoded audio values obtained using the decoder part of Xcodec. Nr) rrrrrrrrrrrrr r 8s 15L(5,,-4rr cleZdZdZdedededeffd Zdejdejfd Z xZ S) ResidualUnitzFResidual block for SemanticEncoder and SemanticDecoder used in Xcodec.config in_channels out_channelsdilationc t|tj|_|j dz dz|z}tj |||j d||dd|_tj ||dd|_y)Nr r F)stridepaddingr&groupsbias)r$r% kernel_sizer+) super__init__nnELU activationunit_kernel_sizeConv1dconv1conv2)selfr#r$r%r&r) __class__s rr.zResidualUnit.__init__Fs} &&(++a/A5AYY    # #   YYr.rr?r=r@rAs@rrCrC^sE x| x# xS xZ] xELLU\\rrCcVeZdZfdZdej dej fdZxZS)SemanticEncoderc t|t|jt|jk7r t dt j|j|j|jd|jdzd|_ |j}g}t|jD]>\}}t|j|j|z}|t||||gz }|}@t j||_y)Nz:Number of strides must match the number of channel_ratios.r r Fr+)r-r.lenstrideschannel_ratios ValueErrorr/r3semantic_hidden_sizer,rI enumerater>rCrF conv_blocks)r6r#r$rYir(r%r7s rr.zSemanticEncoder.__init__rs  v~~ #f&;&;"< <YZ ZII  ' '  ' '      ! #   11  "6>>2 'IAvv::V=R=RST=UUVL 0lTZ[\ \K&K ' ==5rr8r9cZ|j|}|jD] }||} |Sr;)rIrYr6r8blocks rr=zSemanticEncoder.forwards5yy. %% /E .L /rrrrr.rr?r=r@rAs@rrPrPqs#6,ELLU\\rrPcheZdZdedededeffd ZdejdejfdZxZ S) SemanticDecoderBlockr#r$r%r(c ht ||dk(r!tj||dddd|_n:d|z}|dzdz}|dzdk(rdnd}tj ||||||d|_tj |jDcgc]}t||||c}|_ ycc}w) Nr rTrEr rFrR) r-r.r/r3rIConvTranspose1drFrGr"rH) r6r#r$r%r(r,r)output_paddingr&r7s rr.zSemanticDecoderBlock.__init__s  Q;  DIf*Kza'G"(1*/QqN**\;^cDIX^XnXn oH\&, h G o  os B/r8r9cZ|j|}|jD] }||} |Sr;)rIrHrLs rr=zSemanticDecoderBlock.forwards3yy. NN .D -L .rrNrAs@rr`r`s@ | # S Z] .ELLU\\rr`cVeZdZfdZdej dej fdZxZS)SemanticDecoderct|tj|jt |j|j dz|jd|jdzd|_g}t|jD]\}}t |j|j |z}|t|j dz kr)t |j|j |dzz}n |j}|t||||gz }tj||_tj|j|j|jd|jdzd|_y)Nrr r F)r$r%r,r(r)r+)r(r)r+)r-r.r/r3rWr>rUr,r4rXrTrSr`rFrYr5)r6r#rYrZr(r$r%r7s rr.zSemanticDecoder.__init__s_ YY33V886;P;PQR;SST**&&!+    "6>>2 ]IAvf99F#>AVAVWX[\W\A]#]^ %:: 0lTZ[\ \K ]==5YY  ' '  ' '   &&!+   rr8r9c||j|}|jD] }||} |j|}|Sr;)r4rYr5r\s rr=zSemanticDecoder.forwardsCzz,/ %% /E .L /zz,/ rr^rAs@rrfrfs# >ELLU\\rrfc4eZdZdZfdZdZdZdZxZS)XcodecEuclideanCodebookz!Codebook with Euclidean distance.ct|tj|j|j }|j|_|j dtjdg|j dtj|j|j d||j d|jy)NinitedT cluster_sizeembed embed_avg) r-r.rzeros codebook_size codebook_dimregister_bufferr?clone)r6r#rnr7s rr.z XcodecEuclideanCodebook.__init__s  F00&2E2EF#11 Xu||TF';< ^U[[9M9M-NO We, [%++-8rc|jj}|jdjdd}|d|z|zz |jdjddz }|j dj }|S)Nr r T)keepdimrdim)rntpowsummaxindices)r6 hidden_statesrn scaled_statesdist embed_inds rquantizez XcodecEuclideanCodebook.quantizes %))!,00D0A ]!2U!::UYYq\=M=MaY]=M=^^_HHH$,, rc|j}|jd|df}|j|}|j|dd}|S)Nrw)shapereshaperview)r6rrrs rencodezXcodecEuclideanCodebook.encodesO##%--r59o> MM-0 "INNE#2J/ rcFtj||j}|Sr;)F embeddingrn)r6r quantizeds rdecodezXcodecEuclideanCodebook.decodesKK 4::6 r) rrrrr.rrrr@rAs@rrjrjs+9rrjc4eZdZdZdeffd ZdZdZxZS)XcodecVectorQuantizationzY Vector quantization implementation. Currently supports only euclidean distance. r#cBt|t||_yr;)r-r.rjcodebook)r6r#r7s rr.z!XcodecVectorQuantization.__init__s /7 rcb|jddd}|jj|}|SNrr r )permuterr)r6rembed_ins rrzXcodecVectorQuantization.encodes/%--aA6 =='' 6rcb|jj|}|jddd}|Sr)rrr)r6rrs rrzXcodecVectorQuantization.decodes/=='' 2##Aq!,r) rrrrr r.rrr@rAs@rrrs8|8  rrceZdZdZdeffd ZdZd defdZd de jde jfdZ d e jde jfd Z xZ S) XcodecResidualVectorQuantizationzv Residual vector quantization implementation. Follows Algorithm 1 in https://huggingface.co/papers/2107.03312 r#ct|tjt |j Dcgc] }t |c}|_|j|_|j|_ |j |_ycc}wr;) r-r.r/rFrangenum_quantizersr quantizers frame_raterq)r6r#_r7s rr.z)XcodecResidualVectorQuantization.__init__sk --SXY_YnYnSo(pa)A&)I(pq ++#11$33)qsBc`tj|j|jzdz S)zReturn bandwidth per quantizer.i)mathlog2rqr)r6s rget_bandwidth_per_quantizerzEErr9c |j}|j}|0|dkDr+ttdt j ||z }|S)z:Return num_quantizers based on specified target bandwidth.r )rrr>r}rfloor)r6 bandwidthbw_per_qrs r get_num_quantizers_for_bandwidthzAXcodecResidualVectorQuantization.get_num_quantizers_for_bandwidthsL335,,  Y_ Q 9x3G(H!IJNr embeddingsc|j|}|}g}|jd|D]:}|j|}|j|}||z }|j |<t j |} | S)a Encode the input tensor into discrete indices using RVQ, with the number of quantizers selected based on the given bandwidth. Each quantizer /codebook residually quantizes the input and returns the nearest indices in terms of Euclidian distance. N)rrrrappendrstack) r6rrrresidual all_indices quantizerr~r out_indicess rrz'XcodecResidualVectorQuantization.encode$s >>yI .9 (I&&x0G!((1I)+H   w '  ( kk+. rcodesctjd|j}t|D]*\}}|j|}|j |}||z},|S)z9Decode the given codes to their quantized representation.r)device)rtensorrrXrr)r6r quantized_outrZr~rrs rrz'XcodecResidualVectorQuantization.decode4s^ S> #E* 6JAw*I!((1I)I5M 6rr;)rrrrr r.rr>rrr?rrr@rAs@rrr s\4|4F#%,, ELLU\\rrc.eZdZdZeZdZdZdZdZ dZ y)XcodecPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. xcodec input_valuescnt|tjrm|jjj d|j j|j%|jjjyyt|tjtjfrJ|jjj|jjjdyt|tjrtjj|j|jjt!j"|j$|j&|j(dzz }tjj+|j| |yyy)zInitialize the weightsr)meanstdNg?r)ab) isinstancer/Linearweightdatanormal_r#initializer_ranger+zero_ LayerNorm GroupNormfill_r3initkaiming_normal_rsqrtr*r$r,uniform_)r6moduleks r _init_weightsz#XcodecPreTrainedModel._init_weightsIs( fbii ( MM   & &CT[[5R5R & S{{&   &&('r|| < = KK   " " $ MM   $ $S )  * GG # #FMM 2{{&IIfmmv/A/AFDVDVWXDY/YZ[  a 8'+rctjjj}t tjjj dr.tjjj j}||j j||j j|j jD]`}||j|j|j|jfD]&}||j||j(b||jjd||jjd|jjD]f}||jd|j|j|jfD]*}||jd||jd,hy)znApply weight norm in the acoustic encoder and decoder because the original checkpoint has weight norm applied. weight_normrnameN)rr/utilsrhasattrparametrizationsacoustic_encoderr4r5r] res_unit1 res_unit2 res_unit3acoustic_decoderconv_t1)r6rr]res_units rapply_weight_normz'XcodecPreTrainedModel.apply_weight_normYsdhhnn00 588>>22M B((..99EEKD))//0D))//0**00 ,E  $"__eoouO ,HNN+HNN+ , , D))//h?D))//h?**00 ;E  H 5"__eoouO ;HNN:HNN: ; ;rc|j|jfD]}|jD]} tjj j |dt|dss)  L $O9 ;0 mrrz$The Xcodec neural audio codec model.) custom_introceZdZfdZedej fdZdejdejfdZ e d dejde ede edeejeffd Ze dd ejde edeejeffd Ze ddejd e ejde ede edeeejejfeff d ZxZS) XcodecModelct||||_|jdz|_t j |j}|j|_ |j|_ |j|jt||_t||_t j |j"j%|_t)j*|j,|j,|_t)j*|j,|j"j,|_t)j*|j,|jj,|_t5||_y)Nr )r-r.r# hop_lengthpadr from_configacoustic_model_configencoderrdecoderr_adjust_dac_decoderrPencoder_semanticrfdecoder_semanticsemantic_model_configevalsemantic_modelr/r hidden_sizefcfc1fc2rr)r6r#acoustic_modelr7s rr.zXcodecModel.__init__s   $$)"..v/K/KL . 6 6 . 6 6   !6!67 / 7 / 7'33F4P4PQVVX))F..0B0BC99V//1M1M1Y1YZ99V//1M1M1Y1YZ9&Arrcz|jD]]}t|tjst|jt r|jdn |j}|dzf|__t|dr?t|jtjrtj|_yyy)z DAC implemented in Xcodec is slightly different from the HF version. DAC in Xcodec adjusts the output padding in every ConvTranspose1d in the decoder and removes the final `nn.Tanh` activation function. rr tanhN) rrr/rbr(tuplercrrTanhIdentity)rrr(s rrzXcodecModel._adjust_dac_decodersoo' 6F&""4"45-7 u-Mq)SYS`S`)/! % 6 7F # 7<<(I;;=GL)J #rrr9cT|dddddf}tj||j|jf}tj5|j |d}|j }dddtj d}|jdS#1swY2xYw)NrT)output_hidden_statesr rx)rrrno_gradrrrr)r6routputsrstackeds r_extract_semantic_featuresz&XcodecModel._extract_semantic_featuress#Aq!G, uu\DHHdhh+?@ ]]_ 2)),T)RG#11M 2++m3|||""  2 2s BB'r return_dictc ||n|jj}|jd}|dk7rtd|||jjd}n>||jjvr&td|d|jjd|j |j }|j|jdd}|j|}|jd|jdk7rT|jtj|ddd ddf|j|jfjd}tj||gd }|j|jddjdd}|j j#||} | jd d} |s| St%| S) ac input_values (`torch.FloatTensor` of shape `(batch_size, channels, num_samples)`): Float values of the input audio waveform. bandwidth (`float`, *optional*): The target bandwidth in (kbps) supports only values in `config.target_bandwidths`. Defaults to the highest available bandwidth `4.0` kbps. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`]. Returns: `torch.LongTensor` of shape `(batch_size, num_quantizers, codes_length)` containing the discrete encoded audio codes. Nr zAudio must be mono, but got rwz)This model doesn't support the bandwidth z. Select one of .r rrx)r#r rrVtarget_bandwidthsr detachr transposerrr unsqueezercatrrrr) r6rrr channelse_semantic_input e_semantic e_acousticrrs rrzXcodecModel.encodes&&1%>> from datasets import load_dataset >>> from transformers import AutoFeatureExtractor, XcodecModel >>> model_id = "hf-audio/xcodec-hubert-librispeech" >>> model = XcodecModel.from_pretrained(model_id) >>> feature_extractor = AutoFeatureExtractor.from_pretrained(model_id) >>> dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> dataset = dataset.cast_column("audio", Audio(sampling_rate=feature_extractor.sampling_rate)) >>> audio_sample = dataset[0]['audio']['array'] >>> inputs = feature_extractor(raw_audio=audio_sample, return_tensors="pt") >>> outputs = model(**inputs) >>> audio_codes = outputs.audio_codes >>> audio_values = outputs.audio_values ``` NrwF)r r.)rr)r#r rrrr)r6rrrr lengthrs rr=zXcodecModel.forwards\&1% ? 8P8Prr)%rr dataclassesrtypingrrrtorch.nnr/torch.nn.functional functionalrmodeling_utilsrrrr autor configuration_xcodecr rrr rr"rCrPr`rfrjrrrr__all__rrrr'sV! !" :0.  5; 5  5 3+3 3 5+5 5,299,6299&bii<299>%bii%Pbii@ryy,/ryy/d;m8;m;m|GHpP'pPIpPf 1 2r