hPddlmZmZddlZddlmZddlmZddlmZddl m Z ddl m Z dd l mZd d lmZmZmZmZd d lmZe j,eZGddedZGddej4ZGddeZGddeZGddeZGddeZgdZ y))Optional TypedDictN)nn)ACT2FN)Cache)Unpack)logging)deprecate_kwarg)GraniteMoeDecoderLayerGraniteMoeForCausalLMGraniteMoeModelGraniteMoePreTrainedModel)GraniteMoeSharedConfigceZdZUdZej ed<ej ed<eed<eed<ejed<y)GraniteFlashAttentionKwargsa Keyword arguments for advanced Flash Attention, causal-conv1d, and mamba_ssm kernel usage. Use cases include padding-free training and fewer `torch.compile` graph breaks. Attributes: cu_seq_lens_q (`torch.LongTensor`) Gets cumulative sequence length for query state. cu_seq_lens_k (`torch.LongTensor`) Gets cumulative sequence length for key state. max_length_q (`int`): Maximum sequence length for query state. max_length_k (`int`): Maximum sequence length for key state. seq_idx (`torch.IntTensor): Index of each packed sequence. cu_seq_lens_q cu_seq_lens_k max_length_q max_length_kseq_idxN) __name__ __module__ __qualname____doc__torch LongTensor__annotations__int IntTensor{/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/transformers/models/granitemoeshared/modular_granitemoeshared.pyrr&s7"###### __r$rF)totalc`eZdZdZdeffd ZdejdejfdZxZ S)GraniteMoeSharedMLPz~ MLP layer for shared experts Args: config: Configuration object with model hyperparameters. configc`t||j|_|j|_t |j |_tj|j|jdzd|_ tj|j|jd|_ y)Nr F)bias) super__init__ hidden_size input_sizeshared_intermediate_sizer hidden_act activationrLinear input_linear output_linearselfr) __class__s r%r-zGraniteMoeSharedMLP.__init__Hs  ,,!:: !2!23IIdoot7G7G!7KRWXYYt'7'7uUr$ hidden_statesreturnc|j|}|jdd}|j|d|dz}|j|}|S)Nr )dimrr)r4chunkr2r5)r7r9chunked_hidden_statess r%forwardzGraniteMoeSharedMLP.forwardQs^))-8 - 3 3A2 3 >(=a(@ADYZ[D\\ **=9 r$) rrrrrr-rTensorr@ __classcell__r8s@r%r(r(?s2V5VU\\ellr$r(ceZdZdedeffd Zeddd ddejd e ejd e ejde e d e e d e e d e ejde e de e ejejfdeede ej e e ej ej fffdZxZS)GraniteMoeSharedDecoderLayerr) layer_idxctt||||jdk(rd|_yt||_y)Nr)r,r-r0r( shared_mlpr7r)rFr8s r%r-z%GraniteMoeSharedDecoderLayer.__init__Zs3 +"("A"AQ"F$L_`fLgr$past_key_valuepast_key_valuesz4.58)new_nameversionr9attention_mask position_idsoutput_attentions use_cachecache_positionoutput_router_logitsposition_embeddingskwargsr:c l|} |j|}|jd|||||||| d| \}} | ||jzz}|} |j|}|j |\} }|j | }n| |j |z}~ | ||jzz}|f}|r|| fz }|r||fz }|S)aD Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs. Can be used to provide `GraniteFlashAttentionKwargs` for padding-free training and/or improve torch.compile performance. )r9rNrOrKrPrQrRrTr#)input_layernorm self_attnresidual_multiplierpost_attention_layernormblock_sparse_moerH)r7r9rNrOrKrPrQrRrSrTrUresidualself_attn_weightsmoe_hidden_states router_logitsoutputss r%r@z$GraniteMoeSharedDecoderLayer.forward^sN!,,]; ,:4>> , ')%+/) 3 ,  , ( (!=43K3K#KK !55mD +/+@+@+O(= ?? "-M- 0NNM  =43K3K#KK  "  )+ +G   ' 'Gr$)NNNFFNFN)rrrrr!r-r rrArrrbooltupler r FloatTensorr@rBrCs@r%rErEYsHh5h#h%0A6R2637+/,1$)59/4KOO||O!.Ou//0 O "% O $D> OD>O!!1!12O'tnO&eELL%,,,F&GHO45O u  (51B1BEDUDU1U+V"WW XOSOr$rEceZdZUeed<dgZy)GraniteMoeSharedPreTrainedModelr)rEN)rrrrr _no_split_modulesr#r$r%reres ""78r$rec$eZdZdeffd ZxZS)GraniteMoeSharedModelr)c t||tjt |j Dcgc]}t ||c}|_ycc}wN)r,r-r ModuleListrangenum_hidden_layersrElayersrIs r%r-zGraniteMoeSharedModel.__init__sE  mmNSTZTlTlNm n )&) < n  nsA)rrrrr-rBrCs@r%rhrhs 5  r$rhc*eZdZdgZdeffd ZxZS)GraniteMoeSharedForCausalLMzlm_head.weightr)cdt||t||_|j yrj)r,r-rhmodel post_initr6s r%r-z$GraniteMoeSharedForCausalLM.__init__s&  *62  r$)rrr_tied_weights_keysrr-rBrCs@r%rprps*+5r$rp)rprhre)!typingrrrr activationsr cache_utilsrprocessing_utilsr utilsr utils.deprecationr granitemoe.modeling_granitemoer rrrconfiguration_granitemoesharedr get_loggerrloggerrModuler(rErerhrp__all__r#r$r%rs ' ! &0 C   H %)52"))4U#9Up9&?9  O "7 fr$