hi:ddlZddlmZmZddlZddlmZmZddlmZGddejZ GddejZ Gd d ejZ y) N)TupleType)Tensornn)MLPBlockceZdZdZej dfdededededeejded d ffd Z d e jd e jde jd e e je jffdZ xZS)TwoWayTransformera A Two-Way Transformer module for simultaneous attention to image and query points. This class implements a specialized transformer decoder that attends to an input image using queries with supplied positional embeddings. It's useful for tasks like object detection, image segmentation, and point cloud processing. Attributes: depth (int): Number of layers in the transformer. embedding_dim (int): Channel dimension for input embeddings. num_heads (int): Number of heads for multihead attention. mlp_dim (int): Internal channel dimension for the MLP block. layers (nn.ModuleList): List of TwoWayAttentionBlock layers composing the transformer. final_attn_token_to_image (Attention): Final attention layer from queries to image. norm_final_attn (nn.LayerNorm): Layer normalization applied to final queries. Methods: forward: Process image and point embeddings through the transformer. Examples: >>> transformer = TwoWayTransformer(depth=6, embedding_dim=256, num_heads=8, mlp_dim=2048) >>> image_embedding = torch.randn(1, 256, 32, 32) >>> image_pe = torch.randn(1, 256, 32, 32) >>> point_embedding = torch.randn(1, 100, 256) >>> output_queries, output_image = transformer(image_embedding, image_pe, point_embedding) >>> print(output_queries.shape, output_image.shape) depth embedding_dim num_headsmlp_dim activationattention_downsample_ratereturnNc `t|||_||_||_||_t j|_t|D]/}|jjt||||||dk(1t||||_ t j||_y)ak Initialize a Two-Way Transformer for simultaneous attention to image and query points. Args: depth (int): Number of layers in the transformer. embedding_dim (int): Channel dimension for input embeddings. num_heads (int): Number of heads for multihead attention. Must divide embedding_dim. mlp_dim (int): Internal channel dimension for the MLP block. activation (Type[nn.Module], optional): Activation function to use in the MLP block. attention_downsample_rate (int, optional): Downsampling rate for attention mechanism. r)r r rrrskip_first_layer_pedownsample_rateN)super__init__r r r rr ModuleListlayersrangeappendTwoWayAttentionBlock Attentionfinal_attn_token_to_image LayerNormnorm_final_attn) selfr r r rrri __class__s h/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/ultralytics/models/sam/modules/transformer.pyrzTwoWayTransformer.__init__)s(  *" mmo u A KK  $"/'#).G)*a   *3=)]v)w&!||M:image_embeddingimage_pepoint_embeddingcB|jdjddd}|jdjddd}|}|}|jD]}|||||\}}||z}||z}|j|||} || z}|j |}||fS)a Process image and point embeddings through the Two-Way Transformer. Args: image_embedding (torch.Tensor): Image to attend to, with shape (B, embedding_dim, H, W). image_pe (torch.Tensor): Positional encoding to add to the image, with same shape as image_embedding. point_embedding (torch.Tensor): Embedding to add to query points, with shape (B, N_points, embedding_dim). Returns: queries (torch.Tensor): Processed point embeddings with shape (B, N_points, embedding_dim). keys (torch.Tensor): Processed image embeddings with shape (B, H*W, embedding_dim). r r)querieskeysquery_pekey_peqkv)flattenpermuterrr ) r!r&r'r(r+r,layerr0r1attn_outs r$forwardzTwoWayTransformer.forwardSs&*11!4<>> embedding_dim, num_heads = 256, 8 >>> block = TwoWayAttentionBlock(embedding_dim, num_heads) >>> queries = torch.randn(1, 100, embedding_dim) >>> keys = torch.randn(1, 1000, embedding_dim) >>> query_pe = torch.randn(1, 100, embedding_dim) >>> key_pe = torch.randn(1, 1000, embedding_dim) >>> processed_queries, processed_keys = block(queries, keys, query_pe, key_pe) ir Fr r rrrrrNct|t|||_t j ||_t||||_t j ||_t||||_ t j ||_ t j ||_ t||||_ ||_y)a Initialize a TwoWayAttentionBlock for simultaneous attention to image and query points. This block implements a specialized transformer layer with four main components: self-attention on sparse inputs, cross-attention of sparse inputs to dense inputs, MLP block on sparse inputs, and cross-attention of dense inputs to sparse inputs. Args: embedding_dim (int): Channel dimension of the embeddings. num_heads (int): Number of attention heads in the attention layers. mlp_dim (int, optional): Hidden dimension of the MLP block. activation (Type[nn.Module], optional): Activation function for the MLP block. attention_downsample_rate (int, optional): Downsampling rate for the attention mechanism. skip_first_layer_pe (bool, optional): Whether to skip positional encoding in the first layer. rN)rrr self_attnrrnorm1cross_attn_token_to_imagenorm2rmlpnorm3norm4cross_attn_image_to_tokenr)r!r r rrrrr#s r$rzTwoWayAttentionBlock.__init__s0 "=)<\\-0 )2=)]v)w&\\-0 M7J?\\-0 \\-0 )2=)]v)w r%r+r,r-r.c|jr|j|||}n||z}|j|||}||z}|j|}||z}||z}|j|||}||z}|j |}|j |}||z}|j |}||z}||z}|j|||}||z}|j|}||fS)a Apply two-way attention to process query and key embeddings in a transformer block. Args: queries (torch.Tensor): Query embeddings with shape (B, N_queries, embedding_dim). keys (torch.Tensor): Key embeddings with shape (B, N_keys, embedding_dim). query_pe (torch.Tensor): Positional encodings for queries with same shape as queries. key_pe (torch.Tensor): Positional encodings for keys with same shape as keys. Returns: queries (torch.Tensor): Processed query embeddings with shape (B, N_queries, embedding_dim). keys (torch.Tensor): Processed key embeddings with shape (B, N_keys, embedding_dim). r/) rrDrErFrGrHrIrKrJ) r!r+r,r-r.r0r6r1mlp_outs r$r7zTwoWayAttentionBlock.forwards"  # #nnw'WnEG("A~~Q'~:H(G**W% h  6M11Ad1CH$**W%((7#G#**W% h  6M11Ag1Fhzz$}r%)r8r9r:r;rr<r=rr>boolrr?rrr7r@rAs@r$rrsF&(gg)*$)%7%7%7 %7 O %7 $' %7"%7 %7N,||,+0<<,CH<<,Y^YeYe, u||U\\) *,r%rc eZdZdZ ddededededdf fd Zed ejdedejfd Z ed edefd Z d ejd ejdejdejfdZ xZ S)ra An attention layer with downscaling capability for embedding size after projection. This class implements a multi-head attention mechanism with the option to downsample the internal dimension of queries, keys, and values. Attributes: embedding_dim (int): Dimensionality of input embeddings. kv_in_dim (int): Dimensionality of key and value inputs. internal_dim (int): Internal dimension after downsampling. num_heads (int): Number of attention heads. q_proj (nn.Linear): Linear projection for queries. k_proj (nn.Linear): Linear projection for keys. v_proj (nn.Linear): Linear projection for values. out_proj (nn.Linear): Linear projection for output. Methods: _separate_heads: Separate input tensor into attention heads. _recombine_heads: Recombine separated attention heads. forward: Compute attention output for given query, key, and value tensors. Examples: >>> attn = Attention(embedding_dim=256, num_heads=8, downsample_rate=2) >>> q = torch.randn(1, 100, 256) >>> k = v = torch.randn(1, 50, 256) >>> output = attn(q, k, v) >>> print(output.shape) torch.Size([1, 100, 256]) Nr r r kv_in_dimrct|||_||n||_||z|_||_|j|zdk(sJdt j||j|_t j|j|j|_ t j|j|j|_ t j|j||_ y)a? Initialize the Attention module with specified dimensions and settings. Args: embedding_dim (int): Dimensionality of input embeddings. num_heads (int): Number of attention heads. downsample_rate (int, optional): Factor by which internal dimensions are downsampled. kv_in_dim (int | None, optional): Dimensionality of key and value inputs. If None, uses embedding_dim. Raises: AssertionError: If num_heads does not evenly divide the internal dim (embedding_dim / downsample_rate). Nrz$num_heads must divide embedding_dim.) rrr rP internal_dimr rLinearq_projk_projv_projout_proj)r!r r rrPr#s r$rzAttention.__init__s& *&/&;)_<"  9,1Y3YY1ii t/@/@A ii0A0AB ii0A0AB  $"3"3]C r%xct|j\}}}|j|||||z}|jddS)zGSeparate the input tensor into the specified number of attention heads.r*r )shapereshape transpose)rXr bncs r$_separate_headszAttention._separate_heads4s<''1a IIaIqI~ 6{{1a  r%ct|j\}}}}|jdd}|j||||zS)z9Recombine separated attention heads into a single tensor.r*r )rZr\r[)rXr]n_headsn_tokens c_per_heads r$_recombine_headszAttention._recombine_heads;s>,-77(7Hj KK1 yyHg &:;;r%r0r1r2c |j|}|j|}|j|}|j||j}|j||j}|j||j}|j \}}}}||j ddddz}|tj|z }tj|d}||z}|j|}|j|S)a Apply multi-head attention to query, key, and value tensors with optional downsampling. Args: q (torch.Tensor): Query tensor with shape (B, N_q, embedding_dim). k (torch.Tensor): Key tensor with shape (B, N_k, embedding_dim). v (torch.Tensor): Value tensor with shape (B, N_k, embedding_dim). Returns: (torch.Tensor): Output tensor after attention with shape (B, N_q, embedding_dim). rr*r )dim) rTrUrVr`r rZr4mathsqrtr?softmaxrerW)r!r0r1r2_rdattnouts r$r7zAttention.forwardBs KKN KKN KKN  DNN 3  DNN 3  DNN 3 gg1a199Q1a((dii ++}}Tr*Qh##C(}}S!!r%)r*N) r8r9r:r;r=r staticmethodr?rr`rer7r@rAs@r$rrsD ! DDD D  D  D>!5<<!C!ELL!! r rrr%r$rtsN  +q qhs299slk" k"r%