o i0@sTddlZddlmZddlmZmZmZmZmZmZddl m Z m Z ddgZ GdddeZ d d ed ed ed e _  dde ede ede ede ede ededededededefddZde ede ede ede ededededededefddZde ede ede ede ededededededefddZdS)N)Tensor) Optimizer_use_grad_for_differentiable_default_to_fused_or_foreach_differentiable_doc _foreach_doc _maximize_doc)ListOptionalRproprpropcsbeZdZ   ddddddeeded effd d Zfd d ZddZedddZ Z S)r {Gz?g?g333333?gư>2NF)foreachmaximizedifferentiablerrrc sd|ks td|d|dkrdkr|dks-ntd|dd|dt||||||d}t||dS) NgzInvalid learning rate: rg?rzInvalid eta values: z, )lretas step_sizesrrr) ValueErrordictsuper__init__) selfparamsrrrrrrdefaults __class__`/var/www/html/eduruby.in/lip-sync/lip-sync-env/lib/python3.10/site-packages/torch/optim/rprop.pyr s (zRprop.__init__cs@t||jD]}|dd|dd|ddq dS)NrrFr)r __setstate__ param_groups setdefault)rstategrouprr!r"r#%s    zRprop.__setstate__c Cs|dD]q}|jdur q|||j}|jrtd|||j|}t|dkr_d|d<tj|tjd|d<|j j rQ| | t|d|d|d<n| | |d|d<||d||d|dd 7<qdS) Nrz'Rprop does not support sparse gradientsrstepZ memory_formatprevr step_sizer)gradappendZ is_sparse RuntimeErrorr&lentorchZ zeros_likepreserve_formatZdtype is_complexnewZ resize_as_Zfill_complex) rr'rgradsprevsrpr,r&r!r!r" _init_group,s2       zRprop._init_groupcCsd}|durt |}Wdn1swY|jD]8}g}g}g}g}|d\}} |d\} } |d} |d} ||||||t||||| | || | | |dd q |S)zPerforms a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nrrrrr) step_size_min step_size_maxetaminusetaplusrrr)r0Z enable_gradr$r8r )rclosureZlossr'rr5r6rr;r<r9r:rrr!r!r"r(Ps:    z Rprop.step)rrr)N) __name__ __module__ __qualname__r boolrr#r8rr( __classcell__r!r!rr"r s&  $a Implements the resilient backpropagation algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \theta_0 \in \mathbf{R}^d \text{ (params)},f(\theta) \text{ (objective)}, \\ &\hspace{13mm} \eta_{+/-} \text{ (etaplus, etaminus)}, \Gamma_{max/min} \text{ (step sizes)} \\ &\textbf{initialize} : g^0_{prev} \leftarrow 0, \: \eta_0 \leftarrow \text{lr (learning rate)} \\ &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm} \textbf{for} \text{ } i = 0, 1, \ldots, d-1 \: \mathbf{do} \\ &\hspace{10mm} \textbf{if} \: g^i_{prev} g^i_t > 0 \\ &\hspace{15mm} \eta^i_t \leftarrow \mathrm{min}(\eta^i_{t-1} \eta_{+}, \Gamma_{max}) \\ &\hspace{10mm} \textbf{else if} \: g^i_{prev} g^i_t < 0 \\ &\hspace{15mm} \eta^i_t \leftarrow \mathrm{max}(\eta^i_{t-1} \eta_{-}, \Gamma_{min}) \\ &\hspace{15mm} g^i_t \leftarrow 0 \\ &\hspace{10mm} \textbf{else} \: \\ &\hspace{15mm} \eta^i_t \leftarrow \eta^i_{t-1} \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1}- \eta_t \mathrm{sign}(g_t) \\ &\hspace{5mm}g_{prev} \leftarrow g_t \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to the paper `A Direct Adaptive Method for Faster Backpropagation Learning: The RPROP Algorithm `_. a Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 1e-2) etas (Tuple[float, float], optional): pair of (etaminus, etaplus), that are multiplicative increase and decrease factors (default: (0.5, 1.2)) step_sizes (Tuple[float, float], optional): a pair of minimal and maximal allowed step sizes (default: (1e-6, 50)) z z Frr5r6rrrrr9r:r;r<c Csh|dur t||dd\} }|rtjrtd|r"tjs"t} nt} | ||||||| | ||d dS)zpFunctional API that performs rprop algorithm computation. See :class:`~torch.optim.Rprop` for details. NF)Z use_fusedz6torch.jit.script not supported with foreach optimizers)r9r:r;r<rr)rr0ZjitZ is_scriptingr._multi_tensor_rprop_single_tensor_rprop) rr5r6rrrrr9r:r;r<_funcr!r!r"r s& cCs t|D]~\} } || } |s| n| } || } || }t| r4t| } t| } t| } t|}| r@| | }n| | }|||d<|||d<d|| d<| | ||| jtj d} d| | |<| j | |dd| | qdS)Nrrr)value) enumerater0r2 view_as_realmulclonesigngtlteqZmul_clamp_r1Zaddcmul_Zcopy_)rr5r6rr9r:r;r<rriparamr,r*r+rNr!r!r"rDs,      rDcCs^t|dkrdS| rJdt||||g} | D]\\} } } }}dd}|| } || } || } ||}t| | }|rDt|t| | |rQt| | } t||D]}||| d<||| d<d|| d<qZt |||D]}| ||qzt| } tt| D] }d| ||| |<q~dd| D}tj| ||dd qdS) Nrz#_foreach ops don't support autogradcSsdd|DS)NcSs$g|]}t|rt|n|qSr!)r0r2rK).0tr!r!r" szF_multi_tensor_rprop.._view_complex_as_real..r!)Z tensor_listr!r!r"_view_complex_as_realsz2_multi_tensor_rprop.._view_complex_as_realrcSsg|]}|qSr!)rN)rUr,r!r!r"rWIsz'_multi_tensor_rprop..rGrH)r/rZ"_group_tensors_by_device_and_dtypevaluesr0Z _foreach_mulZ _foreach_neg_Z_foreach_copy_Z_foreach_sign_rOrPrQZ _foreach_mul_rRlistrangeZ_foreach_addcmul_)rr5r6rr9r:r;r<rrZgrouped_tensorsZgrouped_paramsZ grouped_gradsZ grouped_prevsZgrouped_step_sizesrErXZsignsrNr+rSZ grad_signsr!r!r"rCs@        rC)NFF)r0rZ optimizerrrrrrr typingr r __all__r __doc__rAfloatr rDrCr!r!r!r"s  p"   9     .   .