o \i"@s>dZddlZGdddZGdddeZGdddeZdS) z'Stochastic optimization methods for MLPNc@s2eZdZdZd ddZddZddZd d Zd S) BaseOptimizera9Base (Stochastic) gradient descent optimizer Parameters ---------- learning_rate_init : float, default=0.1 The initial learning rate used. It controls the step-size in updating the weights Attributes ---------- learning_rate : float the current learning rate 皙?cCs||_t||_dSN)learning_rate_initfloat learning_rate)selfrr /var/www/www-root/data/www/176.119.141.140/sports-predictor/venv/lib/python3.10/site-packages/sklearn/neural_network/_stochastic_optimizers.py__init__szBaseOptimizer.__init__cCs4||}tdd|D|D]\}}||7}qdS)aUpdate parameters with given gradients Parameters ---------- params : list of length = len(coefs_) + len(intercepts_) The concatenated list containing coefs_ and intercepts_ in MLP model. Used for initializing velocities and updating params grads : list of length = len(params) Containing gradients with respect to coefs_ and intercepts_ in MLP model. So length should be aligned with params css|]}|VqdSrr ).0pr r r *sz.BaseOptimizer.update_params..N) _get_updateszip)rparamsgradsupdatesparamupdater r r update_paramss  zBaseOptimizer.update_paramscCsdS)zhPerform update to learning rate and potentially other states at the end of an iteration Nr r time_stepr r r iteration_ends-szBaseOptimizer.iteration_endscCs|rt|ddS)aHDecides whether it is time to stop training Parameters ---------- msg : str Message passed in for verbose output verbose : bool Print message to stdin if True Returns ------- is_stopping : bool True if training needs to stop Stopping.T)printrmsgverboser r r trigger_stopping3s zBaseOptimizer.trigger_stoppingN)r)__name__ __module__ __qualname____doc__r rrrr r r r r s   rcsDeZdZdZ     dfdd Zd d Zd d Zd dZZS) SGDOptimizeraStochastic gradient descent optimizer with momentum Parameters ---------- params : list, length = len(coefs_) + len(intercepts_) The concatenated list containing coefs_ and intercepts_ in MLP model. Used for initializing velocities and updating params learning_rate_init : float, default=0.1 The initial learning rate used. It controls the step-size in updating the weights lr_schedule : {'constant', 'adaptive', 'invscaling'}, default='constant' Learning rate schedule for weight updates. -'constant', is a constant learning rate given by 'learning_rate_init'. -'invscaling' gradually decreases the learning rate 'learning_rate_' at each time step 't' using an inverse scaling exponent of 'power_t'. learning_rate_ = learning_rate_init / pow(t, power_t) -'adaptive', keeps the learning rate constant to 'learning_rate_init' as long as the training keeps decreasing. Each time 2 consecutive epochs fail to decrease the training loss by tol, or fail to increase validation score by tol if 'early_stopping' is on, the current learning rate is divided by 5. momentum : float, default=0.9 Value of momentum used, must be larger than or equal to 0 nesterov : bool, default=True Whether to use nesterov's momentum or not. Use nesterov's if True power_t : float, default=0.5 Power of time step 't' in inverse scaling. See `lr_schedule` for more details. Attributes ---------- learning_rate : float the current learning rate velocities : list, length = len(params) velocities that are used to update params rconstant?T?cs8t|||_||_||_||_dd|D|_dS)NcSg|]}t|qSr np zeros_liker rr r r z)SGDOptimizer.__init__..)superr lr_schedulemomentumnesterovpower_t velocities)rrrr0r1r2r3 __class__r r r xs zSGDOptimizer.__init__cCs,|jdkrt|j|d|j|_dSdS)aPerform updates to learning rate and potential other states at the end of an iteration Parameters ---------- time_step : int number of training samples trained on so far, used to update learning rate for 'invscaling' invscalingN)r0rrr3rrr r r rs zSGDOptimizer.iteration_endscCsd|jdkr|r t|ddS|jdkr|rt|ddS|jd_|r0t|d|jdS) NadaptiverTgư>z# Learning rate too small. Stopping.g@z Setting learning rate to %fF)r0rrrr r r rs    zSGDOptimizer.trigger_stoppingcsDfddtj|D}|_jr fddtj|D}|S)Get the values used to update params with given gradients Parameters ---------- grads : list, length = len(coefs_) + len(intercepts_) Containing gradients with respect to coefs_ and intercepts_ in MLP model. So length should be aligned with params Returns ------- updates : list, length = len(grads) The values to add to params c$g|]\}}j|j|qSr r1rr velocitygradrr r r-z-SGDOptimizer._get_updates..cr;r r<r=r@r r r-rA)rr4r2rrrr r@r rs    zSGDOptimizer._get_updates)rr%r&Tr') r r!r"r#r rrr __classcell__r r r5r r$Hs2r$cs,eZdZdZ d fdd Zdd ZZS) AdamOptimizeraStochastic gradient descent optimizer with Adam Note: All default values are from the original Adam paper Parameters ---------- params : list, length = len(coefs_) + len(intercepts_) The concatenated list containing coefs_ and intercepts_ in MLP model. Used for initializing velocities and updating params learning_rate_init : float, default=0.001 The initial learning rate used. It controls the step-size in updating the weights beta_1 : float, default=0.9 Exponential decay rate for estimates of first moment vector, should be in [0, 1) beta_2 : float, default=0.999 Exponential decay rate for estimates of second moment vector, should be in [0, 1) epsilon : float, default=1e-8 Value for numerical stability Attributes ---------- learning_rate : float The current learning rate t : int Timestep ms : list, length = len(params) First moment vectors vs : list, length = len(params) Second moment vectors References ---------- :arxiv:`Kingma, Diederik, and Jimmy Ba (2014) "Adam: A method for stochastic optimization." <1412.6980> MbP?r&+?:0yE>csHt|||_||_||_d|_dd|D|_dd|D|_dS)NrcSr(r r)r,r r r r-r.z*AdamOptimizer.__init__..cSr(r r)r,r r r r-r.)r/r beta_1beta_2epsilontmsvs)rrrrHrIrJr5r r r s zAdamOptimizer.__init__csjd7_fddtj|D_fddtj|D_jtdjjdjj_ fddtjjD}|S)r:r8cs(g|]\}}j|dj|qS)r8)rH)r mr?r@r r r-sz.AdamOptimizer._get_updates..cs,g|]\}}j|dj|dqS)r8)rI)r vr?r@r r r-cs,g|]\}}j |t|jqSr )rr*sqrtrJ)r rNrPr@r r r-rQ) rKrrLrMrr*rRrIrHrrBr r@r rs"      zAdamOptimizer._get_updates)rEr&rFrG)r r!r"r#r rrCr r r5r rDs . rD)r#numpyr*rr$rDr r r r s ?}