训练期间无法在 tensorflow 优化器中更改 learning_rate

Question

有人能解释一下为什么我不能在训练期间更改 learningrate 吗，在旧的优化器中我可以用 self.updates.append(K.update(self.learning_rate, new_learning_rate)) 更改它但不能再这样做了 self._set_hyper("learning_rate", new_learning_rate) 没有'不起作用，它告诉我：TypeError:__array__() takes 1 positional argument but 2 were given

class SGD(keras.optimizers.Optimizer):
    def __init__(self, learning_rate=0.01, name="SGD", **kwargs):
        """Call super().__init__() and use _set_hyper() to store hyperparameters"""
        super().__init__(name, **kwargs)
        self._set_hyper("learning_rate", kwargs.get("lr", learning_rate)) # handle lr=learning_rate
        
    @tf.function
    def _resource_apply_dense(self, grad, var):
        learning_rate = self._get_hyper("learning_rate")
        new_learning_rate = learning_rate * 0.001
        new_var_m = var - grad * new_learning_rate
        self._set_hyper("learning_rate", new_learning_rate) #dont work
        var.assign(new_var)

    def _resource_apply_sparse(self, grad, var):
        raise NotImplementedError

    def get_config(self):
        base_config = super().get_config()
        return {
            **base_config,
            "learning_rate": self._serialize_hyperparameter("learning_rate"),
        }

Answer 1

作为解决方法，您可以直接访问使用 _set_hyper 设置的属性，如 documentation 中所述：

Hyperparameters can be overwritten through user code

因为它是一个 tf.Variable，您可以使用 assign 来设置一个新值，您的 tf.Tensor :

class SGD(keras.optimizers.Optimizer):
    def __init__(self, learning_rate=0.01, name="SGD", **kwargs):
        """Call super().__init__() and use _set_hyper() to store hyperparameters"""
        super().__init__(name, **kwargs)
        self._set_hyper("learning_rate", kwargs.get("lr", learning_rate)) # handle lr=learning_rate
        
    @tf.function
    def _resource_apply_dense(self, grad, var):
        learning_rate = self._get_hyper("learning_rate")
        new_learning_rate = learning_rate * 0.001
        new_var_m = var - grad * new_learning_rate
        self.learning_rate.assign(new_learning_rate)
        var.assign(new_var_m)

训练期间无法在 tensorflow 优化器中更改 learning_rate

can't changing learning_rate in tensorflow optimizer during training

tensorflow

tensorflow2.0