We prove the optimal regularity for variational problems with convex gradient constraints. We do not assume any regularity of the constraints; so the constraints can be non-smooth, and they need not be strictly convex. When the domain is smooth enough, we show that the optimal regularity holds up to the boundary. In this process, we also characterize the set of singular points of the viscosity solutions to some Hamilton-Jacobi equations. Furthermore, we obtain an explicit formula for the second derivative of these viscosity solutions; and we show that the second derivatives satisfy a monotonicity property.