In this paper, we present an adjoint-based broadband noise minimization framework using stochastic noise generation (SNG). The SNG module is implemented in the open-source multi-physics solver suite SU2 and coupled with the existing Reynolds-averaged Navier-Stokes (RANS) to allow fast assessment of broadband noise sources. In addition, a discrete adjoint solver on the basis of algorithmic differentiation (AD) is developed for the coupled RANS-SNG system to enable efficient evaluation of broadband noise design sensitivities. The adjoint-based RANS-SNG framework developed in this work not only avoids the regularization problem that plagues the adjoint solutions for scale-resolving simulations, but also significantly lowers the computational cost and leads to a faster turn-around time for the initial design evaluation phase. Current results show that the RANS-SNG method can efficiently provide broadband noise level assessment for various configurations without resorting to computationally prohibitive scale-resolving simulations. Furthermore, current results also show that the design sensitivities computed by the AD-based coupled adjoint-RANS-SNG solver is highly accurate. Finally, shape optimizations performed on the basis of such coupled-sensitivity are shown to be effective in removing the broadband noise source near the trailing edge of a NACA0012 airfoil profile as well as around the side-edge of an isolated flap.