This study primarily presents efficient tools for optimization-based image processing using a bilateral filter (BF). Generally, for image restoration, e.g., deblurring, a forward operation and its adjoint operation pair are required to solve inverse problems via iterative approaches such as the gradient method. Image data comprise millions of variables; thus, the operations should be performed as image filters rather than matrix products because of the considerable matrix size. This approach is known as a matrix-free approach, that is, filter form, because it is executed without explicitly generating an enormous matrix. When BF is incorporated into optimization, its matrix-free adjoint BF is required to solve the optimization problem. This study discusses the matrix-free adjoint BF and its constant-time algorithm to solve optimization problems in a practical time frame. The experimental results demonstrate that the proposed method yields sufficient filtering accuracy for solving inverse problems. Furthermore, BF-based optimization improves accuracy by adjusting the image quality of resultant images.