The depletion of fossil diesel fuel presents a significant challenge due to the rising global energy demand. The waste cooking oil (WCO) usage as a biodiesel resource offers a promising approach to address both the energy shortage and emissions concerns associated with fossil diesel. In this manuscript, we investigate the influence of diesel engine load, WCO biodiesel blend percentage and concentrations of nanoparticles on the diesel engine brake thermal efficiency (BTE) and NOx emissions of a single cylinder diesel engine rotates at 1400 rpm speed. The nanoparticles used include multi-walled carbon nanotubes (MW-CNTs) and graphene oxide (GO), each tested at various concentrations are 50, 100, and 150 ppm, dispersed in the fuel using 4% toluene as a surfactant. The experimental study is designed to predict and optimize BTE and NOx emissions using Response Surface Methodology (RSM). The study examines three cases: (1) without nanoparticles, (2) with MW-CNTs, and (3) with GO nanoparticles. Optimum percentage of BTE is 18.97% at load 2.47 kW, WCO biodiesel blend of 3.63%, and GO nanoparticle concentration of 50 ppm. The lowest NOx emissions of 506.45 ppm occurred at load of 2.40 kW, a WCO biodiesel blend of 16.96%, and a MW-CNT concentration of 100.50 ppm. These results confirm that the addition of MW-CNTs or GO nanoparticles enhances engine performance and reduces NOx emissions, demonstrating their efficacy in improving the sustainability of biodiesel blends.