We have developed the ultrasound (US)-guided diffuse optical tomography technique to assist US diagnosis of breast cancer and to predict neoadjuvant chemotherapy response of patients with breast cancer. The technique was implemented using a hand-held hybrid probe consisting of a coregistered US transducer and optical source and detector fibers which couple the light illumination from laser diodes and photon detection to the photomultiplier tube detectors. With the US guidance, diffused light measurements were made at the breast lesion site and the normal contralateral reference site which was used to estimate the background tissue optical properties for imaging reconstruction. However, background optical properties were affected by the chest wall underneath the breast tissue. We have analyzed data from 297 female patients, and results have shown statistically significant correlation between the fitted optical properties ( and ) and the chest wall depth. After subtracting the background at each wavelength, the difference of computed total hemoglobin (tHb) between malignant and benign lesion groups has improved. For early stage malignant lesions, the area-under-the-receiver operator characteristic curve (AUC) has improved from 88.5% to 91.5%. For all malignant lesions, the AUC has improved from 85.3% to 88.1%. Statistical test has revealed the significant difference of the AUC improvements after subtracting background tHb values.