The standard practice in histopathology of breast cancers is to examine a hematoxylin and eosin (H&E) stained tissue biopsy under a microscope to diagnose whether a lesion is benign or malignant. This determination is made based on a manual, qualitative inspection, making it subject to investigator bias and resulting in low throughput. Hence, a quantitative, label-free, and high-throughput diagnosis method is highly desirable. We present here preliminary results showing the potential of quantitative phase imaging for breast cancer screening and help with differential diagnosis. We generated phase maps of unstained breast tissue biopsies using spatial light interference microscopy (SLIM). As a first step toward quantitative diagnosis based on SLIM, we carried out a qualitative evaluation of our label-free images. These images were shown to two pathologists who classified each case as either benign or malignant. This diagnosis was then compared against the diagnosis of the two pathologists on corresponding H&E stained tissue images and the number of agreements were counted. The agreement between SLIM and H&E based diagnosis was 88% for the first pathologist and 87% for the second. Our results demonstrate the potential and promise of SLIM for quantitative, label-free, and high-throughput diagnosis.