The use of mobile phones to conduct diagnostic microscopy at the point-of-care presents intriguing possibilities for the advancement of high-quality medical care in remote settings. However, it is challenging to create a single device that can adapt to the ever-varying camera technologies in phones or that can image with the customization that multiple modalities require for applications such as malaria diagnosis. A portable multi-modal microscope system is presented that utilizes a Raspberry Pi to collect and transmit data wirelessly to a myriad of electronic devices for image analysis. The microscopy system is capable of providing to the user correlated brightfield, polarized, and fluorescent images of samples fixed on traditional microscopy slides. The multimodal diagnostic capabilities of the microscope were assessed by measuring parasitemia of Plasmodium falciparum-infected thin blood smears. The device is capable of detecting fluorescently-labeled DNA using FITC excitation (490 nm) and emission (525 nm), the birefringent P. falciparum byproduct hemozoin, and detecting brightfield absorption with a resolution of 0.78 micrometers (element 9-3 of a 1951 Air Force Target). This microscopy system is a novel portable imaging tool that may be a viable candidate for field implementation if challenges of system durability, cost considerations, and full automation can be overcome.