Currently (LabVIEW 2015) the IMAQ Overlay Multiple Lines 2 or IMAQ Overlay Rectangle, Oval etc do not allow for a "pen" input to change the style (dashed, dotted etc), they only allow for a color input and line width.
One way around this is, to create the overlays using the Picture Functions, make the non-line areas Transparent, and overlay the lines as bitmaps with transparency onto the original image.


I combined some code from the following Community Samples to create this sample:
Community: Convert 2D LabVIEW Picture to IMAQ Image - National Instruments
https://decibel.ni.com/content/docs/DOC-42696
Community: Using Transparency within png files for LabVIEW Vision Development Module - Mario! - National Instruments
https://decibel.ni.com/content/docs/DOC-14565
Although, there are still some issues with this approach. I want to highlight one here to save you time in debugging.
The Unflatten Pixmap.vi function only outputs 24bit values. 8 bits Red, 8 bits Green, 8 bits Blue.
With the Color Box, though you can set some transparency (alpha, the final 8 bits in 32 bit color), which is never output by the Unflatten Pixmap VI.
As a result, you could potentially set some transparency in a Color Box, and thus not ever be able to match your background color to anything output by Unflatten Pixmap because Unflatten Pixmap does not output the full 32 bits. However if you are careful to avoid using 32 bit color in the Color Box (any output with alpha), and stick with 24 bit color, you should be able to avoid this problem.
Example code from the Example Code Exchange in the NI Community is licensed with the MIT license.