Imaq: how to decode PNG

[ThHagel]

Hello Greg,

I found out that your VI is only slow at the first call. Calling it in a loop, the first call with 8bit image takes about 400ms and every following about 15ms on my PC. That is ok, at least I could display received images fast enough, if they would be 8bit of course (unfortunately they are 16bit images and I still can not convert them) ...  

[GregSands]

OK, it looks like the .NET function has 4 bytes of data for both I16 and U16 (which was also the source of the errors before - don't know why I ever thought it was working).  Fixing that up gets a 16-bit image returned correctly, but the values are different to those in the original image.  However, the image seems to read in incorrectly in the first place (it looks like possibly an endian issue??) so perhaps Vision has some problems with 16-bit images anyway.  I've tried massaging the values that come back, but I don't really know enough to do anything sensible - perhaps you can work something out.

Regarding the speed, it seems that the first two .NET calls take roughly 25% and 75% of the time respectively - I don't know enough about .NET to speed them up, but perhaps it's is possible to re-use the MemoryStream or Decoder. I've changed the decoder caching to OnLoad, which doesn't reduce the overall time, but it does put all the time into that function rather than the following one.

Cheers ~ Greg

[GregSands]

A quick follow-up - it seems that there are errors both in the reading of 16-bit PNG files using IMAQ Vision 2009, and in the decoding using the .NET routines.  A PNG reference is found at http://www.schaik.com/pngsuite/pngsuite.html and if the 16-bit images from here are read into Vision, they do not display correctly.

On the .NET side, if I create a 256x256 image with smoothly varying values (-32768 to 32767), then it's obvious that the results of the PNGDecoder are no longer smoothly continuous.  So problems all round, and perhaps you're better not to use 16-bit PNG images at all!!

Cheers ~ Greg

[rolfk]

---

GregS wrote:

A quick follow-up - it seems that there are errors both in the reading of 16-bit PNG files using IMAQ Vision 2009, and in the decoding using the .NET routines.  A PNG reference is found at http://www.schaik.com/pngsuite/pngsuite.html and if the 16-bit images from here are read into Vision, they do not display correctly.

On the .NET side, if I create a 256x256 image with smoothly varying values (-32768 to 32767), then it's obvious that the results of the PNGDecoder are no longer smoothly continuous.  So problems all round, and perhaps you're better not to use 16-bit PNG images at all!!

Cheers ~ Greg

---

The problem might be that .Net does inherent scaling (or maybe a complete lack of that) from whatever the image is to the 32bit it returns. Or a mismatch between the bitdepth of the image, what .NET does, and what you tell IMAQ Vision to treat it like.

In general since LabVIEW has strict defined datatypes you can't expect to assign 16bit values to 32 bit values without some conversion. The LabVIEW types system is great for numeric operations, where you do want to preserve the numeric value as much as possible, but not so ideal for image data, where you usually have to scale whatever datasize you get into whatever datasize you want the image to be. And in IMAQ you tell at creation of an image reference what type it is and if you do not pass it that type of data to fill into the image then things get skewed.

Message Edited by rolfk on 11-05-2009 08:35 AM

[ThHagel]

Hi Greg,

great work! That looks very close to what we are looking for!

I will try some modifications .....

[ThHagel]

Hi Greg,

I think the 16bit information is encoded to the png correctly. I made some tests a longer time ago using the imaq WriteString.vi to encode to png, then I saved that data to a png-file (write binary file) and then I was able to load it back with the Imaq Read File. The read image contained all data as expected. So the Imaq encoding seems to be ok for me. I don't think that's the problem.

It seems that the .net decoder converts the data to unsigned integers. This is why the image in your last Test VI looks cut in half and shuffled. Now here comes a good message: for the use of displaying an i16 image your VI works well with a little modification. Simply subtract 32768 from each returned pixel value and the image looks very close to the original one. It doesn't contain more than 8bit data and your grayscale array image shows that it doesn't work 100%, but it looks good enough:

 

This is a big leap! Check out the modified attached VI.

Concerning the assumed 16bit data loss, I found out that an important parameter for your .Net approach is the PngBitmapDecoder constructor parameter "createOption".

If this one is set to "None" the .net conversion routine decides what is the best way to decode it. I guess it optimizes the image data for viewing on the screen. Because it is only possible to display 8bit grayscale with the computers grafix it seems that the PngBitmapDecoder decides to reduce the data to 8 bit.

If you set the createOption to "PreserverPixelFormat" the conversion routine still mixes up things but it returns higher resolution data. So I guess that parameter must be set to "PreservePixelFormat" and maybe we can find out how to setup the conversion not to scramble the image data. Playing around with the stride parameter delivered some interesting effects, bat I was not able to crack that nut.

Try the Test-VI with createOption set to "PreservePixelFormat" and you will see the effects.

So far so good. Thanks again. If you have any news, let me know.

Greetings,

Thomas

[ThHagel]

Hi Greg,

we made it! Your approach works fine now with 8, 16 and 32 bit images. Even it's performance is acceptable (I measured 25 or less ms conversion time).

Some bit level manipulation, adjustments for the PixelArray size and changes for the stride calculation was necessary to fix it. There was a memory leakage too, that had to be removed using some close reference nodes.

I also tried to get things run faster by keeping references open but I encountered some problems. Unfortunately the only object reference that can stay opened is the MemoryStream, all other have to be constructed in each conversion cycle. Otherwise conversion doesn't work (please don't ask why, my .net experience is 0). I tried to optimize in different ways, but I had no luck (it never really increased the maximum speed). Instead, after I recognized that memory leakage, I decided to close all opened references after each conversion cycle. That's the safest way I think and the routine didn't get any slower.

It works perfect for me. Thank you so much for your support.

Great job!

Please try the attached  VI's to see that it really works.

Best regards,

Thomas

[GregSands]

Hi Thomas --

Yep, looks like setting createOption to PreservePixelFormat gets most of the way there.  There still seems to be an offset - sometimes it's zero, but for other 16-bit images it's up to ~2000, and there's an offset of 7 for your 4kSnake.png image. But you're right, it gives 16-bit output. The "Stride" parameter appears to give the number of Bytes between rows of the image.  So for U8 it's set to the width, I16 and U16 are 2 times the width, and RGB now 3 times (not 4).

Doing this stuffed up the RGB conversion (now with no Alpha channel by the looks) - just needed to be split into individual bytes and then recombined.  The only other issue I can see at the moment is that U16 images are rescaled to fill the range 0 to 65535 - but noone uses those anyway as they're so poorly supported!

There are still problems with reading 16-bit PNG files that have been created using other software, but it all seems to work fine for PNG files written originally from LabVIEW.

Cheers ~ Greg

[GregSands]

Looks like I missed your post while working on mine!  Had to fit it in around other work you see.  But yes, it looks as though we've ended up with much the same solution - the only problem with yours might be if the area of a U8 image is not divisible by 4 (or 16-bit images by 2) then you'll crop off the last pixel or two.  The way I've done it uses data of the correct representation (except RGB which is a little excessive).

Amazing what we can do between us with no knowledge of .NET!!

Cheers ~ Greg

[ThHagel]

Hi Greg,

I have good and bad news.

First the good or at least neutral ones:

1. You are right, I would lose pixel data in case the image size is not divisible by 4 or 2. So I dumped my VI and continued working on yours.

2. If you call your VI in loop it leaks memory. That was already known and easy to fix with close reference nodes. 

3. For 32bit RGB images I prefer to let the .net decoder do the number crunching for us. Therefore I set the createOption-Input to "None" in for 32RGB's. I really like your way of unscrambling data for the 32RGB, but I guess we can stay with the .net conversion and hope that it maybe also provides support for the alphachannel (I didn't test it and I don't know if someone needs it, but we get the same result so we don't lose anything).

4. With this small changes on your VI, I was happy for about an hour.

Then I found out the following (that's the bad news part):

the i16 and u16 decoding doesn't work correct for images with less than 32768 pixels! I have no clue why and I didn't find a way to solve this. I guess this is an issue inside the .net routines. Check out attached VI Test_32768. It allows to change image size during runtime and compares one pixel value of the two images. 

To check that this problem is not hidden on the encoder side, I stored the FileString in a PNG-File and reloaded it using the IMAQ Read File. But this test only approved that encoding works well for any image size, so the problem is on the .net side for sure.

I'm a bit exhausted with this problem. I hope you still have some motivation left.

I'll keep on trying different things and I'll let you know if I find out something.

Greetings,

Thomas