Append data from while-loop into columns

[bastuee]

Good evening,

we use an Ocean Optics HR2000+ spectrometer to record continously spectras of our laser-system. For the spectrometer we use the plug-play drivers, located here

(http://sine.ni.com/apps/utf8/niid_web_display.download_page?p_id_guid=7833BD4A31DA1274E04400144FB7D2...

The intention of our labview-program was to 

• Store in an well defined delay the spectras of the laser

The requirements to the stored file were

• First column: wavelength
• Appended columns: spectrum 
• Header of the column - elapsed time, after starting the program

After considering the forum, one of the solutions which was applied than was to use feedback nodes, together with transposing the data afterwards, as you can see in the attached files.

Our problem:

• for a short run of the vi it works fine
• for a longer run of the vi the file gets messed up (see testlong.txt): in the fifth instance of the loop the instance/wavelength got written again into the spreadsheed-file

Is there anything extremly stupid we have done here ?

[altenbach]

• Since we don't have your instruments, we cannot run your code. It would help if you could attach a smplified version that uses simulated data.
• Your array building is a bit weird. I think you can eliminate the small "built array". I would probably use shift registers.
• "Integration Time []" belongs in a shift register and should be of the correct representation (eliminate he value property nodes!)
• What is a typical value for "time target"?
• I probably would stream each row to disk instead (1 second loop time is plenty). Now you don't even need to accumulate the data in the second feedback node. Why do you want the file data transposed?

[Bob_Schor]

As I understand your basic Instrument, every so often (probably at regular intervals), it gives you a Spectrum as a series of readings (intensity, absorbtion, whatever) as a function of spectral wavelength (or frequency).  Thus at regular intervals you have a 1D array of data (spectrum) that you want to save on disk, and maybe plot as the data come in.

If you consider your multiple samples as a collection of 1D arrays (spectra), the "natural" way to accumulate the samples is to make a 2D array out of them using LabVIEW's Build Array (or the indexing tunnel of a For loop).  However, this builds the arrays with the Spectra arranged in Rows, and you want Columns for your output File.

There are several ways to handle this.  One is to wait until all the spectra are in place, then transpose the entire array, switching Rows and Columns.  For the purpose of this discussion, I'm ignoring the "row and column headers", i.e. the list of spectral frequencies and Elapsed Sample Times.  The disadvantage of this is that nothing gets written until all of the data have been acquired, meaning all of the data need to fit in memory.

Another way is to use Altenbach's suggestion and write the arrays as they are acquired, giving you basically the transpose of the format you wanted.  With this scheme, you write the data to disk as they are acquired, and worry about changing the output format later.

However, there is Another Way available to you (since you are using LabVIEW 2015, which includes the Report Generation Toolkit) -- you could save the data as an Excel Workbook and use the Excel Easy Table to write out each Spectrum as a 2D "Table" -- you start with a 1D array of N spectrum points, use Build Array to make it a 1 Row, N Column array, then transpose this to be an N Row, 1 Column array.  The Easy Table Write also allows you to add Row Headers (which you would make the Elapsed Time) and Column Headers (the Spectral Frequencies, which you only specify for the first Spectrum).  The function also contains "positioning information" so that on subsequent Writes, you can automatically write into the "next column".  We call this "Having Your Cake and Eating It, Too".

Bob Schor

P.S. -- if all you want is the Elapsed Time, just wire False to the Auto-Reset input, ignore (don't wire, leave as defaults) the other inputs (except Error In, of course) and use the Elapsed Time output, which directly tells you how many seconds have elapsed since you first called this function (its initial value, of course, will be 0.00).

[bastuee]

Hi,

thanks for your feedback

• I attached a modified file without the hardware specific vis.(looks probably horrible) Connected the iteration to the header , but i dont understand the behaviour of the program at all anymore
• I want the data transposed, because one day when i will leave this university other people want to continue using this software. And the only solution, the people ( and students :)) can imagine to create out waterfall diagrams is to import such an transposed file into origin . I think the problem would be that with that software its not very comfortable to import 100 files and create such diagrams 

i will try to implement your alternative with the shift-register and the excel based solution. However, i can not find out the failure we made with the feedback-nodes 

[Bob_Schor]

---

@bastuee wrote:

• I attached a modified file without the hardware specific vis.(looks probably horrible) Connected the iteration to the header , but i dont understand the behaviour of the program at all anymore 

---

Neither do I!  Could you please describe what you are trying to model?  Are you attempting to generate 2048 random numbers at one/second and under some condition (which is very unclear to me) write them (transposed?) to a Delimited Spreadsheet File?

Are you also trying to write out the Spreadsheet row and column headers, or are you leaving that out, for now?

Do you want the output file to be a Comma (or Tab) - delimited text file, or do you want an Excel file?

Bob Schor

[bastuee]

1. I create an array with integers from 0-2047 , The line detector has 2048 pixels, which get read out and transformed to a wavelength, which is trivial mathematics
2. Inside the loop i create every second 2048 random numbers , just to have a chaning "spectrum"
3. I connected the iteration number to the header, to better keep track of the things i saved
4. After pressing the stop button the data gets transposed
5. I added an additional flat structure, to get the integer 

The format itself is not important for us/me/other users. Would there be more advantages e.g. for excel than for using a spreadsheet ? I thought it would be beneficial for performance issues ? 

Running the program up to n=13 leads to simulation13.txt. What i dont understand is that the data data is from n=6 to n=13 and the first ones are missing.

When i let the program run for even mor iterations (simulation93.txt) i get

• First column: integer values / wavelength
• N=6 - N=13
• integer values/wavelength
• N=6  - N=95 : N=6 - N=13 differ from the first columns so i guess i am really creating a big mess of stuff here . Even when there are much more easier solutoins than the one with excel which you mentioned, i would just like to know what here is so incredible wrong

Bastian

[Bob_Schor · Accepted solution]

Well, I'm not sure quite why your program went awry -- it was sufficiently complicated that I just re-did it, based on your description.  Here it is, and here's how it works:

The first loop creates the Spectra (starting at Row-to-become-Column 2.  I use "Wait until next multiple" so I can ignore the amount of time required for the computations inside the loop.  As you can see, I use the Elapsed Time output of the Elapsed Time function for the initial Row/Column Header.

When the loop stops, I compute the first Row/Column, sticking NaN (Not a Number) in the first (corner) position.  I'd hoped it would be written as a blank entry in the text file, but it came out as "NaN".  My Write to Delimited Spreadsheet function is the same as yours, but I left the Append to File input unwired so that it retained its default value of False, meaning I create a new file each time.

I tested it for six and 19 seconds, and it produced the expected output (with a period as the Decimal separator instead of a comma, of course).  Try it -- it should do the same for you.

Bob Schor

[bastuee]

Thx. This worked infact very well. Just attached the final version of the combination with the spectrometer.

Will try to find out why the other way does not work anyway.

[altenbach]

There are still many very questionable code constructs.

• Why do you need to convert a 1D array to a 2D array just to get the max value?  Makes no sense.
• The max value gets written to an indicator, but you are reading the value of this indicator in parallel using a local variable. This gives you a race condition because there is no way to tell what happens first. Most likely, the local variable is read first, before the value of the indicator is updated, so your PID calculations are based on a stale value. You should eliminate the local variable and just branch the wire instead.
• Same problem with "black pixel mean". Get rid of the local and branch the wire coming from the control.
• There is a primitive to get the current VIs path.
• What's up with all the unecessary tunnels and ovelapping objects?
• It really is a bad idea to accumulate an evergrowing 2D array at a while loop boundary. If you forget to stop the program, you will eventually run out of memory. Also, if the VI is aborted by accident, you'll lose all data. It would be significantly safer and much more memory efficient if you would stream each new row to disk. If you really need the file transposed, you can convert it later.
• I don't understand the point of the sequence frame.