VI stops saving data to file

[DianeS]

I'm with Steve -- the only thing I can think of that would cause that error is if the bottom loop stopped executing and destroyed the queue.  As a side note, I usually have the "Destroy Queue" function wired to my dequeue loop, not my enqueue loop -- for that reason.  Others, more talented than I, may disagree.

I note that your "stop" button only stops your bottom loop.  Pressing "stop" should stop your entire program, not just half of it.  (Or are you relying on the resulting queue error to stop the upper loop?  I'm not a huge fan of doing it that way, but as I said before, others may disagree.)  For information on how to stop both loops with a single control, search "stop parallel loops" (or similar verbage).  The information you get there should also give you a good idea of how to stop both loops if an error occurs in one of them.

[vickykld]

Diane and Steve,

Thanks to both of you for your responses!

Yes, I get that error when I stop the VI manually by pressing the STOP button in the lower (acquisition) loop.

I pressed it because I noticed that the VI had stopped logging data.

That's exactly why I asked Diane about how to stop both loops if an error occurs. I'll look at threads that describe what you mention. 

But again, the error I get when I manually stop the loop has nothing to do with a file write error, and I don't get any other error messages regarding that. DOes this mean that there is no"write file" error?

Thanks,

Victor

[CoastalMaineBird]

the error I get when I manually stop the loop has nothing to do with a file write error, and I don't get any other error messages regarding that. DOes this mean that there is no"write file" error?

Yes.  The error resulted from your stopping the bottom loop, which killed the queue, but the top loop was still waiting for something in the queue when the queue died.

How are you judging that it stopped logging data?  Are files supposed to appear every X seconds?  

[vickykld]

Hi Steve,

I still did not implement something to stop both loops together. I was researching here on the forum, and got some information, but did not have time to try anything yet.

I know it stopped logging data files for two reasons:

1_ because there is a little clock display of the front panel that tells you the last time it acquired data.This does not match the time of the last data file.

2_ because we have this DAQ system to help monitor our main system, and the latter one is producing data files beyond the time last of the last DAQ file.

So I'm still confused, if there is no write file error, why could it be stopping to write?

Regards,

Victor

[CoastalMaineBird]

1... I think it's perfectly OK to use the error function to stop the upper loop (in contrast to DianeS).  If the lower loop stops, then the upper loop will give an error.  If you check for that specific error code, you can decide whether it was because the lower loop stopped, or some other error, and act accordingly.

2... Another option is to use a boolean indicator (you can hide it if you want), and local variables to it.  Call it "Running".  Set it TRUE before your main loops.  If you get an error in EITHER loop, set it FALSE.  Use RUNNING to control the WHILE loops (loop while running).  You'll have to set a timeout (1000 mSec or so) for the DEQUEUE ELEMENT function, so that the upper loop can check the RUNNING flag every now and then. 

 if there is no write file error, why could it be stopping to write?

I don't know if Windows Server is any different, but I've always been told to stay away from 15k files in one folder, regardless of OS.  The trouble is that when you create a new file, the OS has to search through everything in that folder, to make sure that your new file is not a duplicate name.  I don't know how they do that search, but if it's a linear search, then the search time gets geometrically longer with each file you add.  At some point, you reach a state where the OS is taking all your time, and you have nothing left for yourself. 

  If you can spare the time for testing, try clearing out a folder and using the same file name for every save.  Yes, you lose data that way, but if it doesn't fail, then you know we're on to something.

  Other suggestions:

  Your code does a TRANSPOSE 2-D ARRAY, BUILD ARRAY (where you stick in a new row), and then TRANSPOSE 2-D ARRAY again. 

There's got to be a better way.  That's a lot of shuffling of data.  Use INSERT ROW INTO ARRAY, or write a separate file, or tack it on to the end, or change formats, but find a better way.

  Consider writing into an hourly file.  Put the analog and digital data into a cluster, then write to a DATALOG file.  It's stupidly easy to write a datalog record.  If you really need a spreadsheet later, have another program to read the DATALOG file, and make a spreadsheet out of the records you need.  If you have an hourly file in a daily folder, then you have at most 24 files in a folder, not 15k.

HTH 

[DianeS]

One of the reasons that I don't really like using an error wire to stop a loop is because users tend to lose that warm fuzzy feeling when they hit the "stop" button and every time they do so, an error message appears.  (It also offends my sense of neatness, but that's my problem.)  Of course, you can have an error handler in there that silently handles the error, or something similar, but it's easy enough to just plop a notifier in your code and wrap everything up neatly.  Of course, as Steve said, you can also use a local variable tied to a Boolean, etc.

But back to your actual problem.  I'm seeing a slight disconnect here:

1.  You noticed that the VI had stopped writing data files, so you hit "stop".

2.  You received the error message from the "dequeue" function, which indicates that it was waiting for something when the queue reference was destroyed.

3.  That, of course, implies that you don't have data piled up in your queue, which in turn implies that your upper loop is keeping up with your lower loop.  (I wrote myself a quick VI to test this theory.  When you have items piled up in your queue and you kill the queue before "dequeue" has worked through all of the items in it, you get a different error message.)

4.  If your upper loop is keeping up with your lower loop, then that indicates that it's not taking forever to write your files.  Why?  Because

        a) if it weren't, you'd have data piling up in your queue and therefore you wouldn't get a message that said the queue was waiting for more data when you killed it; and

        b) in order for the dequeue function to be waiting for something, the upper loop had to have completed execution of all of its subVIs (i.e. performed the file write VIs).  If you were stuck in one of your file writing subVIs (i.e. it was taking an excessively long time to write the file, as has been rightly postulated), pressing "stop" would have stopped the lower loop but your upper loop would have continued trying to write the file for as long as it took to do so.  In other words, both of your "write file" subVIs would have to be finished executing before your upper loop could stop. 

So...how long did it take, after you pressed the "stop" button, for the error message to pop up?  Was it instantaneous, or did it take awhile? 

Steve, if there's something nutty about my thought process, please speak up.  It's perfectly possible that I'm just missing something here.

[CoastalMaineBird]

One of the reasons that I don't really like using an error wire to stop a loop is because users tend to lose that warm fuzzy feeling when they hit the "stop" button and every time they do so, an error message appears.

Of course not.  Part of the "act accordingly" that I mentioned would be to swallow the error that you know about and not complain.  I use that technique all the time in TCP operations to swallow a timeout error, which in my case is not really an error, just a condition.

In any case, the only thing possibly (*possibly*) missing from your analysis is the fact that the CPU is not all yours.  If Windows has an attack of indigestion on the 15k files, it is likely stealing time from you.  What that does to LabVIEW's task scheduler I don't know; but it won't be pretty.

Still, your 4A seems spot on, unless the STOP button is hit while Windows is digesting.

[vickykld]

Hi Steve and Diane,

Thanks for your responses!! I was out of the office on Friday, so didn't get a chance to see them.

I'm learning a lot from your comments and suggestions. Thanks!!! I'm sure you can tell I'm no LabView expert.

In response to Diane, the error occurs immediately after I press the STOP button. There is no delay.

Thanks for setting up a test vi, before deploying this VI in the field I did some tests of it for a few hours, and did not get any sort of errors either.

I'm getting convinced that Steve is correct on suggesting that the tremendous amount of files in the folder has something to do with this.

I will look into the DATALOG file format you suggest, never did that or know exactly what it's like. Maybe you can give me some pointers?

I can't (don't wish to) group an hours worth of data into one file, or the analog and digital into one file. We want to keep them separate. But it seems to me that this DATALOG file format is smaller and quicker to save, correct?

Is there a way to automatically tell LabView to create a new folder every hour or so?

I also agree that transposing the data twice is not optimal, but I did not find another way to insert the "transitions" data into the first column of the array.

Best regards,

Victor

[CoastalMaineBird]

I will look into the DATALOG file format you suggest, never did that or know exactly what it's like. Maybe you can give me some pointers?

It's a binary format with a built-in header (that you don't deal with).  You OPEN/CREATE DATALOG FILE with a cluster that defines the type of file you want, you WRITE to it using a cluster of that type, and CLOSE it.  When you read it, you OPEN it, READ it, and CLOSE it, and you have one of your clusters, however complex that might be.

I can't (don't wish to) group an hours worth of data into one file, or the analog and digital into one file. We want to keep them separate. But it seems to me that this DATALOG file format is smaller and quicker to save, correct? 

Yes.  You don't have to convert your numbers into text strings, with tabs and line breaks, and all that.  When reading, you don't have to convert text into numbers. 

It's smaller in all cases except where your cluster is small to begin with.  The header takes a finite amount of space, so storing a 4-byte integer might take 100 bytes.  But storing a 4,000,000 byte structure might take 4,001,000 bytes.

Is there a way to automatically tell LabView to create a new folder every hour or so?

 No.  That's a programmer's job.

If Current Hour < > Previous Hour

    Current Path = MakePathName (CurrentDay,CurrentHour) 

    Create Folder (Current Path)

    Previous Hour = Current Hour

end if

I also agree that transposing the data twice is not optimal, but I did not find another way to insert the "transitions" data into the first column of the array.

Try INSERT INTO ARRAY function on the ARRAY palette. 

Message Edited by CoastalMaineBird on 03-23-2009 11:08 AM

[vickykld]

Hi Steve,

I found a way to create a Directory in LabVIEW, using "NEW". I implemented changes to save the files in separate timestamped folders, one per event.

In tha way, I'll have like 20 to 30 folders per day, each containing about 1000 files.

I'll test this and let you know what happens.

I still did not implement the DATALOG file or the transposing, I'll do that soon.

Thanks again!!

Victor