signal express import file size limit

[Oskar_Bosch]

Trying to import CSV file from a scope. Signal express indicates: not enough memory.

This is on windows 7, x64 with 8Gb ram... Not enough???

Basically these files contain 4 scope channels, each 10M (10 million) points, with timestamp.

I was hoping signal express could read this information.

Now it looks like I have to parse it myself in labview or so.

Disappointing.

Does anyone now a work around, or another method to import?

[Raj_V]

Oskar,

This error is usually seen because of insufficient memory when attempting to bring a large set of datapoints into memory. When Signal Express brings CSV data into memory, it converts each data point into a double precision value. Furthermore, depending on teh operations performed inside your SignalExpress application, it may make multiple copies of the data, thus consuming all the memory. One thing to try would be to increase your system's virtual memory to accomodate for this.

http://zone.ni.com/reference/en-XX/help/371361E-01/lvhowto/enable_lrg_ad_aware/ 

Although the article above is for LabVIEW, the same applies for SignalExpress as well.

[Oskar_Bosch]

I forgot to mention that I installed signal express on windows 7, x64 with 8Gb of RAM.

Does all this still apply?

In that same link, it states:

On 64-bit Windows, LabVIEW can access up to 4 GB of virtual memory by default. You do not have to take any action to enable LabVIEW to access up to 4 GB of virtual memory on 64-bit Windows.

Also, when testing to read this in labview, when getting around the 32 bit integer limitation in labview 32bit, by using 2D arrays to store on set of data, it seems I can easily allocate 4 x 10M element 2D arrays (they are basically 10000 x 1000 in size) using single precision.

From your comment, I will see how that works with double precision.

could I force signal express to use single precision?

Message Edited by Oskar Bosch on 04-21-2010 10:58 PM

Message Edited by Oskar Bosch on 04-21-2010 10:59 PM

[Oskar_Bosch]

In that same link, it states:

On 64-bit Windows, LabVIEW can access up to 4 GB of virtual memory by default. You do not have to take any action to enable LabVIEW to access up to 4 GB of virtual memory on 64-bit Windows.

So that seems to imply that the mentioned hints will not work?

[Raj_V]

Oskar,

You are correct in that you wouldn't need to increase the virtual memory in the x64 system since it is already set to 4GB. That being said, I'm wondering exactly what you plan to do with the data once you bring it into memory in SignalExpress. 

You might want to consider disabling some of the additional features of SignalExpress to free up some memory consumption.

Are you able to bring the values into LabVIEW as double precision? If so, then you might want to consider one other route, which is to simply covert the values into single precision and write them back to a CSV file and then open them up in SignalExpress. This will reduce the size of the data being brought in but I can't be sure if it will be sufficient.

[DFGray]

SignalExpress is a 32-bit application, so it is subject to the normal 32-bit limitations, even on a 64-bit machine.  Internally, data is stored as IEEE double floating point (8 bytes/data point) in almost all cases.  In your case, this results in a 320MByte data set.  This should still fit in memory, but there are a couple of complicating factors:

1. SignalExpress assumes that ASCII based files are not very large since ASCII file storage is inefficient.  As such, it trades memory for speed in converting the file to internal binary.  This process makes several copies of the data.  We understand this assumption is flawed, but that is where we are now.
2. Internally, SignalExpress is similar to LabVIEW in that it has data flow from one block to another.  Like LabVIEW, it trades safety for efficiency, so makes frequent copies of data in doing the various analyses it supports.  This leads to memory issues with large data sets.

You should be able to write a custom application in LabVIEW to handle this amount of data, since you can easily tailor the code to your requirements (and you can use 64-bit LabVIEW).  Unfortunately, SignalExpress does not have this luxury.  We apologize for this inconvenience.  If you have any specific questions on how to do this in LabVIEW, post them to the LabVIEW forums.  You should get a quick response.

[Oskar_Bosch]

So I tried in labview to just create 5 arrays of 10million points each: no issues.

Created them using doubles: no issues.

As soon as I try to import this file into signal express: still getting this error: Not enough memory to complete this operation.

Yes, I can write a labview application to read all this.

But how do I export this in a format that signal express can read it efficiently?

The file is a comma delimited file, with 5 columns (time stamp, and then the scope channels).

Total of 10 million rows.

Anyway, so I imported a subset of the file. Seems like it does not have decent logic analyzer capabilities.

Trying to show a time delay between two rising edges (of different signals). Do not see how to do this.

What is the real intend of this application? Just to replace a traditional scope, and nothing related to digital signal analysis (and timing).

Overall not a very easy to use program (admit, have not read the manual).

Yet I was able to use this scope and do it what I want no issues.

[DFGray]

---

But how do I export this in a format that signal express can read it efficiently?

---

The two most efficient file formats that SignalExpress supports are TDMS (via the logging interface) and NI-HWS (via the step interface).  TDMS is a native LabVIEW format optimized for logging.  NI-HWS ships with many of the modular instrument drivers (e.g. NI-SCOPE, NI-FGEN) and was created to facilitate data transfer between the soft front panels.  Which to use depends on whether you would rather import a log (TDMS) or use a step (NI-HWS).  Note that due to problem 2 above (SignalExpress makes several data copies), SignalExpress, in its current form, will not be able to handle your entire data set, no matter how efficiently you read it in.  You will need to experiment with number of points and channels, depending on the analysis you wish to do.

---

Seems like it does not have decent logic analyzer capabilities.

What is the real intend of this application? Just to replace a traditional scope, and nothing related to digital signal analysis (and timing).

---

 SignalExpress does not currently have a full suite of digital analysis tools.  It is relatively easy to find the difference between two rising edges for analog signals (use the cursors), but that capability does not exist for digital signals yet.

The original intent of SignalExpress was to allow easy use of multiple instruments (scopes, DMMs, function generators, etc.) with on-line analysis.  This has been expanded to include logging and playback capabilities for existing data sets.

[Oskar_Bosch]

Disappointing that SignalExpress is not able to handle this large datasets.

The analysis I'm trying to do is to trend the delay between two raising edges, of the full set of collected points.

so using cursors gets me one measurement. But not all of them (I would like to measure about 16K).

The >$15K scope could not do it, so I guess I can not expect signal express to be able to do it.

Anyone knows of a software tool that could help me with this. Not sure if I want to write it all in labview (again??).