LabVIEW 2018 issues with CrowdStrike

[EricBehrs]

CrowdStrike Falcon is an endpoint protection enterprise software package (i.e. virus scanner run from company servers on all client PCs).  It appears that LabVIEW 2018 executables may not be compatible with CrowdStrike.  I noticed some of my deployed applications at a customer site (I am a LV consultant) were being "quarantined".  The symptom is not pretty:  attempting to run the EXE results in the file being "deleted" from Windows Explorer and Windows showing a dialog implying that the file lacks the correct permissions or otherwise is missing.  In fact, the program is not really deleted (it is recoverable by the IT department running the CrowdStrike admin tools), but to the user it is effectively gone (and of course can't be run).

My point of posting here is:
1. To see if anyone else is running on systems with CrowdStrike deployed and has experienced similar issues (or perhaps is running with CrowdStrike and has not experienced issues?).
2. Assuming the answer to 1 is No (I could not find any existing forum threads mentioning CrowdStrike), to provide a starting point for future folks searching the forums for information about this issue.
3. Perhaps there is a way to build the EXE to avoid the problem, if so I'll update this thread with any information about the workaround.
4. If NI R&D gets involved, provide a place for updates as to any action NI has taken to resolve the issue.

Here is a little bit more detail about what we have discovered so far.
Only LV2018 built EXEs have demonstrated the issue.  I have built identical programs with both LV2016 and LV2017 and they do not get quarantined.  You might think the issue is "What exactly is the program doing?".  But, my test application literally does nothing.  It is an empty VI panel, with an empty diagram.  This single VI is built into an Application using the default options and run.  That's it.  If you do this in 2016/17, no problem (the blank panel launches and is done).  In 2018, it is quarantined.

CrowdStrike has been sent some test applications.  They ran and analyzed the programs and claim that they exhibit "malware-like behavior".  This includes disk access, network access, kernel access, etc. (I won't get into all the technical details now, but might add more in a later post).  CrowdStrike Falcon is "behavior based" protection.  It doesn't do traditional file scanning to find viruses, it actually observes the behavior of the program itself as it starts to run.  The location of the file does not matter.  You can copy the EXE to a different location and it exhibits the same behavior (despite the Windows dialog implying you don't have correct file permission- this is a standard dialog that results from the file "disappearing" from the system).

I have tried this using all the "flavors" of LV2018 (2018, 2018f1, 2018f2, 2018SP1), and all exhibit the same problem.

I have tried disabling VI Server in the INI file.

The LabVIEW 2018 IDE itself seems fine- the issue is only with a built EXE Application using the RTE.  I have also built similar programs into shared libraries (this is something I do quite often in my distributed systems), and these run fine.  It is only the stand alone EXEs that demonstrate the issue.

I am attaching my test project and the built EXEs in 2016/17/18.

Eric Behrs

Behrs Engineering Services

Certified LabVIEW Developer

[TylerC1123]

Hi EricBehrs,

Thanks for bringing this up, I haven't personally worked with or have experience with CrowdStrike but I did have a few questions just to try and get a better grasp.

Have you tried this on multiple machines that have CrowdStrike installed or currently has it been just a single computer? Additionally, it sounds as though you may have already reached out to CrowdStrike, but if not that might be your best course of action at least for right now. Do you know if there is a way to effectively "white list" either EXEs or the RTE within CrowdStrike at least as a potential work around? (Again CrowdStrike themselves might be the best source for this type of info.)

Thanks!

[altenbach]

Not my field of expertise, but I wonder if signing the app or changing the manifest would help?

[rolfk]

---

@altenbach wrote:

Not my field of expertise, but I wonder if signing the app or changing the manifest would help?

---

That would be more something a normal virus scanner would trip over. Not saying that it might not add to the penalty points in CrowdStrike when analyzing an executable but if behavioural analysis is the main culprit, it most be something else.

If I would have to guess it is probably the fact that the actual compiled executable code in the LabVIEW application is still contained in (stripped down) VIs inside the executable. LabVIEW loads the Windows data resource containing the compressed VIs, then decompresses that image and then loads from the VI data resources the compiled object code which is compressed to, links all those binary executable code fragments together and then executes that.

This can look suspiciously like a virus to a behavioural analyzer as they often only contain a small loader stub, which will then load and unpack a binary data blob into memory and likely do all kinds of obfuscation tricks and then the final code relocation and other similar stuff before jumping into the now executable malware code.

Windows does in many ways the same when loading a DLL (it has to) but that must be of course whitelisted somehow in such a tool, or you could not start a single program anymore.

[EricBehrs]

Tyler:

Yes we've tried on multiple machines.  Any PC monitored by CrowdStrike demonstrates the issue.  Yes you can whitelist specific folder paths, or md5 hashes.  But these are just workarounds with limited benefits.  For the folder path option, the whitelisted path is global to the entire organization, so the IT department is going to be very reluctant to do this.  To use the path whitelist option, all developers within the organization would have to be developing from that same folder path on their individual system.  This of course is also a development only solution- once we ship applications to customers (who are running CrowdStrike), they will fail.  For the md5 solution, this would be fine for a specific application distributed within the organization once development is final.  But while still in development, each build generates a new hash so it's not useful. And again, distributing production SW to customers (if their company uses CrowdStrike) would require that they contact their IT department to whitelist it.  Not a very encouraging thing to have to tell them about the SW you are shipping ("trust me, it's fine!").

Rolf, altenbach:

I think I agree with Rolf that the file signing/manifest is not likely to matter (I can't prove that, although I have tried raising manifest level to admin), because CrowdStrike is behavior based, not file content based.  Rolf your explanation of the file unloading process of the VIs at runtime does seem like it's on the right track.  As I mentioned the IT reports show that there are numerous file/dll accesses that take place.  I think the real mystery however is why LV2018 fails and previous compilers did not.  This unloading process has taken place long before LV2018 came along.  We have tried analyzing the error reports generated by CrowdStrike for programs that run and those that don't, and the LV2018 failures do show MORE file access is occurring than the earlier versions.  So this seems like further evidence that this is probably on the right track.

[rolfk]

I'm not sure why you talk about "unloading" VIs. The unloading is not likely any issue for CrowdStrike. DLL loading and file reads should also not be a significant issue. If CrowdStrike considers that bad then it would fall over many applications. Applications not seeming to load any DLL are in fact much more sneaky because you can't really do anything in Windows without linking at least to the platform DLLs either statically or dynamically.

Basically behaverial analysis puts points on various actions and then when a certain limit is reached it blindly triggers.They may have somehow whitelisted LabVIEW applications before in some way or it was sheer luck that they didn't trigger on them, but something in LabVIEW 2018 seems to now account for enough points to trigger CrowdStrike. IMHO it's not exactly a LabVIEW problem, LaVIEW obviously does nothing that is inherently unsafe, or even remotely malicious so CrowdStrike disabling LabVIEW applications is technically a flaw of CrowdStrike. I doubt NI really could do much to resolve this, much of what LabVIEW does has very valid reasons to be done in such a way, and NI is certainly not redesigning LabVIEW because of some virus detection software only to find out that the new version is blocked by some other software again.

The ball really is in CrowdStrikes ballpark and I'm sure NI would work with them if they cared to contact them to give them the details to know how to not falsly trigger on LabVIEW applications.

[GerdW]

Hi,

similar experiences in my company:

Our IT department used to use a virus scanner from TrendMicro and its "cloud" feature GRID. This scanner also deleted ("quarantinized") LabVIEW executables.

I could also pinpoint the problem: my LabVIEW executable was reading an Excel configuration file from our internal server (no problem), but then it was creating a copy of that file in a different folder on the server (apparently bad behaviour for TrendMicro)…

The problem was solved by whitelisting every update of that LabVIEW executable in the TrendMicro scanner.

[KazimirK]

Hi Friends,

I am having this issue too and wonder if any walk-around has been found. Downgrading LV2018 to an earlier version seems to be a sad and painful path. 

Best Regards, 

Kazimir

[KazimirK]

Discovered by accident, enabling debugging while compiling an executable (Advanced -> Enable debugging) somehow prevents the CrowdStrike's atrocious act of removing the file. A minor price - is some inconvenience on the user side to close a pop-up about the missing debugging server. 

Kazimir   

[EricBehrs]

---

@KazimirK wrote:

Discovered by accident, enabling debugging while compiling an executable (Advanced -> Enable debugging) somehow prevents the CrowdStrike's atrocious act of removing the file.

---

We had tried building with and without debugging enabled, but for the apps I tested it didn't make any difference (both were quarantined).

I do have some new info I should have posted earlier, however.  Since CrowdStrike is cloud based, once it is installed it will push out to systems automatically.  I noticed in late March that the quarantines had stopped.  The IT department indicated they had not made any setting changes, but a new CS version had likely pushed out.  The version that was no longer quarantining was 4.23.8603.  I tested this theory in a VM that still had an older ver. of CS- 4.21.8406.  It was still quarantining.  I let the VM get the update to 4.23, and then the same exact LV exe was no longer quarantined.  So, empirical evidence suggests that the issue is resolved in CrowdStrike 4.23 or higher (our current version as of today is 4.26 and we still don't see any quarantines).  It still might be dependent on specific applications and what exactly they are doing, but so far all apps we have tested that were previously quarantined are no longer quarantined.

Although we haven't been able to "close the loop" with CrowdStrike and have them specifically confirm that they addressed the problem with LV2018 EXEs, they had been informed of the issue both by my client, and by NI directly (who we were working with through an SR), so it's possible that they did address the issue (otherwise we can only speculate it was resolved by chance).

Kazimir, I would be interested to know what version of CS you are running?  It would be nice to have some corroborating evidence that 4.23 does in fact resolve this.