cRIO-9056 Read-only file system

[sjwang]

Recently, I found two of my cRIO-9056 encountered file system problem. 

The installed software is NI CompartRIO 19.5, and the output of [uname -a] is;

admin@NI-cRIO-9056-01EF5F64-ECTC1:~# uname -a
Linux NI-cRIO-9056-01EF5F64-ECTC1 4.14.87-rt49-cg-7.1.0f0-x64-41 #1 SMP PREEMPT RT Mon Jun 3 21:14:56 UTC 2019 x86_64 GNU/Linux

As can be seen from the following output of [mount] command, / and /etc/natinst/share were mounted read-only. I also paste the output of [dmesg | grep mmcblk1p4]. It seems that the kernel mount the file system read-only due to the IO error.

NI-MAX connection and the running of RT app seem no problem yet.

Should I run fsck? If yes, what is the proper way to do fsck on root file system?

I am afraid that one of cRIO-9056 in problem was returned from NI after replacing the damaged SDD (or Flash card?).

admin@NI-cRIO-9056-01EF5F64-ECTC1:~# mount
/dev/mmcblk1p4 on / type ext4 (ro,relatime,data=ordered)
devtmpfs on /dev type devtmpfs (rw,relatime,size=448916k,nr_inodes=112229,mode=755)
sysfs on /sys type sysfs (rw,relatime)
proc on /proc type proc (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /var/volatile type tmpfs (rw,relatime,size=234656k)
/dev/mmcblk1p2 on /boot type ext4 (rw,relatime,sync,data=ordered)
/dev/mmcblk1p3 on /etc/natinst/share type ext4 (ro,relatime,sync,data=ordered)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mmcblk0p1 on /media/mmcblk0p1 type vfat (rw,relatime,fmask=0000,dmask=0000,allow_utime=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,quiet,errors=remount-ro)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime)
none on /dev/cgroup/cpuset type cgroup (rw,relatime,cpuset,noprefix,release_agent=/sbin/cpuset_release_agent)
none on /dev/cgroup/cpuacct type cgroup (rw,relatime,cpuacct)

---

---

admin@NI-cRIO-9056-01EF5F64-ECTC1:~# dmesg | grep mmcblk1p4
[ 1.903184] EXT4-fs (mmcblk1p4): mounted filesystem with ordered data mode. Opts: (null)
[1274222.004652] Aborting journal on device mmcblk1p4-8.
[1274232.682667] EXT4-fs warning (device mmcblk1p4): ext4_end_bio:321: I/O error 10 writing to inode 65 (offset 0 size 0 starting block 97281)
[1274232.682674] Buffer I/O error on device mmcblk1p4, logical block 44544
[1274242.925419] EXT4-fs warning (device mmcblk1p4): ext4_end_bio:321: I/O error 10 writing to inode 159 (offset 0 size 4096 starting block 101656)
[1274242.925427] Buffer I/O error on device mmcblk1p4, logical block 48919
[1274253.177682] Buffer I/O error on dev mmcblk1p4, logical block 393216, lost sync page write
[1274253.187175] JBD2: Error -5 detected when updating journal superblock for mmcblk1p4-8.
[1274263.410850] Buffer I/O error on dev mmcblk1p4, logical block 0, lost sync page write
[1274263.419734] EXT4-fs error (device mmcblk1p4): ext4_journal_check_start:61: Detected aborted journal
[1274263.419738] EXT4-fs (mmcblk1p4): previous I/O error to superblock detected
[1274263.437942] EXT4-fs (mmcblk1p4): Remounting filesystem read-only
[1274273.652898] EXT4-fs warning (device mmcblk1p4): ext4_end_bio:321: I/O error 10 writing to inode 156 (offset 0 size 4096 starting block 101655)
[1274273.652906] Buffer I/O error on device mmcblk1p4, logical block 48918
[1274283.895769] Buffer I/O error on dev mmcblk1p4, logical block 0, lost sync page write
[1274283.904650] EXT4-fs error (device mmcblk1p4): ext4_journal_check_start:61: Detected aborted journal
[1274283.904671] EXT4-fs (mmcblk1p4): previous I/O error to superblock detected
[1274283.923109] EXT4-fs (mmcblk1p4): ext4_writepages: jbd2_start: 6144 pages, ino 156; err -30
[1274294.149924] EXT4-fs warning (device mmcblk1p4): ext4_end_bio:321: I/O error 10 writing to inode 160 (offset 0 size 4096 starting block 101657)
[1274294.149932] Buffer I/O error on device mmcblk1p4, logical block 48920
[1274294.157363] JBD2: Detected IO errors while flushing file data on mmcblk1p4-8
[1274304.380949] Buffer I/O error on dev mmcblk1p4, logical block 0, lost sync page write
[1274304.389835] EXT4-fs error (device mmcblk1p4): ext4_journal_check_start:61: Detected aborted journal
[1274304.390184] EXT4-fs (mmcblk1p4): previous I/O error to superblock detected
[1274314.623589] Buffer I/O error on dev mmcblk1p4, logical block 0, lost sync page write
[1274314.632559] EXT4-fs (mmcblk1p4): previous I/O error to superblock detected
[1274324.867475] Buffer I/O error on dev mmcblk1p4, logical block 0, lost sync page write
[1274324.876356] EXT4-fs error (device mmcblk1p4): ext4_journal_check_start:61: Detected aborted journal
[1361182.580365] EXT4-fs (mmcblk1p4): error count since last fsck: 5
[1361182.580371] EXT4-fs (mmcblk1p4): initial error at time 1652409024: ext4_journal_check_start:61
[1361182.580377] EXT4-fs (mmcblk1p4): last error at time 1652409065: ext4_journal_check_start:61
[1447712.687480] EXT4-fs (mmcblk1p4): error count since last fsck: 5
[1447712.687487] EXT4-fs (mmcblk1p4): initial error at time 1652409024: ext4_journal_check_start:61
[1447712.687492] EXT4-fs (mmcblk1p4): last error at time 1652409065: ext4_journal_check_start:61
[1534242.794376] EXT4-fs (mmcblk1p4): error count since last fsck: 5
[1534242.794383] EXT4-fs (mmcblk1p4): initial error at time 1652409024: ext4_journal_check_start:61
[1534242.794389] EXT4-fs (mmcblk1p4): last error at time 1652409065: ext4_journal_check_start:61
[1620772.899536] EXT4-fs (mmcblk1p4): error count since last fsck: 5
[1620772.899542] EXT4-fs (mmcblk1p4): initial error at time 1652409024: ext4_journal_check_start:61
[1620772.899548] EXT4-fs (mmcblk1p4): last error at time 1652409065: ext4_journal_check_start:61
[1707303.011019] EXT4-fs (mmcblk1p4): error count since last fsck: 5
[1707303.011025] EXT4-fs (mmcblk1p4): initial error at time 1652409024: ext4_journal_check_start:61
[1707303.011031] EXT4-fs (mmcblk1p4): last error at time 1652409065: ext4_journal_check_start:61
[1716136.870986] EXT4-fs (mmcblk1p4): previous I/O error to superblock detected
[1716147.077308] Buffer I/O error on dev mmcblk1p4, logical block 0, lost sync page write
[1716147.086196] EXT4-fs error (device mmcblk1p4): ext4_remount:5145: Abort forced by user
[1716389.052431] EXT4-fs warning (device mmcblk1p4): htree_dirblock_to_tree:978: inode #23954: lblock 0: comm ls: error -5 reading directory block
[1716833.979509] EXT4-fs error (device mmcblk1p4): ext4_find_entry:1438: inode #23954: comm mount: reading directory lblock 0
[1716833.991880] EXT4-fs (mmcblk1p4): previous I/O error to superblock detected
[1716844.211254] Buffer I/O error on dev mmcblk1p4, logical block 0, lost sync page write
[1717175.177573] EXT4-fs error (device mmcblk1p4): ext4_find_entry:1438: inode #23954: comm mount: reading directory lblock 0
[1717175.190948] EXT4-fs (mmcblk1p4): previous I/O error to superblock detected
[1717185.420374] Buffer I/O error on dev mmcblk1p4, logical block 0, lost sync page write
[1793833.113331] EXT4-fs (mmcblk1p4): error count since last fsck: 8
[1793833.113338] EXT4-fs (mmcblk1p4): initial error at time 1652409024: ext4_journal_check_start:61
[1793833.113343] EXT4-fs (mmcblk1p4): last error at time 1652851861: ext4_find_entry:1438: inode 23954
[1880363.237054] EXT4-fs (mmcblk1p4): error count since last fsck: 8
[1880363.237061] EXT4-fs (mmcblk1p4): initial error at time 1652409024: ext4_journal_check_start:61
[1880363.237066] EXT4-fs (mmcblk1p4): last error at time 1652851861: ext4_find_entry:1438: inode 23954
[1966893.348392] EXT4-fs (mmcblk1p4): error count since last fsck: 8
[1966893.348398] EXT4-fs (mmcblk1p4): initial error at time 1652409024: ext4_journal_check_start:61
[1966893.348404] EXT4-fs (mmcblk1p4): last error at time 1652851861: ext4_find_entry:1438: inode 23954
[2053423.445451] EXT4-fs (mmcblk1p4): error count since last fsck: 8
[2053423.445458] EXT4-fs (mmcblk1p4): initial error at time 1652409024: ext4_journal_check_start:61
[2053423.445463] EXT4-fs (mmcblk1p4): last error at time 1652851861: ext4_find_entry:1438: inode 23954
[2139953.556282] EXT4-fs (mmcblk1p4): error count since last fsck: 8
[2139953.556289] EXT4-fs (mmcblk1p4): initial error at time 1652409024: ext4_journal_check_start:61
[2139953.556294] EXT4-fs (mmcblk1p4): last error at time 1652851861: ext4_find_entry:1438: inode 23954
[2226483.637769] EXT4-fs (mmcblk1p4): error count since last fsck: 8
[2226483.637776] EXT4-fs (mmcblk1p4): initial error at time 1652409024: ext4_journal_check_start:61
[2226483.637781] EXT4-fs (mmcblk1p4): last error at time 1652851861: ext4_find_entry:1438: inode 23954