Creating a MPI Cluster with the myRIO

[KHartley]

I had built a raspberry pi cluster and after I was done I immediately wanted to build one with a out myRIOs. I figured I'd document the process in case I get side tracked and someone wants to run with the torch or should I get it working then someone will have some steps to follow. I'm interested in the benefits of a cluster have access to a asic like device (FPGA) at each node. I will finally see if the usual means of replication can distribute the image incase users choose not to self compile.

STOP:This document is a work in progress. Save yourself some time. When the process is documented and working I'll remove this label.

I had previously followed several guides and tutorials to get the pis setup so I'll post those for easy reference.

1. http://www.southampton.ac.uk/~sjc/raspberrypi/pi_supercomputer_southampton.htm

2. http://coen.boisestate.edu/ece/files/2013/05/Creating.a.Raspberry.Pi-Based.Beowulf.Cluster_v2.pdf

Useful Links:

1. http://www.mpich.org/downloads/

2. tduffy summary of installing c and c++ https://decibel.ni.com/content/message/65322#65322

Walkthrough:

1. Create a MPI user account, give the admin account a password. The  user account will be what we use to communicate to each node.
   From MAX select Set Permissions which should open the WIF
   >add pasword to the admin by clicking change password
   >add a account "mpirio" by clicking the +
   >add a password of "mpi" for "mpirio"
   >add "mpirio" to the user group
   
   
2. Add a home folder for the mpirio user
   >open ssh as admin
   >cd ..
   >mkdir mpirio
   >chown mpirio mpirio
   >mkdir mpirio/mpich_install
   >chown mpirio mpirio/mpich_install
   
3. Set time in MAX (otherwise you may get warnings about timestamps from the future...)
   >from time settings tab in max, set date/time and time zone
4. edit /etc/opkg/angstrom-base-feed.conf to look like:
   >vi /etc/opkg/angstrom-base-feed.conf
   remove the # to resemble the line below. Use i to insert then delete the "# ". You also need to remove the "us." from the feed as the server name changed. See Jack's document for other feeds supported.
   src/gz angstrom-base http://feeds.anstrom-distribution.org/feeds/next/ipk/eglibc/armv7a/base/
   save the file and exit (escape, shift : (colon), wq, <enter>)
5. remove the lock on libc6
   >opkg flag ok libc6
6. update libc6
   >opkg upgrade libc6
7. Install binutils
   >opkg install binutils
   >opkg install binutils-symlinks
8. install gcc packages (unable to compile simple c code, add install binutils above,
   >opkg install gcc
   >opkg install gcc-symlinks
9. C++ support (required per https://decibel.ni.com/content/message/65322 see BradM's Post)
   >opkg install g++
   >opkg install g++-symlinks
   1. (optional) Fortran
      >opkg install gfortran
      >opkg install gfortran-symlinks
      
10. Install the Standard C library
    note: you must cd to your home directory then use the -t option to specify the directory you are in. This is because the default temp directory is not large enough to download the package to. This may take awhile.
    >cd ~
    >opkg install libc6-dev -t /
    >opkg install libstdc++-dev
    >opkg install make
11. Make sure the stdc++ library is in usr/lib and create a link if its incorrect
    cd ~/../../usr/lib
    ln libstdc++.so.6 libstdc++.so
12. Go ahead and make sure the language of your choice works. A simple hello world should do.
13. Download mpi source
    >mkdir ~/mpich
    >wget ~/mpich/http://www.mpich.org/static/downloads/3.1.1/mpich-3.1.1.tar.gz
14. Unpack source
    >tar xfz ~/mpich/mpich-3.1.1.tar.gz
15. Make build directory.
    >cd ~
    >mkdir mpich_build
16. Configure the build (if you installed fortran drop the disable)
    >cd ~/admin/mpich_build
    >/home/admin/mpich/mpich-3.1.1/configure -prefix=/home/mpirio/mpich-install --disable-fortran --disable-f77 --disable-f90
17. make the build
    >make
18. Install MPI
    >make install
19. Export the path to mpi (is a temporary setting)
    >export PATH=$PATH:/home/mpirio/mpich-install/bin
20. Check for success (should see path to mpi functions)
    >which mpicc
    >which mpiexec
21. Make a testing directory to make sure everything works
    >cd ~
    >mkdir mpi_testing
    >cd mpi_testing
22. Get myrio app and add it to a machine file
    >ifconfig
    >vi machinefile
    >press i and enter ip address 10.2.132.61 in my case, press escape, wq, press enter
23. Test mpiexec
    >mpiexec -f machinefile -n 1 hostname
    >this should output the hostname from MAX
24. Test c code by running the example cpi.c, this program will calculate pi across multiple nodes, rebuilt again and fixed g++ before configuring same error
    >mpiexec -f machinefile -n 2 ~/mpich_build/examples/cpi, tested recompiling cpi.c same error, wrote simple c code using mpicc same error
    >affects python also http://lists.mpich.org/pipermail/discuss/2014-March/002396.html
    >error emailed mpich.discuss
    [mpiexec@NI-myRIO-1900-03037f65] control_cb (/home/admin/mpich-3.1.1/src/pm/hydra/pm/pmiserv/pmserv_cb.c:200): assert (!closed) failed
    [mpiexec@NI-myRIO-1900-03037f65] HYDT_dmxu_poll_wait_for_event (/home/admin/mpich-3.1.1/src/pm/ydra/tools/demux/demux_poll.c:76): callback returned error status
    [mpiexec@NI-myRIO-1900-03037f65] HYD_pmci_wait_for_completion (/home/admin/mpich-3.1.1/src/pm/hdra/pm/pmiserv/pmiserv_pmci.c:198): error waiting for event
    [mpiexec@NI-myRIO-1900-03037f65] main (/home/admin/mpich-3.1.1/src/pm/hydra/ui/mpich/mpiexec.c:44): process manager error waiting for completion
25. currently looking at openmpi
    >cd ~
    >mkdir openmpi
    >mkdir openmpi_build
    >cd openmpi
    >wget http://www.open-mpi.org/software/ompi/v1.8/downloads/openmpi-1.8.1.tar.gz
    >tar xfz openmpi-1.8.1.tar.gz
    >cd ~/../mpirio
    >mkdir openmpi_install
    >cd ~/openmpi_build
    >/home/admin/openmpi/openmpi-1.8.1/configure --CCFLAGS=-march=armv7-a --CXXFLAGS=-march=armv7-a --CCASFLAGS=-march=armv7-a -prefix=/home/mpirio/openmpi_install -q
    >make all install (takes atleast 2 hours compiling on target, note research crosscompiling)
    Results:

  CXXLD    otfprofile

otfprofile-collect_data.o: In function `std::list<StackType, std::allocator<StackType> >::_M_insert(std::_List_iterator<StackType>, StackType const&)':

collect_data.cpp:(.text._ZNSt4listI9StackTypeSaIS0_EE9_M_insertESt14_List_iteratorIS0_ERKS0_[std::list<StackType, std::allocator<StackType> >::_M_insert(std::_List_iterator<StackType>, StackType const&)]+0x3c): undefined reference to `std::_List_node_base::_M_hook(std::_List_node_base*)'

otfprofile-collect_data.o: In function `std::list<StackType, std::allocator<StackType> >::_M_erase(std::_List_iterator<StackType>)':

collect_data.cpp:(.text._ZNSt4listI9StackTypeSaIS0_EE8_M_eraseESt14_List_iteratorIS0_E[std::list<StackType, std::allocator<StackType> >::_M_erase(std::_List_iterator<StackType>)]+0x1c): undefined reference to `std::_List_node_base::_M_unhook()'collect2: ld returned 1 exit status

make[10]: *** [otfprofile] Error 1

make[10]: Leaving directory `/home/admin/openmpi_build/ompi/contrib/vt/vt/extlib/otf/tools/otfprofile'

make[9]: *** [all-recursive] Error 1

make[9]: Leaving directory `/home/admin/openmpi_build/ompi/contrib/vt/vt/extlib/otf/tools/otfprofile'

make[8]: *** [all-recursive] Error 1

make[8]: Leaving directory `/home/admin/openmpi_build/ompi/contrib/vt/vt/extlib/otf/tools'

make[7]: *** [all-recursive] Error 1

This is not supported by NI. Do not contact NI support as they will not be able to help. Feel free to send me (Kyle-H) a pm if you'd like to see this finished or have any ideas why a step is failing.

Issues:

1. When I try to compile c code I see this error. gcc: error trying to exec 'cc1'
   >install cpp, the default angstrom kernel feed placed cc1 into cpp
2. I see a bunch of lines when compiling the mpi source that date happened in the future
   >set date/time in max for system
3. While running the configuration I'm presented with an error that the script couldn't figure out the ar interface.
   

   Opening up the config.log which appears at /home/admin/mpich/mpich-3.1.1/config.log, I found this (see config.log below).

   the configure script expects the ar command to accept the cr and u modifiers which the GNU versions supports.

   http://linux.about.com/library/cmd/blcmdl1_ar.htm
   > install binutils-symlinks

   config.log

   configure:6676: checking for ar

   configure:6692: found /usr/bin/ar

   configure:6703: result: ar

   configure:6729: checking the archiver (ar) interface

   configure:6745: gcc -c     conftest.c >&5

   configure:6745: $? = 0

   configure:6747: ar cru libconftest.a conftest.o >&5

   ar: invalid option -- 'r'

   BusyBox v1.19.4 (2013-06-20 07:58:10 CDT) multi-call binary.

   Usage: ar [-o] [-v] [-p] [-t] [-x] ARCHIVE FILES

   configure:6750: $? = 1

   configure:6756: ar -NOLOGO -OUT:conftest.lib conftest.o >&5

   ar: invalid option -- 'N'

   BusyBox v1.19.4 (2013-06-20 07:58:10 CDT) multi-call binary.

   Usage: ar [-o] [-v] [-p] [-t] [-x] ARCHIVE FILES

   configure:6759: $? = 1

   configure:6778: result: unknown

   configure:6794: error: could not determine ar interface

   
   
4. Examples ended up in mpich-3.1.1 directory
   >make sure to run the configure script while in the build directory
   
5. How do I replicate my system? I tried using dd but now the status light is blinking twice
   >Make sure you run any partition functions from the safe mode as this can and has caused a corrupt os on the linux rt controler. So I did end up really messing up a myrio with this. Don't use dd on the root fs while in run mode.
6. running the /configure script says cannot find iostream.h
   >opkg install libstdc++-dev
   

[BradM]

Ah, I see now. Yes, it is using ar to create static libraries. Install binutils-symlinks

[KHartley]

Brad you are a god! Thanks its well past the breaking point in the configure script.

[BradM]

Glad I could be helpful.