Raspbian Cluster SD Card Setup
For smaller pre-configured image, create this setup to be cloned:
1. Change performance (Advanced in raspi-config) to High (1000MHz)2. Change password
3. Change Boot to "CLI" (not graphical desktop); you can always start the graphical desktop with startx
4. Change GPU memory to 16
5. Localization:
- Set Country to US
- Set Timezone to US/Pacific
- Set keyboard to United States, English (US)
6. Grow the partition in preparation for installing software ,but don't expand too much or cloning will be slow!
sudo fdisk /dev/mmcblk0
Welcome to fdisk (util-linux 2.25.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): p
Disk /dev/mmcblk0: 14.9 GiB, 16021192704 bytes, 31291392 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x3e7fd08e
Device Boot Start End Sectors Size Id Type
/dev/mmcblk0p1 8192 131071 122880 60M c W95 FAT32 (LBA)
/dev/mmcblk0p2 131072 8060927 7929856 3.8G 83 Linux
Command (m for help): d
Partition number (1,2, default 2): 2
Partition 2 has been deleted.
Command (m for help): n
Partition type
p primary (1 primary, 0 extended, 3 free)
e extended (container for logical partitions)
Select (default p): p
Partition number (2-4, default 2): 2
First sector (2048-31291391, default 2048): 131072
Last sector, +sectors or +size{K,M,G,T,P} (131072-31291391, default 31291391): 4.6G
Created a new partition 2 of type 'Linux' and of size 4.6 GiB.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Device or resource busy
The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).
7. Reboot
8. Run
sudo resize2fs /dev/mmcblk0p2
The result of df should be something like:
Filesystem 1K-blocks Used Available Use% Mounted on /dev/root 4668808 3522772 901592 80% / devtmpfs 494112 0 494112 0% /dev tmpfs 498428 0 498428 0% /dev/shm tmpfs 498428 6728 491700 2% /run tmpfs 5120 4 5116 1% /run/lock tmpfs 498428 0 498428 0% /sys/fs/cgroup /dev/mmcblk0p1 61384 20328 41056 34% /boot tmpfs 99688 0 99688 0% /run/user/1000
Software
- Temporarily configure in a network if using wireless, otherwise you are on wired and already networked.
- Set up Spark as the cluster manager using the instructions here
After cloning
- Run raspi-config or the GUI Raspberry Pi Configuration tool and "Expand Filesystem"
- Change hostname from colorN to the cluster color and node number for this SD card, e.g., green0, green1, etc.
- Log in as the spark user (su - spark) and edit ~/spark/conf/, setting the SPARK_MASTER_IP and SPARK_WORKER_MEMORY variables, e.g.,:
SPARK_MASTER_IP=192.168.0.120 SPARK_WORKER_MEMORY=512m