Kurt && Hadoop && Spark

去Spark官網下載要的版本 http://spark.apache.org/downloads.html

目前最新是1.2.1

解壓縮到USB並mount後

cp -r /mnt/spark-1.2.1-bin-hadoop2.4 /opt/spark

設定路徑

vi ~/.bashrc

export SPARK_HOME=$opt/spark
export PATH=$PATH:$SPARK_HOME/bin

source ~/.bashrc

cp /opt/spark/conf/spark-defaults.conf.template /opt/spark/conf/spark-defaults.conf

vi /opt/spark/conf/spark-defaults.conf

spark.master spark://master01:7077
spark.eventLog.enabled true
spark.eventLog.dir hdfs:///user/spark/eventlog

cp /opt/spark/conf/slaves.template /opt/spark/conf/slaves

vi /opt/spark/conf/slaves

master01
slave01
slave02

複製給slave01 slave02

scp -rp /opt/spark slave01:/opt

scp -rp /opt/spark slave02:/opt

cd /opt/spark/sbin

start-all.sh 啟動spark

登入網頁檢查看有沒有啟動

http://master01:8080/