Stephanie Kim - Software Developer

The ransomware anthem. The disk loss story. Avoid all. Make a backup today:

Linux Tape Backup With mt And tar Command Howto

How To Back Up a Web Server

How To Set Red hat / CentOS Linux Remote Backup / Snapshot Server

Debian / Ubuntu Linux Install and Configure Remote Filesystem Snapshot with rsnapshot Incremental Backup Utility

HowTo: Backup MySQL Databases, Web server Files to a FTP Server Automatically

How to compile and install Tarsnap on a Ubuntu/Debian Linux

Required : Spark, Python installed on machine, where the script will be launched.  

Script automatically sets up Apache Spark and HDFS on the cluster for you. For Spark Installation see my previous post.

1. set SPARK_HOME by export $SPARK_HOME=path to bin folder in spark

2. Clone Script Repository from AMPLAB EC2  

3. Older versions of spark (before 2.0) contains the script in SPARK_HOME/ec2 folder

4. Select branch from the git and clone the repository 

5. cd ~ wget https://github.com/amplab/spark-ec2/archive/branch-2.0.zipLink Text

6. sudo unzip branch-2.0.zip -d $SPARK_HOME (unzip to spark home)

7.(move the script to Ec2 folder in SPARK_HOME)mv $SPARK_HOME/spark-ec2-branch-2.0 $SPARK_HOME/ec2

8. (Provide proper owner ship)sudo chown -R ec2-user:ec2-user $SPARK_HOME/ec2 (Most of the time Ec2 user is ubuntu)8.

9.$SPARK_HOME/ec2/spark-ec2 –version (Confirm version)

10. For running the script you need to have .pem file, which is required for sshing to diffrent nodes, 

11. If you are running script on ec2, transfer it using ssh -i filepathofpem ubuntu@DNSNameofmachine:Path

Login to your AWS Management Console and select the Identity & Access Management service.

Create New group by navingating to Groups in the left side menu, and then select Create New Group at the top of the page.

On Step 1. Group Name, set the Group Name to a value like spark-launch and go to the Next Step.    On Step 2. Attach Policy, select AdministratorAccess to grant a pre-defined set of administrative permissions to this Group. Go to the Next Step.    On Step 3. Review, select Create Group. You return to the Groups dashboard, and should see your new Group listed on the dashboard.    Next, click on the name of your group to show summary details. Go to the Users tab and select Add Users to Group.    On the Add Users to Group page, select your IAM user account and then Add Users. You return to the Groups summary detail page, and should see your user account listed in the Group. Your account can now be used to run the spark-ec2 script.

You can add customized policies in place of admin policy.

If you want to use services other then Ec2 you need permissions to those, though you can gain access to those by passing ACCESS keys and Secret Access Keys, its never a good idea, a cleaner way is to identify the roles for each node, and give proper access there rather then sharing keys.  Make sure you provide appropriate roles at the start because once done cant be updated)

Go to the Identity & Access Management service, if you are not already there. Navigate to Roles in the left side menu, and then select Create New Role at the top of the page.

Set appropriate role (Name, type review)

For running the script get the following parameters: a) VPC - Click on Services header and then select VPN in Networking tab in service list(where you find all services offered by AWS) select one from list b) key-pair : Name of your Pem File c) identity-file : Your PEM File d) region : Region is defined in the DNS Name ec2-35-Z-Y-X.us-west-2.compute.amazonaws.com (Here us-west-2 is the region) d) slaves : Specify number of slaves you need, Lets assume we need a cluster with 4 slaves and 1 master) e) specify : instance type : for list of instance type Ec2 for this example I feel m4.2xlarge should be good. f) spark-version :  Specify spark version in our  case 2.0.2 g) zone :  Can be found from Ec2 Instance page, in this case it is us-west-2c h) instance profile -> Name you provide to your cluster

Make sure your pem file has appropriate permission else you will get errors. Cluster hangs in ‘ssh-ready’ state using Spark 1.2.0 EC2 launch script Spark EC2 SSH connection error SSH return code 255 Permission Denied etc

Set permission by chmod 400 Path to .pem file

Export you secret keys and access keys export AWS_SECRET_ACCESS_KEY=AaBbCcDdEeFGgHhIiJjKkLlMmNnOoPpQqRrSsTtU export AWS_ACCESS_KEY_ID=ABCDEFG1234567890123

To get [Access Keys] (http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html)

Takes a little bit of time in sshready state

Run following command

$SPARK_HOME/ec2/spark-ec2 –key-pair=SparkCluster –identity-file=/home/ubuntu/SparkCluster.pem –region=us-west-2 –vpc=vpc-461c2622 –slaves=4 –instance-type=m4.2xlarge –spark-version=2.0.2 –zone=us-west-2c –instance-profile-name=sparkour-cluster launch sparkour-cluster

Other errors due to security group doesnt have proper access Should have port 22 open for SSH , for that go to security group in created instance list and click on security groups on the right column Also make sure inbound rule allows access from the web browser’s location to port 8080. 27, Start spark cluster by selecting master and running : $SPARK_HOME/bin/pyspark –master spark://ip-172-31-24-101:7077

Destroy cluster permanently by $SPARK_HOME/ec2/spark-ec2 destroy clustername

I noticed there is no good visualization of the real Information Security triad, so i made one. You’re welcome. 

Tell Congress to ban the sale of Shark Fins

An estimated 100 million sharks are killed for their fins every year.

Shark Fin Soup is a delicacy in China and other Asian Countries. It is a soup that is flavored with either chicken or some other broth, since the fin itself has little to no flavor. The fin is primarily to add texture to the dish and because it is a delicacy. It is served at special events and can exceed $100 a bowl. The soup is often claimed to have health benefits, such as increasing your appetite, improving your kidneys, lungs and bones. However there is no evidence to support these claims and the reality is that shark meat is barely fit for human consumption. It has a very high level of mercury and the United States Environmental protection agency advises women and young children to stay clear of it.

Because of this dish, an estimated 100 million sharks are killed each year, primarily through a process called finning. Finning is where fisherman take Sharks and they remove the top fin, often times while the shark is still alive. Since there is no “use” and no profit to be made from the rest of the shark, there are thrown back to drown, since sharks cannot breath without having their top fin to aid them in moving through the water.

Because of this Shark populations have dropped between 60 and 90% worldwide in the last 15 years and most experts believe that many shark species will go extinct in the next 10 years.

The largest problem is that, the shark fin trade is a multi-billion dollar industry, which makes it a steep uphill battle to fight. One pound of dried shark fin can sell for $300 or more. While the fishing of Shark in US waters is Illegal, the sale and purchase of shark fins remains legal in the US. In order to protect many shark species from extinctions, we want the US Congress to pass a law outlawing the sale and distribution of Shark fin in the US.