#s3cmd

How to Install the latest version of s3cmd tool on Linux

Please install the required packages before installing the s3cmd tool and download the source zip from github:

sudo yum install unzip python-pip wget https://github.com/s3tools/s3cmd/archive/master.zip

Unzip the downloaded source zip file and move to the unzipped directory:

unzip master.zip cd s3cmd-master/

Once you have moved to the unzipped directory then just run this command:

sudo python…

Problem:

Newly created S3 buckets (outside of the US east region) seem to issue 307 redirects, which many libraries do not handle properly.

In particular version 1.0.0 of the `s3cmd` library, gets into a fail loop which is reminiscent of other problems with bucket references:

[Errno 32] Broken pipe

The `aws` perl library also fails but reports an error:

307 Temporary Redirect

Solutions:

- Only create S3 buckets in US Standard region.

- (Untested) Figure out a different hostname/bucketname to refernce when using these libraries.

Some people noticing this:

https://github.com/aws/aws-sdk-ruby/issues/172

Ever wondered how to download a directory from S3 using s3cmd? Here you go.

s3cmd ls s3://yourBucket/path/to/dir/ | awk '{print $4}' | xargs -n 1 -P 10 -I {} s3cmd get {} .

Explanation of xargs: It executes the input gotten from the pipe.  The parameter n says to pass one input argument at a time and P=10 will create 10 threads. -I={} is the way of defining the argument.  This is similar to saying $1 in shell script, sys.argv[1] in python, etc., It substitutes argument where ever you mention {}.

In case you need to get first n files from the directory, lets assume n=5

「CentOS6.3でs3syncが不安定な場合の対処」後もs3syncがBroken pipeを吐きまくることがあって安定しないので、s3cmdを使用してみる。

#yum --enablerepo=epel install s3cmd

#s3cmd --configure →設定ファイル ~/.s3cfg を作成する

以上で設定は完了。以下のコマンドでS3とディレクトリの同期ができる。