#large-data

Answer: 24TB RAID 6 configuration #answer #development #dev

24TB RAID 6 configuration

I am in charge of a new website in a niche industry that stores lots of data (10+ TB per client, growing to 2 or 3 clients soon). We are considering ordering about $5000 worth of 3TB drives (10 in a RAID 6 configuration and 10 for backup), which will give us approximately 24 TB of production storage. The data will be written once and remain unmodified for the lifetime of…

Resolved: In Bash, are wildcard expansions guaranteed to be in order? #programming #it #dev

In Bash, are wildcard expansions guaranteed to be in order?

Is the expansion of a wildcard in Bash guaranteed to be in alphabetical order? I am forced to split a large file into 10 Mb pieces so that they can be be accepted by my Mercurial repository.

So I was thinking I could use:

split -b 10485760 Big.file BigFilePiece.

and then in place of:

cat BigFile | bigFileProcessor

I could do:

cat…

How to: How do large companies backup their data? #computers #fix #development

How do large companies backup their data?

How do companies who handle large amounts of data, for example Google or Facebook, backup everything?

According to this Google platform article on Wikipedia, Google has an estimated 450,000+ servers each with a 80+ GB hard disk. That’s a lot of data. Do they really keep 1+ GB of backup for every 1 GB of data?

Answer [by Mark Henderson]: How do large…

Resolved: Transfer 10TB of files from USA -> UK datacenter #solution #fix #development

Transfer 10TB of files from USA -> UK datacenter

I am migrating my server from the USA to the UK from one data center to another, my host said I should be able to achieve 11 megabytes per second.

Operating system is Windows Server 2008 at both ends.

My average file size is around 100MB and the data is split across 5 2TB drives.

What would be the recommended way to transfer these files?

FTP

SMB

R…

Fixed: How do large companies backup their data? #fix #solution #answer

How do large companies backup their data?

How do companies who handle large amounts of data, for example Google or Facebook, backup everything?

According to this Google platform article on Wikipedia, Google has an estimated 450,000+ servers each with a 80+ GB hard disk. That’s a lot of data. Do they really keep 1+ GB of backup for every 1 GB of data?

Answer [by Mark Henderson]: How do large…

Solution: Transfer 10TB of files from USA -> UK datacenter #it #solution #answer

Transfer 10TB of files from USA -> UK datacenter

I am migrating my server from the USA to the UK from one data center to another, my host said I should be able to achieve 11 megabytes per second.

Operating system is Windows Server 2008 at both ends.

My average file size is around 100MB and the data is split across 5 2TB drives.

What would be the recommended way to transfer these files?

FTP

SMB

R…

How to: "Large data" work flows using pandas #computers #fix #answer

"Large data" work flows using pandas

I have tried to puzzle out an answer to this question for many months while learning pandas. I use SAS for my day-to-day work and it is great for it’s out-of-core support. However, SAS is horrible as a piece of software for numerous other reasons.

One day I hope to replace my use of SAS with python and pandas, but I currently lack an out-of-core workflow for…

How to: "Large data" work flows using pandas

"Large data" work flows using pandas

I have tried to puzzle out an answer to this question for many months while learning pandas. I use SAS for my day-to-day work and it is great for it’s out-of-core support. However, SAS is horrible as a piece of software for numerous other reasons.

One day I hope to replace my use of SAS with python and pandas, but I currently lack an out-of-core workflow for…