#parallel-processing

In parallel processing, data is divided among multiple processing units. They either share memory or communicate over a network. This depends on the architecture. Each processor works on a designated part of the data, either independently or through coordinated task scheduling. In many cases, the computed results are aggregated at the end to produce the final output. A key factor in efficient…

How-to: Parallel shell loops #development #solution #fix

Parallel shell loops

I want to process many files and since I’ve here a bunch of cores I want to do it in parallel:

for i in *.myfiles; do do_something $i `derived_params $i` other_params; done

I know of a Makefile solution but my commands needs the arguments out of the shell globbing list. What I found is:

> function pwait() { > while [ $(jobs -p | wc -l) -ge $1 ]; do > sleep 1 > done > } >

T…

Fixed: Is a higher core count or higher clock speed more beneficial to a computer's performance? #computers #solution #dev

Is a higher core count or higher clock speed more beneficial to a computer’s performance?

With lowering silicon costs and rising consumer needs, manufacturers seem to be pushing one of two things: clock speed and/or core count. With the way things are going, it doesn’t seem that clock speed of processors is rising anymore, but the number of processor cores.

I remember only a few years back, I…

Resolved: Parallelizing rsync #fix #it #development

Parallelizing rsync

I just moved and discovered after some trial and error that somewhere between my home and my remote server, there is some throttling going on…but the throttling is not very intelligent. It only throttles individual connections. So if I copy one 1 GB file, it will proceed merrily at 150 kBps. But if I initialize 10 copies, each of them will go at 150 kBps (i.e. I get much…

Is there an equivalent to ‘continue’ in a Parallel.ForEach?

I am porting some code to Parallel.ForEach and got an error with a continue I have in the code. Is there something equivalent I can use in a Parallel.ForEach functionally equivalent to continue in a foreach loop?

Parallel.ForEach(items, parallelOptions, item => { if (!isTrue) continue; });

Answer: Is there an equivalent to ‘continue’ in…

How to wait for all threads to finish, using ExecutorService?

I need to execute some amount of tasks 4 at a time, something like this:

ExecutorService taskExecutor = Executors.newFixedThreadPool(4); while(...) { taskExecutor.execute(new MyTask()); } //...wait for completion somehow

How can I get notified once all of them are complete? For now I can’t think about anything better than setting…

How to: Optimal number of threads per core

Optimal number of threads per core

Let’s say I have a 4-core CPU, and I want to run some process in the minimum amount of time. The process is ideally parallelizable, so I can run chunks of it on an infinite number of threads and each thread takes the same amount of time.

Since I have 4 cores, I don’t expect any speedup by running more threads than cores, since a single core is only capable of…

How to: Difference between concurrent programming and parallel programming

Difference between concurrent programming and parallel programming

I have a question: what is the difference between concurrent programming and parallel programing? I asked google but didn’t find anything that helped me to understand that difference. Could you give me an example for both?

For now I found this explanation: http://www.linux-mag.com/id/7411– but “concurrency is a property of the…