Shell Script to curl in parallel
Shell script to curl urls in a file in a parallel manner:
This script can be used to simulate load.
The main script accepts two args.1)input file 2)n.o threads
eg: nohup ./curl_load.sh urls_list.csv 20 &
main script (curl_load.sh)
---------------------------------
#!/bin/sh if [ "$#" -ne 2 ] then echo "Incorrect number of arguments" echo "arguments: 1)input file 2)n.o threads" exit fi line_count=$(wc -l < $1) num_threads=$2 lines_per_file=`expr $line_count / $num_threads` echo "splitting into chunks with each chunk having" $lines_per_file "lines" file_name=`echo $1 |cut -f1 -d.` mkdir ${file_name}_chunks; cd ${file_name}_chunks split -l $lines_per_file ../$1 chunk_
for item in ./*;do
../curl_urls.sh $item & done wait echo "done curl-ing"
child script(curl_urls.sh)
--------------------------------
#!/bin/sh echo "processing $1" for line in `cat $1`; do curl -s $line > /dev/null echo $line >> $1.status done
