#mktemp

---Lately almost all that I’ve been doing is working on a sophisticated program mainly for my own use at this point.  I love it, but each tiny thing I think about tends to blossom into so many little chores that I can barely get them all done each day.  This function below is one tiny part of the overall program, and is actually much more complicated than it needed to be in the long run.  The reason for this was because I developed it before the thought had occurred to me that it would be a good idea to limit the allowed path-name sizes up front to much less than the common limit of 255 character-long paths in Windows.  I say “common” because it is possible to have longer path names now, but most programs still don’t recognize this fact, so it causes a lot of things to bomb if you allow path-names longer than 255 characters, even though Windows does allow them now, at least on the better file systems.  Anyway, so I just got done today adding in this new lower limit, which would basically keep this function from ever needing to try to strip away at names to make them fit as I’ve done below.  My idea for doing this was that I wanted a way to avoid bombing the program out either by core-dump or via graceful exit in such cases, and I thought this would be kind of fun.  After all, who knows what odd path-names might end up resulting from such nearly-random chopping away at various parts of them!  LOL.  And I felt somewhat proud of how complicated that I’d ended up making it even though the logic still seems correct and it therefore still should work, my undoubtedly putting in way too many “assert” statements just to be sure that I’m not wrong about this, being that what assert does is intentionally bomb the program if the resulting calculation that it’s making ends up being zero.  So as long as all assert statements continually end up with Non-zero values than in them then everything is fine and the program logic is okay.  So what this specific function does is it is an inner part to my own mktemp type of function where a temporary file name (or folder) that has a unique name can be created.  All that this inner function, formtempname, does here then is to come up with a sequence of likely names, and then it is the job of the outer function mymktemp to actually try creating the file/folder using the resulting name from this function.  And as long as it keeps repeating the same calls to this function, like when it needs multiple temp-files all at once, then it will quickly get a set of unique temporary file names having different sequential numbers within their names.  And the reason that I developed this instead of using the normal mktemp or smktemp functions?  Well...frankly...because although I’m compiling on Windows via Cygwin’s source code, I don’t actually INCLUDE Cygwin’s functions while compiling, which allows me to remain free from having to use their DLL file each time I run my programs.  The only drawback to this, however is that this forces me to instead link with the Mingw set of Linux-like functions, and some of them just aren’t really either there or working correctly.  And mktemp and smktemp are examples of some of these functions that just aren’t working in Mingw’s set of supplied functions.  So anyway, doing all of this intense programming has just made me become extremely obsessed with it lately to my admitted detriment, but hopefully I’ll be able to get a little better handle on trying to get some of my other things done as well.  I just get literally like a dog with a bone it would seem when it comes to my working on C programming.  C++ is fun as well but it honestly takes me a bit more time, whereas I’m really in a rush right now to try to develop this program as quickly as I can for myself.  Oh and by the way, the reason that I write so obnoxiously to others with all of my source-code so compressed together like you see here is because I have a horrible memory.  Therefore, I’ve found, that the more of it I can see at once on the screen, the faster and easier it is for me to both write and maintain programs.  It’s just the way I have to do it given my poor memory.  Once something runs of the screen, it just becomes almost instantly vague to me.  So for anyone interested, here is this example of a complicated function that I just wrote for my program a couple of days ago:    :)

This is the final stop on our tour of the lesser known coreutils. If you haven't already done so, you should read part 1 and part 2 first. Once again let's dive in with some cool commands!

mktemp

mktemp will create a temporary file or directory with a unique name. You can also pass in a template and it will safely create a file with a unique name matching it. The template must contain at least 3 'X' characters, but can contain more. The more X characters you use, the more random characters will appear in the name. If you just run the command with no arguments it will use /tmp as the default location and the template tmp.XXXXXXXXXX. If you specify only a template, the file will be created in the current working directory.

$ mktemp /tmp/tmp.FDxW98zfqW $ mktemp sometemp.XXXXXXXXXX sometemp.nlk4ihtGic $ mktemp --tmpdir=/tmp mytemp.XXXXXXXX /tmp/mytemp.ys6mtKKT

As you can see, the command will output the name of the of the temporary file it has just created. You can easily capture this in a shell script using a variable and a command substitution.

$ TEMPDIR="$(mktemp)" $ echo $TEMPDIR /tmp/tmp.eMspLL5dWc $ echo "Hello" > $TEMPDIR $ cat $TEMPDIR Hello

Combining mktemp with unlink can create a way to write data to a file without anyone being able to read it externally. The unlink command will remove the file name from the file system, but if there are still open streams the data on disk remains intact until those streams are closed. The key is to use the exec command to open file descriptors (streams) to the file before unlinking it.

$ TEMPDIR="$(mktemp --tmpdir=/tmp mytemp.XXXXXXXX)" $ exec 5>$TEMPDIR $ exec 6<$TEMPDIR $ echo "Hello to file descriptor" >&5 $ cat $TEMPDIR Hello to file descriptor $ unlink $TEMPDIR $ cat $TEMPDIR cat: /tmp/mytemp.M1kjasnR: No such file or directory $ echo "Hello to file descriptor after unlink" >&5 $ cat <&6 Hello to file descriptor after unlink

After you unlink the file, the two descriptors becomes like a named pipe. Anything you write to fd 5 will be stored until it is read from fd 6. If you try to cat from fd 6 twice in a row, the second read will not return anything, as it was consumed by the first read. This is the best way to store "secret" data in a script without worrying about nosy users peeking at the file.

touch

touch is a command that I am sure most Linux and Unix users are aware of. When used with a file name, it will create an empty file if the file does not exist. If the file does exist, it will update the last accessed and last modified timestamps.

$ ls -la -rw-rw-r-- 1 james james 897 Dec 17 14:49 dart_blog_part_4.html $ touch dart_blog_part_4.html $ ls -la -rw-rw-r-- 1 james james 897 Jun 14 07:32 dart_blog_part_4.html

The reason I bring this up is because of the lesser known options that let you mess with the timestamps of the file. You can change a file to have any timestamp you want, in the future or the past.

# use -t option to set a specific date $ touch -t 400006091148.12 dart_blog_part_4.html -rw-rw-r-- 1 james james 897 Jun 9 4000 dart_blog_part_4.html # use -d or --date to set a specific date with a date string $ touch -d "next tuesday" dart_blog_part_4.html -rw-rw-r-- 1 james james 897 Jun 18 2013 dart_blog_part_4.html # use the timestamp of a different file $ ls -la -rw-rw-r-- 1 james james 897 Jun 18 2013 dart_blog_part_4.html -rw-rw-r-- 1 james james 661 Jan 10 14:00 linux_journal_blog.html $ touch -r linux_journal_blog.html dart_blog_part_4.html -rw-rw-r-- 1 james james 897 Jan 10 14:00 dart_blog_part_4.html -rw-rw-r-- 1 james james 661 Jan 10 14:00 linux_journal_blog.html

ptx

ptx is another one of those interesting tools that I would catalog with tsort (See part 2). It is a very specialized command. It will produce a permuted index, including context, of the words in an input file. Of course it is better shown then described.

$ cat test.txt Roses are red Violets are blue Chocolate is sweet and so are you $ ptx test.txt Roses are red Violets are blue Chocolate is sweet and so are you Chocolate is sweet and so/ Roses are red Violets are blue sweet and so/ Roses are red Violets are blue Chocolate is are blue Chocolate is sweet and so are you /are red Violets is sweet and so are/ Roses are red Violets are blue Chocolate are/ Roses are red Violets are blue Chocolate is sweet and so blue Chocolate is sweet and so are you /Violets are Roses are red Violets are blue Chocolate is sweet and so are/ red Violets are blue Chocolate is sweet and so are you /are sweet and so are/ Roses are red Violets are blue Chocolate is are blue Chocolate is sweet and so are you /red Violets Violets are blue Chocolate is sweet and so are you /are red Chocolate is sweet and so are you /are blue

Now it should be a bit more clear. The words to the right of the break in the center are sorted in alphabetical order. Each word is printed the same number of times it appears in the file, and each word is shown with its surrounding context. There are some options to change how the output is displayed, or to isolate specific words.

# ignore case when sorting, and break on end of line $ ptx --ignore-case -S "\n" test.txt and so are you Roses are red Violets are blue and so are you Violets are blue Chocolate is sweet Chocolate is sweet Roses are red Roses are red and so are you Chocolate is sweet Violets are blue and so are you # same as above, and print only those lines with "are" in them $ ptx --ignore-case -S "\n" -W "are" test.txt Roses are red Violets are blue and so are you

pr

pr can be used to paginate or columnate a text file to prepare it for printing. Basically it will read a plain text file in and add a header, footer, and page breaks. It can also be used to format the plain text file like a magazine article (2 column style). Honestly it does do a nice job, however the 2 column style is a bit quirky.

# Show the first 10 lines of the plain text file $ head -10 dart.txt Dart - A new web programming experience James Slocum [Section: Introduction] JavaScript has had a long standing monopoly on client side web programming. It has a tremendously large user base and countless libraries have been written in it. Surely it is the perfect language with no flaws at all! Unfortunately, this is simply not the case. JavaScript is not without its problems, and there exists a large number of libraries and "trans-pilers" that attempt to work around JavaScripts more quirky behaviors. JQuery, # Format it with pr, and show the first 10 lines $ pr -h "Dart - A new web programming experience" dart.txt | head -10 2013-02-28 10:19 Dart - A new web programming experience Page 1 Dart - A new web programming experience James Slocum [Section: Introduction] JavaScript has had a long standing monopoly on client side web programming.

As you can see in this example, pr has added the requested header info, as well as page numbers and a date. There are a ton of flags to control different header, footer, and page formatting options. One of the strange things about the columnate -COLUMN --columns=NUM behavior is that it truncates the lines instead of reformatting them.

# You can see that it just chops off the end of the line # instead of wrapping it $ pr -2 -h "Dart - A new web programming experience" dart.txt | head -10 2013-02-28 10:19 Dart - A new web programming experience Page 1 Dart - A new web programming experi defined with the class keyword. Eve James Slocum and all classes descend from the Ob single inheritance. The extends key [Section: Introduction] other than Object. Abstract classes JavaScript has had a long standing some default implementation. They c

It's a good thing that we already have a tool to help us with this! We can combine the fmt utility and pr to generate cleaner 2 column text. We simply need to restrict the original text to 35 characters per line.

$ fmt -w 35 dart.txt | pr -2 -h "Dart - A new ... " | head -10 2013-06-12 10:14 Dart - A new web programming experience Page 1 Dart - A new web programming going to need to grab a copy experience James Slocum from dartlang.org. I personally chose to install only the SDK; [Section: Introduction] JavaScript however, there is an option to has had a long standing monopoly grab the full Dart integrated

Now we can clearly see that the columns are wrapping correctly. Now I know most of you might be thinking "Who writes with just plain text? Why is this still a thing?". Well actually I still write a lot in plain text! Vim is my main editor for everything, not just code. So it makes sense to do all of my writing in plain text. If I need to format the text for publications, it is easy to use tools like LaTeX (pronounced lay-tek) or simply copy and paste it into OpenOffice. In fact plain text is the only format pretty much guaranteed to exist and be supported for as long as there are computers.

nl

nl prints out the contents of a file with the lines numbered. cat --number does the same thing, but nl has a ton more options. With nl you can control the style and format of the output. Want to only number lines that match a basic regex? No problem with the -b p<regex> flag.

# number all lines (including blanks) $ nl -b a test.txt 1 Roses are red 2 Violets are blue 3 Chocolate is sweet 4 and so are you 5 6 Roses are the prettiest flower 7 but I like potatoes more 8 they will grow eyes 9 even in a dark drawer # number only those lines with the word Roses $ nl -b pRoses test.txt 1 Roses are red Violets are blue Chocolate is sweet and so are you 2 Roses are the prettiest flower but I like potatoes more they will grow eyes even in a dark drawer

You will have to excuse the bad poetry, as I just needed some simple test content. As you can see the nl utility has a lot of great options to help number the lines in a file.

split, csplit

split can be used to either split a text file into several files based on lines, or a binary file into pieces based on size. This is a great tool for working around the size limitations on email attachments. The two most useful flags for working with binary files are the -b --bytes flags which allows you to specify the size of each partial file, and the -n --number flags which allow you to specify how many partial files you want. You only use one or the other.

$ du -h VIDEO.ts 107M VIDEO.ts $ split -b 10M VIDEO.ts VIDEO_part_ $ du -h * 10M VIDEO_part_aa 10M VIDEO_part_ab 10M VIDEO_part_ac 10M VIDEO_part_ad 10M VIDEO_part_ae 10M VIDEO_part_af 10M VIDEO_part_ag 10M VIDEO_part_ah 10M VIDEO_part_ai 10M VIDEO_part_aj 6.3M VIDEO_part_ak

To reassemble the files, simply cat them back together and redirect the output to the file name you want.

csplit works a bit differently, csplit is used to split a file based on a "context line." The context line can be specified as regular expression pattern.

# This will copy up to but not including the lines with 'Roses' $ csplit test.txt /Roses/ {*} 0 66 99 $ ls xx00 xx01 xx02 test.txt $ cat xx00 $ cat xx01 Roses are red Violets are blue Chocolate is sweet and so are you $ cat xx02 Roses are the prettiest flower but I like potatoes more they will grow eyes even in a dark drawer

The 'xx' prefix is configurable with the -f --prefix=PREFIX flags. So if you want to give each part a more usable name, it's not a problem. The suffix (the numbers in this case) is also settable with the -b --suffix-format=FORMAT flags. Any printf() format can be passed in. See this page for more details on that.

seq

seq will generate a sequence of numbers and print them to standard out. There are three ways to execute this command. The first is just with the final number N. This will produce a sequence from 1-N. The next way is to provide a start S and final N. This will produce a sequence S-N. The last way is to provide a start S, final N, and an increment I. This will produce a sequence S-N, incrementing by I each time.

$ seq 5 1 2 3 4 5 $ seq 0 5 0 1 2 3 4 5 $ seq 0 2 5 0 2 4

seq is rarely used directly and is instead used as part of a command chain or a shell script loop.

# use seq as a counter in a loop to find prime numbers for i in $(seq 0 15); do if [ $(factor $i | cut -d : -f 2 | wc -w) -eq 1 ]; then echo "$i is prime" fi done # Use seq and shuf to generate a pick 6 quick pick $ seq 1 49 | shuf | head -6 | sort -n | xargs printf "%d,%d,%d,%d,%d,%d\n" 2,18,28,33,43,48

timeout

timeout is an awesome command when you need to put an upper limit on the amount of time a program should/can run. You can tell if a command timed out by the return value. A return of 124 indicates that the command timed out. 125 is returned if timeout fails, 126 is if the command cannot be run, and 127 if the command cannot be found. If the command runs successfully it will return the normal command exit status. You can check the return status with the command echo $?.

# cat will wait forever for input when invoked without a file # use timeout force cat to only run for 5 seconds $ timeout 5 cat # This is useful in scripts if you want data from the Internet # but don't want to hang forever $ timeout 5 curl http://jamesslocum.com

timeout also has the -s --signal flags to allow you to control what signal gets sent to the process. The default signal is 15 or TERM. You can change it to 9 or KILL to guarantee a process dies and does not catch or ignore the signal.

Links and further reading

This concludes my tour of the lesser known coreutils. Of course there are dozens of more "well known" coreutils that you can read up on. I highly recommend reading the man pages and info pages as well.