10000 hours

On June 25, 2018, I discovered a vulnerability with the mongoz server (https://github.com/yandex/mongoz). The vulnerability allows for a resource exhaustion attack against a mongoz server. Details are outlined below:

Issue:

A standard mongoz server listening on a port will listen for client connections. When a client connects to the server to perform a command, it is possible to have a client send data to the server such that the server keeps the connection open. The connection’s lifetime is then controlled by the client.

The result is that the connection will stay open, and only when the client closes it will the resources on the server side get cleaned up. If we extend this activity, such that many clients connect to a mongoz server, then it will cause a denial of service.

Possible Solution:

Introduction

So it’s been a while since I’ve posted an entry. I’ve been really busy with learning a lot of stuff, but I hadn't had the time to post these new things that I’ve been learning, until now! I’ve started taking the PenTesting With Kali course available from Offensive Security, and I am looking forward to taking the OSCP exam when the time comes. The following post, and future posts will talk about what I’ve been learning through the PenTesting With Kali course.

Active Information Gathering

After we have performed our passive information gathering, we can start with some active information gathering on our target. The goal of the active information gathering phase is to identify weaknesses in our target. During the active information gathering phase, we can do the following:

Enumerate visible hosts in our target’s network.

Perform port scanning on visible hosts and enumerate running services on each host.

Perform vulnerability scanning.

DNS Enumeration

When computers refer to a host, they do so using a 32-bit number, however humans have a hard time referring to hosts this way, so humans have devised a way to map these 32-bit numbers (host IP addresses) to their human-readable names. DNS (domain name service) allows users in a network to resolve domain names (i.e www.google.com) with the IP address that corresponds to that domain name. Because DNS servers are the authority of the IP address mapping to domain names, we can leverage this information in order to find possible hosts to attack.

DNS enumeration can give us information about the available public and private servers of our target. With DNS enumeration, we can obtain server IP addresses, server names, and server functionality. 

We can interact with DNS servers using the following tools:

host 

dig

nslookup

For example, we can show a list of DNS servers that serve a domain, using the following command: 

host -t ns adomain.com

This second example can show us mail servers that serve a domain, using the following command:

host -t mx adomain.com

Forward DNS Lookup

We can do a forward dns lookup in order to, for example, determine if the target has a web server. The following example shows how it’s done using the host command:

host www.adomain.com

Note that we can guess more hosts using the command shown above using a batch script and giving it an input of a list of common host names. Common host names can be:

www

ftp

mail

owa

proxy

router

admin

www2

firewall

mx

pop3

Note that there may be more IPs that are not obtained using the forward dns lookup method. We can use reverse DNS lookup to find more IPs.

Reverse DNS Lookup

If the DNS server was configured to use PTR records, we may be able to get more IP addresses that were missed during the forward DNS lookup. Using the IPs retrieved from performing a forward DNS lookup, examine the range of IPs. Then, perform a reverse DNS query for each IP within that range. If an IP resolves to a name, then it is valid.

DNS Zone Transfers

A DNS Zone transfer is a bit like a database replication. When we do a DNS zone transfer, we are basically “slurping” up what the DNS server knows. We do this by requesting the DNS server to give us the contents of its zone file.

Q: What is a zone file?

A: It contains the mapping of IP address to host names for which the DNS is an authority.

Q: What is an example of bad DNS server configuration?

A: Mixing together the internal and external DNS namespaces; Allowing for DNS public zone transfer requests.

We would request for a DNS zone transfer by the following method:

List the DNS servers available: host -t ns adomain.com

Trigger a zone transfer from a DNS server: host -l adomain.com ns2.domain.com

Looking Back At the General Proof-of-Concept Process

I intended on creating shellcode that would just run calc.exe, just to make things as simple as possible. In this post, I will describe the issues that I ran into, and how I solved them.

I found a good tutorial on shellcode at ProjectShellcode and I used "Shellcode Tutorial 3: Windows Command Execution Shellcode". The following assembly code was my first attempt:

;Calls calc.exe. Modified from http://projectshellcode.com/node/20 [SECTION .text] BITS 32 global _start _start: jmp short GetCommand ;Jump to location of command string CommandReturn: ;Define a label to call so that string address is pushed onto stack pop ebx ;ebx now points to the string xor eax,eax ;clear eax push 3 ;push showwindow param onto stack push ebx ;push command string onto stack mov ebx,0x7c86114d ;place winExec address into ebx call ebx ;call WinExec({cmd},0) xor eax,eax ;remove return value of WinExec push eax ;push null onto stack as parameter value mov ebx, 0x7c81caa2 ;place ExitProcess address into ebx call ebx ;call ExitProcess(0) GetCommand: call CommandReturn ;Call the return label so the return address (location of string) is pushed onto stack db 'cmd.exe /c calc.exe',0 ;write direct bytes of the command string

The Issue

When I tested out the shellcode in the test vulnerable application (which I mention in my last post, called "Buffer Overflow Proof-of-Concept: Part 1"), I noticed that every piece of the above assembly code gets put into the buffer EXCEPT for "/c calc.exe". So, what this means is that the command string that was given to WinExec to run was "cmd.exe", instead of "cmd.exe /c calc.exe".

 The Process of Getting a Working Shellcode

So, now I was left wondering what the issue could be. First, I needed to ask myself if the actual shellcode worked OUTSIDE of the environment of the test vulnerable application. Because I had tested the assembly code by creating a C program that calls a string (which was the hex values of the opcodes of the shellcode assembly) like a function, this was easy to answer - "Yes! the actual shellcode does work". Second, I needed to ask myself if the command string was too long and somehow getting cut off. The answer to this was that the command string was actually ok, and that there was still space left in the buffer of the test vulnerable application for more characters. So, the issue was something else entirely... 

In general, I find it quite helpful to write about the issues that I am currently facing. It helps me organize my thoughts on what I know of the problem, and what I may need to look at next. So, the next thing that I did was write on Security StackExchange, with the post "In buffer overflow exploit, 0x20 (space character) not appearing in memory and is replaced by null". 

Lo, and behold! The title of my question on Security StackExchange gave me the idea to ask myself: How is the test vulnerable application taking input? The answer is that I created the test vulnerable application to take input by using the fscanf function. The fscanf function will take one character at a time from an input stream until it sees a whitespace. So, in the case of the opcode for the shellcode above, everything is fine, until the test vulnerable application gets to the 0x20 character in the last portion of the shellcode. 

So, I really wanted to get calculator to pop up (next up, I will see if I can use metasploit's encoder to give me a calculator without doing my workaround). What I did was modify the shellcode in the following way:

The Working Shellcode

;Calls calc.exe. Modified from http://projectshellcode.com/node/20 [SECTION .text] BITS 32 global _start _start: jmp short GetCommand ;Jump to location of command string CommandReturn: ;Define a label to call so that string address is pushed onto stack pop ebx ;ebx now points to the string xor eax,eax ;clear eax push 3 ;push showwindow param onto stack push ebx ;push command string onto stack mov ebx,0x7c86114d ;place winExec address into ebx call ebx ;call WinExec({cmd},0) xor eax,eax ;remove return value of WinExec push eax ;push null onto stack as parameter value mov ebx, 0x7c81caa2 ;place ExitProcess address into ebx call ebx ;call ExitProcess(0) GetCommand: call CommandReturn ;Call the return label so the return address (location of string) is pushed onto stack db 'calc',0 ;write direct bytes of the command string

 Note that in the last line of this working shellcode, I use plain old "calc" as input to WinExec.

Introduction

I wanted to gain some hands-on experience with developing a proof-of-concept for a buffer overflow type of attack. 

Buffer overflow attacks basically come about when an application has a buffer (a block of memory set aside) in which input data gets stored that is larger than the buffer that is to hold that data. This results in the ability of an attacker to gain control of the application's flow of instruction, and to have the ability to redirect the application to run the attacker's own code.

I wrote a simple test application that contains a buffer overflow vulnerability to test out the process of gaining control over a vulnerable application, and this post describes how I did it.

Background information

The basic idea of how a stack works is the following:

The stack data structure is like a stack of plates. We are limited to what we can do with this data structure. When we want to store data onto the stack, we can only "push" data (like the plates analogy) onto the top of the stack. When we want to retrieve data, we can only "pop" from the top of the stack. So, the stack data structure is a "last in, first out" sort of structure.

In an application, the stack is used to be able to keep track of needed data that each function uses when it runs. In terms of memory location, the stack grows from higher memory address to low memory address. In other words, the stack grows downward in the application's process space.

The top of the stack is tracked by a register called "ESP" (Extended Stack Pointer). Whenever data gets pushed onto the stack, the ESP register's value gets decremented, and whenever data gets popped off of the stack, the ESP register's value gets incremented.

During a call to a function, memory on the stack is set aside for a few pieces of data. The called function's parameters, the address of the next instruction to be run once the called function returns, the "base pointer", and local variables are all saved onto the stack.

Once the function that has been called has finished, this function needs to clean the stack by removing the data that was allocated for it. It does this by decrementing the ESP register and placing the address of the next instruction to be run into a register called the "EIP" (Extended Instruction Pointer). 

Vulnerable applications will take data and store this data into a buffer without doing any checks on whether the data will fit the memory allocated to store that data. Without any bounds checking, an attacker will be able to feed the vulnerable application with a large piece of data and be able to overwrite EIP, thus having control over the next instruction that the application should run!

The Proof-of-Concept Process

I created the test application that included the vulnerability, and the code is shown here:

#include "stdafx.h"

void readFunc(const char *, FILE*);

int main(int argc, const char * argv[]) { FILE * pFile = NULL;

readFunc(argv[1], pFile);

printf("The end.");

return 0; }

void readFunc(const char* arg, FILE * pFile) { char strInput[32] = {0}; const char * strMode = "r"; pFile = fopen(arg, strMode);

fscanf(pFile,"%s", strInput);

fclose(pFile); }

I used VS 2012 to build the above test application. In order to simplify the attack scenario, I did the following:

In Project Properties->C/C++->Code Generation->Security Check, I turned off this option (set to /GS-). This is so that in the stack, security cookies aren't embedded around the buffer, and so when I overwrite the buffer and then write past the stack frame, the test application will still keep running.

In Project Properties->Linker->Advanced->Data Execution Prevention (DEP), I turned off this option (set to /NXCOMPAT:NO). This is so that when I fill the stack with shellcode, it will be able to run.

In order to be able to run the application on WinXP, I used the "Visual Studio 2012 - Windows XP (v110_xp)" platform toolset.

I chose Windows XP SP2 as the environment that I used to run the vulnerable application.

First, I needed to try to control the test application's EIP register. I did this by trying to feed the application a long list of autogenerated characters. I used Metasploit's  pattern_create ruby script. Given a string length, this script will generate a string with a known pattern. I then ran Immunity Debugger on my vulnerable application. I noted the contents of the EIP register. I then used Metasploit's pattern_offset ruby script to determine the offset location of the hexadecimal pattern that I saw in the EIP register.

Next, I created a python script to generate the input string for the test application (shown below). It also includes something called a "NOP slide". A NOP is an instruction that does nothing. This NOP slide is used to (eventually) place some shellcode in a correct offset such that it can be recognized as valid instructions. 

Again, I ran Immunity Debugger on my vulnerable application and fed the application the string that was generated by my python script (shown above). Below is the resulting output from Immunity:

As we can see from above, the EIP is filled with 42424242, which is BBBB in hex.

Next, I needed a way to redirect the application to run instructions written found in the stack. The instruction is "jmp esp" (FF E4) and I found this instruction by the following method:

In Immunity Debugger, went to "View"->"Executable Modules". There is a list of libraries that have been linked with the vulnerable application. I started with ntdll.dll.

I opened ntdll.dll and did a Ctrl-F in order to find the expression "jmp esp". I then took note of the memory location of this instruction. This is shown below (top left window pane, first line - for me, this instruction lives in the address 7C941EED):

Once I found the memory location of the "jmp esp" instruction, I substituted that for the contents of the EIP in my python script, as shown below:

(Note: the EIP string in the above python script is written backwards, in order to match the little-endian format of memory address)

Finally, I was able to redirect the application to run the next instruction from the stack (note that I took out the shellcode and only placed the nop slide into the stack - I still need to work out why the shellcode isn't working and this will be the topic of the next post). I also found out about the INT 3 opcode (0xCC). Debuggers use this opcode to put in debug breakpoints in code. This opcode comes in handy when you put it int he middle of the NOP slide in order to just do a check on your proof-of-concept.

"Weee! I'm riding a nop slide!" - Sandra 

Next post will be all about shellcoding! Stay tuned!

Source:

Today, I wanted to experiment with metasploit and to take my first dive at penetration testing. This was my lab setup:

VMWare workstation 10.0.1

KaliLinux on VM1 (set up this VM to have "host-only" networking)

Metasploitable2-Linux on VM2 (set up this VM to have "host-only" networking)

I wanted to have an isolated network with just the target machine and the attacker, so that's why I used "host-only" networking. KaliLinux is a decked out image of Debian that has all of the tools for penetration testing already set up. Metasploitable is basically a linux image having many exploitable services set up on it, in order for us to learn how to hack.

Then I followed along with the tutorial video (source link below). Here is the summary:

We start metasploit console using the command:

msfconsole

Using the command:

use auxiliary/scanner/smb/smb_version

we can scan hosts on the network to see if they are running Samba and if they are running Samba, we can determine the version that they are using. We can then look up this version and see if there are any vulnerabilities associated with it.

Using the command:

use exploit/multi/samba/usermap_script

we can exploit the target machine running the vulnerable Samba server and give it a payload (automatically generated with the exploit script) which will then force the target machine to call back to our attacker machine, which will eventually lead us to obtain a command shell on the target machine, with the attacking machine having root user rights.

Source: 

I've been looking at different ways of understanding how DLLs work and why they're set up the way they are. I wanted to understand more about DLLs because they've caused some confusion for me during my development process. Also, I wanted to understand the Windows system on a much deeper level. So, without further ado: let's talk about the Portable Executable File Format!

The portable executable (PE) file format is the way an executable's data is arranged, such that the system's memory loader can load the executable and the system can run it.

Every time an executable is loaded, it can live in a different virtual address compared with the previous time that executable was loaded. This is called "virtual address relocation" and this happens because other executables are loaded in the same address space. The OS loads various DLLs into each program's address space, and of course depending on the features needed by the program, different DLLs will be loaded. The PE's headers point to different sections of the executable and uses relative virtual addresses (RVAs) to point to these sections. RVAs are an offset into the file and uses a base address (the virtual address at load time of the executable) as the place from which to offset.

 Executable images are divided into sections. These sections are treated differently by the memory manager when the module is loaded, depending on whether that section is readable, writable, or executable. There are two general sections: Code section and Data section. The code section (a.k.a text section) is where the code for the executable lives. The data section is where the initialized variable data live.

Dynamically linked libraries (or DLLs) are the mechanism by which a program can be broken up into more than one executable file. Each of the smaller executable files are responsible for a different feature of the program's functionality. One reason why DLLs are beneficial is that the program only loads the modules right when a certain feature is needed, and this helps to save memory space. Another reason why DLLs are helpful is that individual features of the program can be upgraded without having to upgrade the entire program.

Remember: DLLs are different from static libraries! Static libraries are linked into the executable at compile time, whereas DLLs are linked at load/runtime.

There are two methods for loading DLLs:

Static linking (not the same as compile-time static linking, as mentioned previously).

Run-time linking. 

DLL static linking  is when an executable has a reference to another executable in its "import table". The import table is a list of all modules used and each function used in those modules. The Windows loader loads all modules mentioned in the import table and resolves the references to each function that the executable plans on using.

On the other hand, run-time linking is when the executable has the ability to decide to load another executable during run-time and call a function in that called executable. Run-time linking is more difficult to implement as a programmer compared with DLL static linking. However, DLL static linking is easier for a reverse engineer because he/she can easily use the import table to determine which modules are used and which functions are called by those modules. 

Summary of Important concepts:

An executable can live in a different virtual address in memory from the last time that you loaded it

Executables have different "headers" that give information about different parts of the executable.

Relative Virtual Addresses (RVA) are used to point to different locations  in an executable

DLL static linking is different from compile-time static linking

Recently, I've had the privilege of learning COM at work. I dealt a tiny bit with it during my internship, but not at the same depth as now. I had never really took the time to take in the concepts of COM, but now, I am more aware that when learning about something, it's best to dig into the concepts before the details. So, without further ado, I introduce my (most current) summary of COM concepts (which will be built upon as I keep learning):

* coclasses

* interfaces

* class factories

* component housing

About coclasses:

The term "coclass" is just a fancy term for a COM object. A COM object inherits from at least one abstract class - IUnknown. The inherited abstract class is a way to define an "interface". An interface can be thought of as a contract signed by the COM object which says that the COM object MUST implement the methods declared in the abstract class. The code that wants to call the COM object's methods will actually call them indirectly through those interfaces...

About interfaces:

The IUnknown abstract class declares three methods, which the COM object must implement, or else, it too will become abstract. Remembering the concept of C++ pure virtual methods is really helpful here. Whatever code (we'll call it the client) is interested in using the COM object will actually be given the pointer to the COM object's virtual function table pointer! In this way, the client will be able to call the methods that the COM object was forced to implement because of what it inherited from its abstract classes. Remember that the virtual function table is just an array of function pointers. These function pointers point to the actual implementation of the methods mentioned in the abstract class inherited by the COM object. When it comes to creating the COM object, the client code doesn't do this directly. The client code actually goes through class factories...

About class factories:

A class factory is just a COM object whose soul purpose is to create one kind of COM object when it is requested to do so by the client code. When a class factory processes a client's request for a COM object, it creates an instance of the COM object and then returns a pointer to its interface to the client that made the request. The client, having the pointer to the COM object's interface, can now make calls to run the COM object's methods. The COM object and class factory live together in what's called the component housing...

About component housing:

We need to remember that COM objects don't just sit in memory as .obj files. They must live in some kind of binary packaging. We call this the component housing. These component housings can either be DLLs or executables. In general, the COM runtime is responsible for loading the COM objects from these binary packages, so that the client code can use them.

I was given the task to find out what was going on with an ftp server running on a Debian Linux machine. The symptom of the issue was that the ftp server, during an FXP transfer, would run fairly well for a few hours. There would be a lot of files transfered over that time period, but then over time, the ftp server would stop servicing connections. When the ftp server was in this bad state, it was possible to connect to the server, but the server would not ask for a username or password, and would just not let the user sign in and do anything.

My initial instinct was to create a script to mimic the fxp file transfers that were going on. Because of the nature of the FXP protocol (which is just using FTP protocols, but in a special way), I decided to create an expect script for this task. This would be a bit of work, since I had never in my life created an expect script.

The reason why I chose to create an expect script, instead of a bash script was because of the following: In an FXP transaction, there are three actors - a source server, a destination server, and a client that orchestrates the file transfer. The client would first issue a "quote PASV" command to the destination server. This would cause the destination server to go into a passive listening state, and would reply back to the client with a port on which the server is listening to connections for the file transfer transaction. Next, the client issues a "quote PORT a1,a2,a3,a4,a5,a6" command to the source server, where a1-a6 are the given port values from the destination server. This tells the source server that it should use the given port in order to do the subsequent file transfer. Next, the client issues a "retr filename.ext" to the source server, and very quickly after that has occured, will issue a "stor filename.ext" to the destionation server. Because I required the port value reply from the destination server, creating an expect script was ideal. With an expect script, I could just use regular expressions to look at the replies from sending out the various FTP commands to each server, and when I required the port value from the destination server, I could just obtain that from a special expect variable after the regular expression was done and send it as part of the next command to the source server.

So, with my expect script in hand, I eventually got the ftp server into the bad state. I knew that I could look at the state of all of the sockets that the machine was using, with the netstat --tcp command line call, as well as lsof -nl | egrep "TCP|UDP". This was a good first start to get insight into the problem: There were multiple processes of the same name with sockets in the CLOSE_WAIT state.

What the who now? 

I expected one process with multiple sockets - but what was this CLOSE_WAIT socket state business? After much researching about TCP/IP socket concepts, I found out that CLOSE_WAIT is a socket state that an application's socket would go into, if the remote side disconnected first - it is then the responsibility of the application that owns the CLOSE_WAIT state socket to explicitly close it, otherwise this socket will never be reusable. So, it's ok for the socket to be in a CLOSE_WAIT state, just as long as it doesn't take a lifetime (or never) to close. This information told me two things: 

The ftp server was not closing its sockets properly after it was finished with them, but this only occurred during a special situation and that's why it takes a long time to rear its ugly head.

There were sleeping processes that owned these CLOSE_WAIT state sockets, but who were they? They didn't belong there for sure.

After much discussion with a work colleague, I found out that there was a process that was being forked in the thread that did the actual data servicing of the connections (not the connection accept thread, which hands off the connections to the data servicing thread). Very interesting! I did more testing using my script, and I noticed as I watched the output of lsof -nl | egrep "TCP|UDP" that there was a specific process name (which my colleague had mentioned) that shows up (but very briefly) - this was during a normal run of the ftp server.

I did some background reading on process forking and found out that open file handles are copied from the parent process to the child process during a fork. I theorized that if the socket (which is really just a file handle) were being copied to the new child process, and then the child process for whatever reason couldn't continue (so would then be a sleeping process), it would still have the socket. Then, if the parent process would issue a close on this socket, it would only affect the socket within the parent process - not the child's, so it wouldn't really be closed in the operating system's eyes.

When I got the ftp server (in debug mode) into the bad state again, I was able to look at the program's call stack, and noticed that the forked process was trying to clean up a resource that was locked by a mutex. Since mutex state are also copied between parent and child, the child was waiting for the mutex to unlock, but there was no one to unlock it, so the child would just sit there. This was the cause of the multiple processes being in this sleep state. After much researching and discussing with colleagues, I found out that a child process should use the _exit() instead of the exit() in order to finish, and the ftp server child process was using exit()! Yes, there is a huge difference!

exit() will call the destructors of objects to clean up, BUT _exit() will not! Using _exit() would cause the child to immediately terminate, and not have to deal with the locked resource that I had mentioned earlier.

Once the change was implemented, there were NO more extra sleeping processes, just the one for the ftp server that I had expected! :) 

That was a fun ride - I got to learn a lot from colleagues on how to do more advanced debugging techniques in Eclipse, and I also got to teach myself how to write expect scripts. On top of that I got to delve deeper into the concepts of TCP/IP. 

I've recently been tinkering with C#, and I started thinking about how unmanaged resources should get cleaned up in C# code. I figured this would be a good way to learn the little details of C# if I kept some C++ concepts in my mind.

So one big difference between C# and C++ is that C# has a garbage collector that cleans up resources (i.e memory) when they are no longer in use by the application. In contrast, C++ has no such mechanism, and it is up to the application developer to clean up resources when they are no longer in use. With C#, the application developer shouldn't really force a garbage collection whenever he or she feels like it (or thinks that time would be a good idea) because doing so would incur a performance hit. What about if the application that I was developing in C# called (i.e via SWIG) a C++ library that allocated some memory - how would the garbage collector handle this? I'm pretty sure that the garbage collector wouldn't really know anything about this allocated memory, and if my C# application doesn't take the explicit steps to clean it up, I'd have a memory leak. So, how would we take care of so called "unmanaged" resources in C#? 

The answer to this is to use a design pattern (recommended by our trusty MSDN documentation) that calls for the following:

1. Encapsulating the unmanaged resource in a base class that implements the IDisposable interface.

2. Derive the base class - add some additional functionality around the unmanaged resource.

The code example from the document "Implementing a Dispose Method" (see sources list below) was a good place for me to start looking at how this was done. The following are just some summary notes that I compiled about the code example in order to help me distill the basic ideas:

Have a base class that implements IDisposable

In base class: When implementing Dispose(), we want it to not be virtual so that the derived class can’t override it.

In base class: When implementing Dispose(), we want it to first call the base class’ Dispose(bool disposing) method, with disposing = true (disposing = true means that our code called Dispose(bool disposing) and in this scenario, we want to release both managed and unmanaged resources).

In base class: When implementing Dispose(), after we call the base class’ Dispose(bool disposing) method, we want to take this class off of the Finalization Queue in order to improve performance (since it would be a performance penalty otherwise). This is done by calling GC.SuppressFinalize(this) in Dispose().

In order to keep track of whether the base class has been disposed (in order to avoid a “double free” – borrowed from C++), the base class needs to have a private bool attribute that keeps track of this (this is called “disposed” in the example).

In base class: there is a virtual Dispose(bool disposing) method that needs to do specific things in two scenarios:

1. When disposing is true, this means that our own code has been able to call Dispose(bool disposing), and this means that we can release both managed and unmanaged resources.

2. When disposing is false, this means that the Finalize() method needed to be called and our code was not able to call Dispose(bool disposing) – rememeber that Finalize() is there as a safeguard just in case Dispose(bool disposing) was not called. In this scenario, we only want to release unmanaged resources, since the managed resources will be taken care of by the garbage collector.

In base class: Its destructor (which is what Finalize() uses) is where we would call Dispose(bool disposing) with disposing set to false.

In base class: Whenever we do something to the base class, we need to check if its private attribute to keep track of whether the base class has been disposed has been set to true. If it has, the base class must through an ObjectDisposedException().

In derived class: It will need to override Dispose(bool disposing), and here it will do its own dispose (much like the base class’ Dispose(bool disposing)) and it will be done in a try block. It will then call the base class’ Dispose(bool disposing) in the finally block.

Sources:

1. "Destructors (C# Programming Guide)" http://msdn.microsoft.com/en-us/library/66x5fx1b(v=vs.80).aspx

2. "Cleaning Up Unmanaged Resources" http://msdn.microsoft.com/en-us/library/498928w2(v=vs.80).aspx

I've been reviewing some of my C++ concepts, and I came across my good ol' friend, polymorphism. Polymorphism is what allows C++ programs to decide the kind of functionality to use during runtime.

Say we have an abstract base class, called "vehicle". We call vehicle abstract because we would never instantiate any objects of type "vehicle". We tell the compiler that we can never instantiate an object of type "vehicle" by making at least one of its methods a pure virtual function - say, a method called "goFast". Imagine also that we have two classes that inherited from "vehicle", called "AcuraIntegra", and "SuperFastSpeedboat". In each of the two child classes, we define the "goFast" method - one method of "goFast" will apply to how an "AcuraIntegra" would go fast, and the same for "SuperFastSpeedboat".

Next, somewhere in our code (where we're allowed), we would create a pointer of type "vehicle", and also we would instantiate an AcuraIntegra and a SuperFastSpeedboat. It would be nice to be able to point to a type of vehicle, and just do the "goFast" method, without knowing before compiling the kind of vehicle to which we were pointing - allowing for flexibility. Polymorphism is that thing that allows us to do this magic.

But there's always a price to pay. Polymorphism is implemented using an array of function pointers (the vtable), and there's one for each class that has at least one virtual function. Note that there only needs to be one vtable for the many instantiated objects of the type of class that has at least one virtual function, but this vtable does take up space in process memory. Polymorphism also requires that each object have a pointer to the vtable, which also takes up space in process memory. Another price to pay is that polymorphism needs to use extra instructions in order to be able to look up the proper function to use for the type of data that is being pointed to by the pointer of the abstract type - this takes time.

So, it is always good to think about a functionality's advantages and the price we pay for having those advantages. Depending on what we'll be using it for, perhaps we'll feel alright paying for it. I suppose polymorphism's overhead is one reason why C is usually preferred over C++ when it comes to embedded applications.

Sources:

1. "Dynamic Binding C++". Schmidt, D.

http://www1.cse.wustl.edu/~schmidt/PDF/C++-dynamic-binding4.pdf

2. "C++ Performance Tips". Gilpin, A. 

I was trying to figure out how a specific Linux utility worked. I will call this utility "util". I don't want to name anything specific, just in case ;). The company that made this utility have an SDK, but for the life of me, I couldn't see a mention in their documentation about how I could use an API to mimic util's functionality. 

I didn't have access to the source or project files, so I couldn't look using the regular means. After much thinking, I thought that maybe I could work through it backwards using objdump and strace.

As I had mentioned in one of my earlier posts, strace is a Linux tool that can show us the system calls that are used by an executable. I wondered if the functionality of util that I was interested in was being implemented using ioctl calls. It turns out that it wasn't, but the beginning of the strace output gave me an interesting lead. I noticed that util was using a specific shared object that was not part of gcc or anything usually Linux related. I decided to look at the symbols listing of this mysterious shared object, using objdump: "objdump -dR libmystery.so". I noticed that there were symbols being exported with very interesting names - names that related to the functionality that I was trying to figure out.

I googled the name of the shared object library that util was using. I saw a changelog talking about my mystery library, and the changelog description mentioned the functionality that I was interested in. I was getting closer! Now I needed to figure out how I could use this shared object library. It turns out, after much (like a lot of) digging in google, I found out that the company who made the utility had just recently released the API for what I needed. I was in luck! I found the header file for the API, and tested it out using a small program that linked with the mystery library (the same one that util was using) - it worked.  

I've been thinking about process memory layout and how linked lists live in memory. I think its a good idea to think about this, since it's another way of thinking about how linked lists work, and it helps me remember how to make them.

When memory is allocated to a process, the process sees it as one solid chunk, and it sees it as starting at zero. In reality, the physical memory that is allocated to the process is chunked and each chunk can be in different physical locations. Within the memory allocated to a process, we can see two major sections: data and text. Within the data section, we can see two subsections: the heap, and the stack. 

Whenever a function in a process gets called, that function gets a section of memory on the stack. This is where local variables, the function's input parameters, and the return address for the next instruction to execute (the function caller's next instruction) get placed. Speaking in terms of C, whenever a function calls malloc, a section of memory in the heap is set aside for that call. The stack grows from high memory address to low, and the heap grows from low memory address to high, so they grow towards each other. How does the concept of process memory layout that we've been talking about relate to linked lists? We can remember how to make a linked list by thinking about those concepts.

Before a linked list is born, a pointer to a node data type first points to NULL. This pointer exists on the stack, and there is nothing related to linked list data on the heap. When we create the first node, that's when we request that memory be allocated for it on the heap, and the pointer can now have the address of the section of memory in the heap when this request for memory is granted.

Of course, the node that was just created will need to have a data member that points to NULL, which represents the next node on the list. We can keep requesting for memory on the heap for our linked list, but this can't go on forever.

When we no longer need the linked list, we remember the process memory layout concept, and we can then remember that we need to clean up the heap and free the memory that we've just allocated for the nodes of the linked list. So, our cleanup routine would just require the pointer to the start of the linked list, and it would just traverse the list and free each node. Remember that if a node consists of pointers to memory locations on the heap, we need to free those individually and then free the node.

Viewing linked lists in terms of how the data structure uses the process stack and heap really helps me remember how it should be coded.  

Sources:

1."Using GNU's GDB Debugger: Memory Layout and The Stack". Salzman, P. 

Much like The Transformers, static libraries are indeed more than meets the eye. In Linux, static libraries can be examined by using the nm command, and issued like this in the command line:

nm libThatYouWant.a

The function of nm is to list the symbols that exist within the static library. When you link in a static library to a project, the references that have been used in the project are resolved by looking at the symbols that have come from the static library. 

The idea above sounds simple enough, though things can get a bit confusing when you are dealing with multiple static libraries that depend on other static libraries that get linked into a project. 

The main concept to remember when linking in static libraries is this: the linking order (as listed in your project's linker settings) of the static libraries that the project uses is important. References first, definitions later. I remember it in this manner: list those that refer first, before those that define - when it comes to static libraries.

To give an example, say we have a project that builds an object file (executable), called test. This project links in both libsomething.a and libawesome.a. In libawesome.a, there are references to symbols in libsomething.a. Within libsomething.a are the actual definitions to those symbols.

If you've set the linker order to be:

1. libawesome.a

2. libsomething.a

Then the linker is happy, and all is well. Had you linked in the opposite order, I can guarantee that there would have been much wailing and gnashing of teeth. 

I read a very interesting (and very excellent) article the other day called "A Collection of Examples of 64-bit Errors in Real Programs" by Andrey Karpov. It got me to thinking about differences in 32-bit and 64-bit programming in C, and some fundamental concepts that involve this issue.

This may seem very basic, but easy to overlook: pointers are the same size, no matter the type to which they point. Pointer size is also dependent on the data-type model. The data-type model can be ILP32 or LP64, as mentioned in the Sun article mentioned in the sources list below. The name ILP32 tells us that integers, longs, and pointers are 32-bit in size in 32-bit applications. Similarly, the name LP64 tells us that longs and pointers are 64-bit in size in 64-bit applications, but that integers and floats are the same size as before (32-bits).

With a 32-bit pointer, we can address up to 4GB (since 2^30 is 1GB, which is 30 bits, then we multiply by 2^2 (an additional 2 bits), in order to get the 32-bit representation). But what happens if we're used to storing an address that's 32-bits in length in an integer, and never thought about the case where addressing could be more than 32-bits? We could have an application that we wrote in a 32-bit mindset, which has instructions for using pointers and casting them as integers, or storing them as integers. This would be ok if we only had addresses that were less than 4GB, but anything above that would cause problems because of the truncation of the bits after bit 31.

Andrey Karpov (mentioned in the source list below) talks about this case in his article, specifically in the section "Example 2 - Unnecessary Type Conversions". Another variant of this truncation issue is in the section "Example 4 - Confusion of int and int*". 

I must say that before reading Andrey Karpov's article, I was not aware of this issue, but now it's something that I definitely think about.

Sources:

"A Collection of Examples of 64-bit Errors in Real Programs".                    Karpov,  A. http://www.codeproject.com/KB/cpp/Examples-64-bit-Errors.aspx

What is const? How do we use const? Who am I? Why are we here? These are just some of the things that go through my mind on a regular basis. In this post, I'll show how I think about the first two questions, and I'll leave the last two for some other day.

So, when we use const in C, we apply it to an identifier - some variable that we will use. What we are saying to the compiler when we tell it that an identifier is const is this: "I don't want to change what's in the variable through assignment operations on that variable". (If we wanted to be sneaky, we could get the address of that variable that we've just declared const and place the address in some other variable, and then modify the contents through the second variable.) But, for regular usage we would say, for example: 

const int * p

int * const j

Those two examples above look kind of strange. Sometimes, it's easy to forget on which part of the declartion the const applies. There is a nifty tool that I use called the "Magic Decoder Ring for C Declarations" from the book called "Expert C Programming: Deep C Secrets". There's a general form of the magic decoder ring in that book, but I'll talk about the shortened version for the specific case above. The condensed decoder steps are:

1. Find the left-most identifier.

2. From the left-most identifier, look to the left of it and determine if you need one of the three in the list below. The first one you encounter in this list from top to bottom will apply first:

const - say "read-only"

volatile - say "volatile"

* - say "pointer to a(n)"

3. The tokens that you have not touched form the basic type of the declaration. i.e "extern int"

Now to try the decoder on the first declaration in the above example. It is read as: "p is a pointer to a read-only integer" - we can change the address to that which p points, but we can't change the value inside that address.  

The second declaration would be read as: "j is a read-only pointer to an integer" - we can't change the address to that which j points, but we can change the value to which it is pointing. 

Wasn't that easy? I really enjoy the magic decoder ring. It's so useful for removing confusion when it comes to C declarations.

Sources:

I know that when I've read over the minute details of concepts of C, I have come across the idea that there is really a difference between a declaration and a definition, and that I should be aware of this. At the time that I read this, I kind of skimmed it and it didn't really dawn on me until the time came when I needed to build an open project from source, which happened to depend on openssl. 

This project had a piece of source code that used a function that was defined in an openssl .lib file (but at the time, I didn't know it). I was building the project from the command line (using the Visual Studio compiler), so I could see all of the .obj files being created and which parts of the project depended on which .obj files and .lib files. When it came time for the .mak file to build the project's executable, it was telling me that an error occurred because a .obj file was referencing a "symbol" for which it couldn't find the definition, otherwise known as the LNK2019 "Unresolved external symbol". After some experimenting, I found out that the version of openssl that I was using was slightly newer, and the function being used by this open project was not defined in this newer version, so I used the previous version of the openssl library, and it worked out well.

So, where does the idea of declaration vs. definition come in? The answer to this is that the compiler took the piece of source code that used that specific openssl function and tried to find its definition, which is basically the blueprint of the function. The definition is the thing that has memory set aside for it. The declaration, on the other hand, is just the thing that mentions that definition. I like to think of it, much like the author of "Expert C Programming: Deep C Secrets", in the sense that a person can declare luggage at the airport, and can declare it as many times as they want (basically referring to this piece of luggage), but there is only one definition of the luggage (its essence). So, back to the compiler and the linker: the compiler was telling us that "yes, I see that you are referring to a function, but I don't see its blueprint. I will assume that its blueprint is defined elsewhere". When it came time for the linker to do its job, it told us "hey, the compiler has flagged me to look for the blueprint of this function, but I can't find it, so I can't use this function!".

The lesson: Whenever you are referring to a function (or an extern variable) in your code, you need to make sure that its definition exists, either in some other .obj file, or some .lib file.

Sources:

1. van der Linden, P. "Expert C Programming: Deep C Secrets". 1994.

I've been thinking about how nice it would be to have a library that people could use in order to deal with a lot of hardware interactions in Linux, without having to use direct ioctl calls in their own code. It is quite possible that there is one already in existence, and I've just not managed to find it yet, but if it doesn't exist yet, the following ideas come into mind when I think about implementing something like this. Mind you, this is an initial thought, and some more digging around needs to be done to see if this would be a good idea to implement.

Say, I wanted to be able to mimic the functionality of a certain program and how it interacts with the specific device driver in order to interact with hardware X. Let's say, further, that this program is run through the command-line.  I could run strace on it to see exactly the kinds of ioctl calls it makes in order to talk to the device driver - these exact ioctl calls can be used in the hardware library that we want to make - you'd include <sys/ioctl.h>, of course. The ioctl calls can then be wrapped in functions that other applications can use in order to get at the hardware in user-space. 

Sources:

"Controlling Hardware with ioctls". Corsetti, L. 

        http://www.linuxjournal.com/article/6908

    2. "Fun with strace and the GDB debugger". Zimmerly, W. B.