#cffi

pypy와 cpython 사이의 호환성 이슈는 C라이브러리 연동과 연관된 것이 많다. 그래서 pypy를 보다보니 C연동에 대해서는 각 방법들을 한 번 봐야겠다는 생각이 들었다. 얼마 전에 ctypes를 보고 오늘 cffi를 마저 보았다. 아직 머리에서 깨끗하게 정리는 안 되도 쪼금 감은 오는 것 같다.

pypy에서는 새롭게 cffi라는 모듈을 지원하는데, 이 모듈은 사용법이 재미있는 것이 C코드를 문자열 그대로 파라메터로 넘겨주면 이를 파싱해서 인터페이스를 구성해주어서 사용법이 많이 편하다. (luajit ffi라이브러리를 따왔음.)그리고 ctype스타일로 외부 라이브러리를 호출하는 API방식 외에 외부 라이브러리를 호출하는 C모듈을 자동으로 만들어 연동해주는 ABI방식도 지원한다. ABI방식이 API방식보다 성능이 더 좋다고 한다. 다만, 처음 실행시 컴파일 시간이 들어가서 시동시간은 더 느릴 수도 있다.

Every decent implementation of Common Lisp has a foreign function interface. When we need to call functions defined in external libraries and following a calling convention of C, we tell our Lisp system (1) where to find the library, (2) how to find the function, (3) what types of arguments it expects. Then we call it.

There is a common portability layer for many FFI implementations: CFFI. I'll use it for my example here, but the problem I'm thinking of doesn't depend on it. It does depend, however, on OS dlopen facility that we find in the POSIX world.

What we see in many Lisp libraries that use CFFI:

(define-foreign-library frobnicator (cffi-features:unix "/usr/lib/libfrobnicate.so")) (load-foreign-library 'frobnicator)

How many problems do you see in this code? Zero? Then delay publishing your libraries which use FFI, until you finish reading this entry.

Two things are wrong with "/usr/lib/libfrobnicator.so":

Hardcoded library path. When a developer had to specify a foreign library to load, he probably looked up its location on his own system and added it to define-foreign-library, without thinking that it could have another location on the user's machine.

Misguided attempts to get it right are likely to add something like "/usr/lib/x86_64-linux-gnu/libfrobnicator.so" to the picture, possibly conditionalized on cffi-features:x86-64. It remains wrong. The only right thing to do is to omit the directory and rely on dlopen() to figure it out.

Underlying native FFI may be incapable of doing it this way, doing a user-level library search before dlopen. Then native FFI is wrong: complain to your vendor about your Common Lisp implementation.

Developer's library used at runtime. When a shared library is installed on a typical unix-like system, we see something like libfrobnicate.so.1.2.3, with libfrobnicate.so.1.2 and libfrobnicate.so.1 as symlinks, and maybe libfrobnicate.so symlink as well. How should we decide which name to use in define-foreign-library?

If it's a Linux distribution with a package manager, an important clue can be acquired by examining packages where this files and symlinks belong. A version-less libfrobnicate.so is usually in a separate package, with its name ending in -dev or -devel, together with include files which make it possible to compile programs depending on this library. It's not really different on most Unix-like systems other than Linux, even if it's not that obvious from package layouts.

For ELF-based systems, each shared library has a soname that serves as a key when a dynamic loader searches for program or library dependencies; there is an entry in the library header where the soname is stored. The soname of our libfrobnicate.so is probably "libfrobnicate.so.1" (no "/usr/lib/" or other paths; a major version follows the suffix, subversions omitted).

When we build a C program, the linker's command line includes something like -lfrobnicate (no suffix, no directory). The linker searchs some directories (depending on its configuration and command line arguments) for libfrobnicate.so, then it reads a soname from the file (libfrobnicate.so) itself, and this soname ends up in a generated executable. Thus "libfrobnicate.so" is never used for loading the program after it was compiled; only its soname ("libfrobnicate.so.1") is seen and used by the dynamic loader.

If we use a bare soname with no paths in our foreign library reference, we maximize the ability for OS to do the right thing on a user's machine. Unless we know for sure that OS is incapable of doing the right thing anyway, our code should be like this:

(define-foreign-library frobnicator (cffi-features:unix "libfrobnicate.so.1")) (load-foreign-library 'frobnicator)

These is a common objection to this practice: "I don't care on the exact major version". Well, the reason to have major versions in sonames is allowing library vendors to make any incompatible change to the library interface. Are you sure that your code will survive any incompatible change? Then I would like to see your code and learn to provide such robustness.

Several major versions can have enough in common to allow Lisp code to remain the same. Fortunately, CFFI allows a list of alternatives instead of a single name:

(define-foreign-library frobnicator  (cffi-features:unix (:or "libfrobnicate.so.2" "libfrobnicate.so.1")))

More alternatives can be added easily when the old code is verified to work with newer and newer major versions of the library. That's the only way to do it reliably. Remember that for an incompatible major upgrade, getting an instant failure is our best hope: sometimes, the interface change will cause our code to do the wrong thing silently, or worse, to work most of the time and fail spontaneously.

There is a class of situations where it can be right to use something other than soname, and even to hardcode library path. Some shared libraries are not supposed to be integrated with the rest of the system: linker won't normally see it when looking for -lname, and dynamic loader won't find it where expected. It's often the case for modules loaded into scripting languages, such as Perl, Python or TCL. If you call such a library from Common Lisp (and it may be actually useful), feel free to hardcode something like "/usr/lib/perl/5.14/auto/Encode/Encode.so" into your FFI code.

Most of the time, linking against unversioned name of a shared library seems to work, as long as you have it installed (that's why this practice has become so widespread). There are some exceptions, however. It may be interesting for Linux users to take a look at libncurses.so or libc.so under their /usr hierarchy (in /usr/lib, /usr/lib32 or /usr/lib/x86_64-linux-gnu): these files are not shared libraries at all. They contain linker directives, so the linker has a way to know where to find real libraries.