The Code Dungeon

Revelations and frustrations of a Fortran/C programmer

Conda-forge: creating Windows recipes for Autotools based Python projects

It is probably fair to say that Anaconda has become the most popular Python distribution in scientific circles. This should not come as a surprise as it comes with many features that make it stand out from its competitors:

  • a convenient installer
  • allows for installing several versions of the Python interpreters side-by-side using environments
  • has access to the most recent versions of many important (scientific) packages
  • does not require admin privileges!

Though Linux and macOS ship with their own Python interpreters, these are in many cases insufficient for development as access to the latest releases of Python packages is usually not possible with the default package managers (outdated packages only) or using pip, especially when binary extensions are required that depend on non-Python libraries.

With Anaconda, it is quite easy to write and build new recipes for Python and non-Python software packages and upload them to a personal distribution channel. Conda-forge makes this even easier: just write the recipe, open a pull request on GitHub, and wait for the continuous-integration buildbots to generate binary packages for Linux (Circle-CI), macOS (Travis-CI) and Windows (AppVeyor). After merging the pull request on GitHub, the binary packages will be automatically uploaded to the conda-forge channel, at which point they become available to all Anaconda users by executing the command:

1
conda install --channel conda-forge my-package

This procedure works fine as long as the installation script for the Python package uses distutils or setuptools, as it ensures that all files are installed in the appropriate location and that any binary extensions are compiled with the same compiler that was used to compile the Python interpreter. But what happens when the Python package uses a different buildsystem?

Enters my personal flagship project xraylib, a library providing convenient access to physical databases relevant in the field of X-ray physics. This library consists of a core shared library, written in ANSI-C, and comes with bindings for about a dozen of other languages such as Perl, IDL, Ruby, Lua, Fortran, and of course Python. In fact there are two Python bindings: the first extension module is generated with SWIG (deals with scalar arguments) and the second one with Cython (deals with NumPy arrays as arguments). In order to support building the core library as well as the many bindings using a single build and installation system (and thereby ignoring the recommended build systems each of these languages bindings has for binary extensions), I use Autotools, which consists of Autoconf, Automake and Libtool. Libtool is key in this setup, as it enables building both shared libraries (the core C library as well as the Fortran bindings) as well as dynamically loadable plugins (Python, Perl, Lua, Ruby and IDL bindings), in a platform-independent manner! See also this old blogpost if you are interested in reading how I accomplished this for the Python bindings.

A typical autotools project may be built and installed through the following three commands (after unpacking the source tarball):

1
2
3
./configure
make
make install

The xraylib configure script figures out where the Python extensions need to be installed in the last command, thereby ensuring they will get picked up by the interpreter without fooling around with the PYTHONPATH environment variable. This works well in a conda recipe on both macOS and Linux, and has led to several people uploading xraylib packages to their personal conda channels for these two platforms.

Windows however is an entirely different beast.

Over the years, many have asked me to come up with providing better Python support for xraylib on Windows, in particular via pip and conda, as the Python binary extensions I provide in my xraylib Windows SDKs do not integrate easily with Python interpreters, in particular due to their dependency on specific NumPy versions.

Read on →

easyRNG: random number generators and distributions for C and Fortran

The C++ 2011 standard introduced a collection of templates, grouped together in the <random> header of the C++ standard library, with the goal of providing a number of random number generators (RNGs) as well as several commonly used random number distributions (RNDs) to sample from. Though reception appears mixed, at least they provided a welcome alternative to the very unreliable rand that ships with the standard C library.

However, this does not mean anything to those who are writing code in C and Fortran, and have no plans to switch to C++ in order to obtain RNG and RND functionality. Personally, I have been using for many years the RNGs and RNDs offered by the GNU Scientific Library (GSL) and its Fortran bindings FGSL (of which I am a major contributor). I have found GSL and FGSL to offer both an elegant interface as well as excellent performance on all major platforms and architectures. A potential drawback for their adoption however, are their license: the GNU General Public License, which makes it impossible to include into proprietary, closed source software, though this is no concern for me personally as all my projects are open source.

This ended up giving me the idea for a new open source project easyRNG, a thin wrapper around C++11’s <random> templates, with an API inspired by GSL and FGSL. The name for the project was obvious actually:

  • Easy to implement and maintain: I didn’t have to write the random number generations and distributions myself (which is really, really hard)
  • Easy to use: based on the established and popular APIs offered by GSL and FGSL, which should feel familiar to everybody (see examples)
  • Easy to build and install: no dependencies required apart from a C++11 compliant compiler, and the C++ standard library to link against. Those in need of the Fortran bindings will have to install a Fortran compiler with full 2003 and partial 2008 support. GNU autotools was used for the installation script so Windows users will have to install a suitable shell (e.g. msys2) to build easyRNG. It shouldn’t be hard to get it working in Visual Studio though.
  • Easy to redistribute: I picked the 3-clause BSD license, which means it can be included (even with modifications) into proprietary software, provided the original copyright notices and disclaimer

After a couple of weeks of work I launched easyRNG on Github, where one can download the code, browse the docs and of course fork the code!

Read on →

Embedding PLplot in Gtk DrawingAreas

Update: I have turned this post into a Github project called Gtkmm-PLplot, under the GPLv3

Scientific plotting and Gtk+ have never been good friends. Over the years there have been a number of efforts like GtkExtra, GtkDataBox, GoatPlot, GtkGraph… Unfortunately these packages all appear to share a lack of active development (most are in fact abandoned), poor to non-existent documentation and in most cases lack of support for Gtk+3.

For about a year or two I have been a minor contributor to GtkExtra, and I had started migrating the plotting widgets to Gtk+3 but I had to abandon my effort due to lack of knowledge of Cairo and Gtk+ internals, and more importantly a lack of time. I still use it in the Gtk+2 based GUI of my XMI-MSIM package though.

I have recently been looking into alternatives to GtkExtra as I switched to Gtk+3 (actually Gtkmm3, as I mostly code in C++ nowadays) for my new projects (e.g. BAM-utils). I came across a package called PLplot, which I used a long time ago for producing simple plots in PNG format in command-line utilities.

This package used to come with Gnome bindings that ensured easy integration into Gnome based GUIs but it looks like they have been removed some time ago (I am referring here to PLplot 5.11.0, the current stable release). Instead the documentation mentions support for a cairo driver, which can be embedded in Gtk based user interfaces. Currently the documentation that covers this feature is limited to a README file and a minimal cairo-only example that demonstrates how to use the PLplot cairo driver through the extcairo device by producing a PostScript file.

PLplot has the great advantage over the aforementioned Gtk+ based packages that it is very actively developed by several volunteers and comes with decent documentation, including a lot of examples written in the many languages supported by the toolkit. On the downside, the developers seem to have the unfortunate habit of regularly breaking backwards compatibility in their releases, as they did in their latest 5.11.0 release both at the API level as well as by changing the name of the pkg-config packages…

In this blogpost I will explain how I managed to embed PLplot into Gtkmm through the GtkDrawingArea widget class, and demonstrate it using a small program that allows one to zoom in on the data, save the plot in a file and print it.

Read on →

HDF5 on Windows: UTF-8 filenames support

In a previous post, I have detailed my efforts in getting HDF5 compiled on Windows 64-bit using the MinGW-w64 compiler, which is currently (still) unsupported by the HDF5 developers. Now, I am reporting on yet another problem I encountered with HDF5 on Windows, that is blatantly being ignored by the developers: UTF-8 filename support.

This was brought to my attention by a Japanese user of my software package XMI-MSIM, who ran into trouble running the program when logged in with a username consisting of Japanese characters. The problem could be traced back to an HDF5 file that has to be opened, which is located in a subdirectory of the user’s homedirectory. After investigating the HDF5 code, it became quite clear that this issue is caused by the internal use of the _open function, which is known not to support the Unicode UTF-8 characterset, necessary to represent the Japanese characters. The HDF5 website confirmed this issue with the following statement:

Read on →

Gtk2 64-bit Windows Runtime Environment Installer: now on GitHub!

As predicted in my first post on the Gtk2 64-bit Windows Runtime Environment Installer, I have indeed ventured into compiling Gtk (2.24.24) and all its dependencies, mainly because I was getting increasingly unhappy with the old Gtk 2.22.1 based bundle that is being distributed by the Gtk project. It was in fact this very bundle that I was using to generate the installer I announced in my first post on this subject.

So, after spending about 10 hours of compiling (and recompiling a couple of times when I didn’t get the configure options right) of more than a dozen software packages, I ended up with a fully working (at least until now…) collection of DLLs. I updated the code from my initial installer to include the new DLLs and uploaded the new installer here. I also uploaded a zip-file (sdk) containing all executables, DLLs, linking libraries (.dll.a) and headers, that can be used by anyone not willing to replicate my compilation effort. In fact I recommend that people using this installer to distribute the Gtk runtime along with their own program, to compile and link against this sdk, in order to avoid any link issues at runtime…

You may notice that I did not use the exact same dependencies in my compilation stack as those offered by the official Gtk bundle. I followed the Hexchat flowchart and ended up with additional dependencies of libffi and harfbuzz. libexpat was replaced by libxml2. This explains why the new installer is considerably larger than the previous one.

Last but not least, I forked the original repository of the installer from its sourceforge repository and created my own personal copy on GitHub. Check the README file for more information.


Building a CUDA device function library with autotools

I have recently started a new development branch in my xraylib project, where I will gradually be adding support for nVidia’s CUDA platform. Essentially the goal is to create a library containing CUDA device function equivalents of most functions that are currently offered by xraylib. Since there is not really much point to include functions that deal with strings, I will be leaving those out.

Compiling such a library has been supported by nVidia since CUDA version 5.5, but for some reason I never got their examples working. However, their recent 6.0 release seems to have fixed things (at least for me). In this post I will share some of the tricks I had to come up with to generate such a library.

Read on →

Gtk2 64-bit Windows Runtime Environment Installer

Update: this post has been largely superseded by http://tschoonj.github.io/blog/2014/09/29/gtk2-64-bit-windows-runtime-environment-installer-now-on-github

As I already mentioned in my last post, I have been busy compiling software with MinGW-w64 for the Windows 64-bit platform.f. One of the packages I rely on is Gtk+ (version 2 for now), and its many dependencies. Fortunately however, the Gtk+ website offers binaries for this architecture, so I didn’t have to bother compiling them myself.

In the end I also managed to build all the other dependencies I needed for XMI-MSIM, so the last step for me was to update my NSIS installer for Windows 64-bit machines. It didn’t take long for me to realize that one of the crucial components I install is the Gtk Windows Runtime Environment Installer, which really facilitates installing Gtk+ based applications. Unfortunately, Alexander Shaduri, the developer only distributes a 32-bit version of this package, and he has assured me through some emailing that he had no plans to write a 64-bit version due to the experimental nature of the packages that are produced by the Gtk+ people.

This being said, I still needed such an installer so I wrote one based on his NSIS script, using the official 64-bit Gtk+ packages for Windows 64-bit. The only major change I introduced was dropping support for the compatibility DLLs, which were redundant anyway. I am sharing the modified NSIS script as Github Gist so start hacking away at it. This script has some dependencies, apart from the DLLs, which are necessary to build the installer. Since I didn’t change them at all, use the files that the original developer is sharing on sourceforge.

The installer itself can be obtained here. If you would like to see an example on how to use this installer from within your own NSIS script, have a look at e.g. my XMI-MSIM Windows 64-bit installer.

Read on →

Building a 64-bit version of HDF5 with MinGW-w64

I have finally decided to try building a 64-bit version of the Windows package of XMI-MSIM. As is the case for my 32-bit builds, I am using the MinGW compiler as provided by TDM-GCC, although obviously this time I opted for the 64-bit variant. Now XMI-MSIM has a considerable amount of dependencies (GTK+, GSL, HDF5, libxml2, libxslt, curl, json-glib and others), I am again forced to compile these all again from source before I can even think of compiling my actualy software. One exception here is GTK+ and its dependencies since the good people from the GTK project are offering 64-bit builds (compiled with MinGW-w64) on their website.

In the past, the HDF5 build has always cost me most of my time, compared to the other packages, to get it compiled on MSYS, especially the Fortran bindings part. Now since I have dropped the dependency for these Fortran bindings in my latest version, because of my past experiences, I hoped that compiling them with MinGW-w64 might somehow have become easier.

Alas…

Read on →

Building libtool modules: Python bindings

One of the software packages (xraylib) I am working on, consists of a C-library with bindings to a number of other languages such as Python, Fortran 2003, IDL, Java, Ruby, Lua and Perl. Apart from IDL and Fortran 2003, the source code for these bindings is generated automatically using swig, although with considerable language-specific modifications to the swig interface file.

The bindings source code afterwards needs to be compiled into a dynamically loadable library, which will then be loaded at runtime by the program whenever the user needs to use a function or variable from the bindings (actually from the underlying C-library, but exposed through the swig generated bindings). As a rule, each programming language that supports such dynamically loadable libraries comes with specific instructions on how to generate the libraries from the bindings source code, often using platform independent tools.

However, these tools never seem to integrate well with autoconf and automake and lead to quite complicated Makefile.am files. A considerable more easy approach would consist of relying on libtool’s functionality of generating these dynamically loadable libraries (called modules in the libtool documentation). In this series of posts I will share my experiences on generating bindings to the aforementioned languages using libtool in a relatively clear and easy way. All the code has been tested on Linux and Mac OS X. It may also work on Windows though, provided you install a bash shell with the required tools.

In this first post I will discuss Python extension modules.

Read on →

Exporting and printing a GtkExtra plot canvas

My software package XMI-MSIM relies on GtkExtra for the visualization of the simulated X-ray fluorescence spectra. This library contains a number of useful widgets, such as a spreadsheet widget and a number of plotting tools, for your Gtk2 based GUI’s. Recently, I have become a minor contributor of this project, and I have been particularly active in improving the buildsystem and also generating RPM files for a number of Fedora/RHEL based distros, and DEB files for several recent releases of Ubuntu and Debian. Full instructions on how to obtain access to these packages, which are hosted by my old research group, can be found here.

In todays blog post I am presenting an example of how one can relatively easily add printing and exporting capabilities to GtkExtra’s GtkPlotCanvas widget. I will demonstrate this with a short program that I have written that contains a plot of a sine function, and two buttons that allow the user to invoke the dialog windows that will set up the exporting and the printing of the canvas, respectively.

Read on →