Python for Machine Learning and Maya Binary Compatibility

other

Many Machine Learning libraries support a Python interface. Many DCC applications support Python. But you will often find that these DCC applications and software libraries are binary incompatible. Here is how to solve this problem in Maya for Windows.

Autodesk Maya's Console

DLL error when importing the Theano library into Maya’s Python interpreter.

Python is the standard scripting language in the VFX and Animation industries. All prominent DCC applications like Maya, Houdini, C4D, Modo, and Nuke support it.

Another space where Python is quite a standard is in the Machine Learning and Data Science fields. According to Kaggle’s survey (https://www.kaggle.com/surveys/2017) 76% professionals in these fields use Python at work. In practicality, that means many Machine Learning frameworks support some sort of Python interface.

These frameworks don’t perform the training of ML models inside Python. They do such computations in C++ code running on the CPU or in CUDA code running on the GPU. But, they use Python as an interface for data manipulation and model configuration.

From this overlap, one would expect Machine Learning libraries to be readily available in a software like Maya, through Python. But that is not the case.

The first bump in the road

The first bump in the road is the Python version in Maya and many VFX and Animation software. This industry has chosen to not upgrade from Python 2, to Python 3. Apart from a few exceptions most DCC applications are bundled with some Python 2.7.x build. But in the Machine Learning field, some prominent players, like Google, have chosen to drop support for these legacy versions of Python.

TensorFlow, Google’s ML library is off-limits unless you access it through C++. We are left then with alternatives like Torch, Theano, CNTK, and others.

The second bump in the road

But then arrives the second bump in the road: binary compatibility.

Since these ML libraries perform their computation outside Python, their Python packages are not pure, and are composed, in part, of compiled code. As a standard Python 2 packages for Windows are compiled with Microsoft’s Visual Studio 2008. BUT, and this is a big BUT, Maya’s Python interpreter is compiled with whatever version of Visual Studio Maya was compiled with.

You can attest this by running a standard Python install and then running your Maya20**\bin\mayapy.exe interpreter. Note that MSC v.1*** is the compiler that has been used for that interpreter. In Linux you would get something like: GCC 4.*.*.

Microsoft Prompt running Python

Two Python interpreters, a default Windows x64 interpreter and the one bundled with Maya 2018.

Code that has been compiled with different versions of VS are, for the most part, not compatible. That means most of these libraries, in fact most of their dependencies, will crash when you try to run them through Maya.

So, what now?

So, if you want to run any of these libraries you will probably need to build many of their dependencies from source using the appropriate version of Visual Studio for your Maya installation. Here is a list:

  • Maya 2013-2014 – Visual Studio 2010 (MSC v.1600)
  • Maya 2015-2017 – Visual Studio 2012 (MSC v.1700)
  • Maya 2018 – Visual Studio 2015 (MSC v.1900)

After installing the appropriate version of Visual Studio you’ll need to download the libraries from their GitHub repos. Just Google for Github repo and the name of the library. Here is numpy’s repo as an example (https://github.com/numpy/numpy).

After that you will need to run the repo’s setup.py in build mode, so it builds a version of the repo using the proper compiler. There are some tricks involved in doing this. I recommend you read this article for a thorough explanation: http://around-the-corner.typepad.com/adn/2017/05/how-to-build-pillow-on-windows-with-maya-2017.html.  Apart from using the proper compiler you want setup.py to run on the proper Python interpreter (Maya’s). You might bump into additional complications in each library, which, for the most part, can be solved by reading and interpreting the error messages.

Share

Finally, if you want to save other people’s time you can then distribute this Maya friendly Python extensions building a Wheel, that can be installed using PIP. For that I recommend reading this article: https://packaging.python.org/tutorials/distributing-packages/#id74.

To get access to Maya compatible builds of libraries such as Numpy, ZeroMQ, Cython and Pandas join my mailing list below and I’ll send you links to those builds, ready for use.

Subscribe and get access to many Maya compatible Python libraries

* indicates required



Do you have experience with this issue in other platforms such as Linux and macOS? Would you care to share?

Have you built other Python packages? Can you share a link in the comments so we can extend this list?

Leave a Reply