Requirements
- ComputeCpp CE 0.1.2
- Python 2.7
- Ubuntu 14.04.05 LTS Headless Server
- AMD R9 Nano GPU
Intro
This short post aims to guide through set-up process for TensorFlow with OpenCL support. It’s “copy-paste” type of post. There is nothing ground breaking in it, all the instructions can be found across other websites / forums.
The aim was to put all relevant information in one place which sould make it more convenient/easier to go through the set-up process.
Dependencies
The assumption is that you have a vanilla server with 14.04.05 LTS Ubuntu64 installed.
Install Ubuntu kernel 3.19.0-79-generic using:
1 2 3 4 | $ sudo apt-get install linux-image-3.19.0-79-generic linux-image-extra-3.19.0-79-generic linux-headers-3.19.0-79-generic $ sudo apt-get remove linux-image-4.2.0-42-generic $ sudo update-grub $ sudo reboot |
Install Git,LLVM and Build Essential using:
1 | $ sudo apt-get install build-essential git llvm |
Install JDK8 & Bazel using:
1 2 3 4 5 6 7 | $ sudo add-apt-repository ppa:webupd8team/java $ sudo apt-get update $ sudo apt-get install oracle-java8-installer $ echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list $ curl https://bazel.build/bazel-release.pub.gpg | sudo apt-key add - $ sudo apt-get update && sudo apt-get install bazel $ sudo apt-get upgrade bazel |
Install the AMD OpenCL 1.2 headers using:
1 2 3 4 5 6 | $ sudo apt-get install ocl-icd-opencl-dev opencl-headers $ wget --referer=http://support.amd.com https://www2.ati.com/drivers/linux/fglrx_15.302-0ubuntu1_amd64_ub_14.01.deb $ wget --referer=http://support.amd.com https://www2.ati.com/drivers/linux/fglrx-core_15.302-0ubuntu1_amd64_ub_14.01.deb $ sudo dpkg -i fglrx-core_15.302-0ubuntu1_amd64_ub_14.01.deb fglrx_15.302-0ubuntu1_amd64_ub_14.01.deb $ sudo adduser $(whoami) video $ sudo reboot |
Install ComputeCpp CE 0.1.2:
Visit https://computecpp.codeplay.com, register and download Ubuntu 14.04 release, unzip and copy the content – bin, include and lib are essential – to /usr/local/computecpp ( you will need sudo to do it )
Install Numpy using:
1 2 3 4 | $ sudo apt-get install python-numpy python-dev python-wheel python-mock python-psutil $ sudo pip install --upgrade pip $ sudo pip install py-cpuinfo portpicker numpy $ sudo pip install --upgrade scipy |
Set up
1 2 3 4 5 | $ export TF_NEED_OPENCL=1 $ export HOST_CXX_COMPILER=/usr/bin/g++-4.8 $ export HOST_C_COMPILER=/usr/bin/gcc-4.8 $ export COMPUTECPP_TOOLKIT_PATH=/usr/local/computecpp $ export COMPUTE=:0 |
You might consider adding the above to the ~/.bashrc and apply them to the current terminal seesion by:
1 | $ . ~/.bashrc |
Verification
At this point you should be able to verify your set up.
1 | $ clinfo |
Should have returned something along these lines:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 | Number of platforms: 1 Platform Profile: FULL_PROFILE Platform Version: OpenCL 2.0 AMD-APP (1912.5) Platform Name: AMD Accelerated Parallel Processing Platform Vendor: Advanced Micro Devices, Inc. Platform Extensions: cl_khr_icd cl_amd_event_callback cl_amd_offline_devices Platform Name: AMD Accelerated Parallel Processing Number of devices: 2 Device Type: CL_DEVICE_TYPE_GPU Vendor ID: 1002h Board name: Device Topology: PCI[ B#1, D#0, F#0 ] Max compute units: 64 Max work items dimensions: 3 Max work items[0]: 256 Max work items[1]: 256 Max work items[2]: 256 Max work group size: 256 Preferred vector width char: 4 Preferred vector width short: 2 Preferred vector width int: 1 Preferred vector width long: 1 Preferred vector width float: 1 Preferred vector width double: 1 Native vector width char: 4 Native vector width short: 2 Native vector width int: 1 Native vector width long: 1 Native vector width float: 1 Native vector width double: 1 Max clock frequency: 1000Mhz Address bits: 64 Max memory allocation: 3002336256 Image support: Yes Max number of images read arguments: 128 Max number of images write arguments: 64 Max image 2D width: 16384 Max image 2D height: 16384 Max image 3D width: 2048 Max image 3D height: 2048 Max image 3D depth: 2048 Max samplers within kernel: 16 Max size of kernel argument: 1024 Alignment (bits) of base address: 2048 Minimum alignment (bytes) for any datatype: 128 Single precision floating point capability Denorms: No Quiet NaNs: Yes Round to nearest even: Yes Round to zero: Yes Round to +ve and infinity: Yes IEEE754-2008 fused multiply-add: Yes Cache type: Read/Write Cache line size: 64 Cache size: 16384 Global memory size: 4242251136 Constant buffer size: 65536 Max number of constant args: 8 Local memory type: Scratchpad Local memory size: 32768 Max pipe arguments: 16 Max pipe active reservations: 16 Max pipe packet size: 3002336256 Max global variable size: 2702102528 Max global variable preferred total size: 4242251136 Max read/write image args: 64 Max on device events: 1024 Queue on device max size: 8388608 Max on device queues: 1 Queue on device preferred size: 262144 SVM capabilities: Coarse grain buffer: Yes Fine grain buffer: Yes Fine grain system: No Atomics: No Preferred platform atomic alignment: 0 Preferred global atomic alignment: 0 Preferred local atomic alignment: 0 Kernel Preferred work group size multiple: 64 Error correction support: 0 Unified memory for Host and Device: 0 Profiling timer resolution: 1 Device endianess: Little Available: Yes Compiler available: Yes Execution capabilities: Execute OpenCL kernels: Yes Execute native function: No Queue on Host properties: Out-of-Order: No Profiling : Yes Queue on Device properties: Out-of-Order: Yes Profiling : Yes Platform ID: 0x7f8a310c4a18 Name: Fiji Vendor: Advanced Micro Devices, Inc. Device OpenCL C version: OpenCL C 2.0 Driver version: 1912.5 (VM) Profile: FULL_PROFILE Version: OpenCL 2.0 AMD-APP (1912.5) Extensions: cl_khr_fp64 cl_amd_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_gl_sharing cl_khr_gl_depth_images cl_ext_atomic_counters_32 cl_amd_device_attribute_query cl_amd_vec3 cl_amd_printf cl_amd_media_ops cl_amd_media_ops2 cl_amd_popcnt cl_khr_image2d_from_buffer cl_khr_spir cl_khr_subgroups cl_khr_gl_event cl_khr_depth_images cl_khr_mipmap_image cl_khr_mipmap_image_writes [...] |
The above output means that there is an OpenCL platform with devices – GPU (The R9 Nano) and the CPU (mostlikely) available
Call computecpp_info which is located in the bin folder of the ComputeCpp installation:
1 | $ /usr/local/computecpp/bin/computecpp_info |
The output should look something like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 | ******************************************************************************** ComputeCpp Info (CE 0.1.2) ******************************************************************************** Toolchain information: GLIBCXX: 20150426 This version of libstdc++ is supported. ******************************************************************************** Device Info: Discovered 2 devices matching: platform : device type : -------------------------------------------------------------------------------- Device 0: Device is supported : YES - Tested internally by Codeplay Software Ltd. CL_DEVICE_NAME : Fiji CL_DEVICE_VENDOR : Advanced Micro Devices, Inc. CL_DRIVER_VERSION : 1912.5 (VM) CL_DEVICE_TYPE : CL_DEVICE_TYPE_GPU -------------------------------------------------------------------------------- Device 1: Device is supported : YES - Tested internally by Codeplay Software Ltd. CL_DEVICE_NAME : Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz CL_DEVICE_VENDOR : Intel(R) Corporation CL_DRIVER_VERSION : 1.2.0.25 CL_DEVICE_TYPE : CL_DEVICE_TYPE_CPU ******************************************************************************** |
At this point, we have verified that we have SYCL 1.2/OpenCL 1.2 supported device(s) aviable in our system.
TensorFlow
Finally, lets get TensorFlow, there are three places worth consideration:
- Official upstream (stable branch):
1
$ git clone https://github.com/tensorflow/tensorflow.git
- Development branch – Benoit Steiner fork OpenCL hub:
1
$ git clone https://github.com/benoitsteiner/tensorflow-opencl.git
- Experimental branch – Mine for high pace experimental:
1
$ git clone https://github.com/lukeiwanski/tensorflow-opencl.git
1 2 | $ cd tensorflow-opencl $ ./configure |
Configure is an interactive bash script that sets-up enviroment for TensorFlow.
If you exported the variables mentioned in “Set Up” section you should be able to press enter for all the options. Otherwise you will have to anwsare few questions:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | Please specify the location of python. [Default is /usr/bin/python]: Do you wish to build TensorFlow with Google Cloud Platform support? [y/N] No Google Cloud Platform support will be enabled for TensorFlow Do you wish to build TensorFlow with Hadoop File System support? [y/N] No Hadoop File System support will be enabled for TensorFlow Found possible Python library paths: /usr/local/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages Please input the desired Python library path to use. Default is [/usr/local/lib/python2.7/dist-packages] Using python library path: /usr/local/lib/python2.7/dist-packages Do you wish to build TensorFlow with OpenCL support? [y/N] y OpenCL support will be enabled for TensorFlow Do you wish to build TensorFlow with CUDA support? [y/N] No CUDA support will be enabled for TensorFlow Please specify which C++ compiler should be used as the host C++ compiler. [Default is /usr/bin/clang++-3.6 ]: /usr/bin/g++-3.6 Please specify which C compiler should be used as the host C compiler. [Default is /usr/bin/clang-3.6 ]: /usr/bin/gcc-3.6 Please specify the location where ComputeCpp for SYCL 1.2 is installed. [Default is /usr/local/computecpp]: |
After that is finished you can either run TensorFlow testsuit (recommended):
1 | $ bazel test --config=sycl -k --test_timeout 1600 -- //tensorflow/... -//tensorflow/contrib/... -//tensorflow/java/... -//tensorflow/compiler/... |
or, build pip package (pip package):
1 | $ bazel build --local_resources 2048,.5,1.0 -c opt --config=sycl //tensorflow/tools/pip_package:build_pip_package |
1 2 3 4 5 | $ mkdir _python_build $ cd _python_build $ ln -s ../bazel-bin/tensorflow/tools/pip_package/build_pip_package.runfiles/org_tensorflow/* . $ ln -s ../tensorflow/tools/pip_package/* . $ python setup.py develop |
Keep in mind OpenCL support at this point is experimental and in early stage.
Next Steps
TensorFlow introduction: https://www.tensorflow.org/get_started/basic_usage
OpenCL support status: https://github.com/tensorflow/tensorflow/issues/22
Thanks to @neuralsandwich for help with sorting this out.