Bug 101614

Trevor,

have you tried compiling with devenv?  We don't have a problem compiling 17.1.3 there.

We haven't tried compiling with mingw.

Hi George,
Unfortunately no, I have not. 

My solution is Qt-based and I use QMake across all platforms (OSX, Linux and Win). For Win, I use the MinGW/GCC toolchain.

Just the check, I ran the installation on the latest git source as there were some updates to simd16intrin.h since 17.1.3. However, the behaviour is the same. The build fails in precisely the same place. You can see the log output here: https://gist.github.com/trevorsandy/b7c3275dabe6494c247e3ebece28ebbd

Perhaps the SWR driver is not currently buildable on Win/MinGW ?

I've seen several Win/MSVS build configurations, including those in the source for AppVeyor and Travis, but none appear to target osmesa with llvm and swr drivers - which is the configuration I'm looking to build.

Cheers,

Trevor,

We haven't targeted mingw as a compile platform for windows, yet.  That's not to say we are not going to, we just haven't gotten to it yet.

The compiler error seems to be in swr proper, meaning that it is not related to osmesa.  OSMesa should be independent from swr, so if llvmpipe compiles with osmesa on windows, swr should too.

Having said that, I just tried to compile mesa/swr with mingw64, and I am having problems, too, but at a different location.  Namely:

C:\Python27\python.exe 'C:\Python27\Scripts\scons.py swr=1 -j 1 build=debug toolchain=mingw libgl-gdi osmesa
scons: Reading SConscript files ...
Checking for MSVC ...  no
Checking for GCC ...  yes
Checking for Clang ...  no
Checking for win_flex ...  no
Checking for win_bison ...  no
scons: Found LLVM version 3.9
Checking for X11 (x11 xext xdamage >= 1.1 xfixes glproto >= 1.4.13 dri2proto >= 2.8)... no
Checking for XCB (x11-xcb xcb-glx >= 1.8.1 xcb-dri2 >= 1.8)... no
Checking for XF86VIDMODE (xxf86vm)... no
Checking for DRM (libdrm >= 2.4.75)... no
scons: done reading SConscript files.
scons: Building targets ...
  Archiving build\windows-x86_64-debug\mesa\libmesa.a ...
The command line is too long.
scons: *** [build\windows-x86_64-debug\mesa\libmesa.a] Error 1
scons: building terminated because of errors.

Have you hit this?  (that's an incremental build; building with -j 1 after I hit a compiler error for the full build)

George,

I haven't personally experienced this error but I did come across it in several places. In fact, Bug 94072 - error: The command line is too long when building MESA on Windows with MinGW-W64 I think covers this behaviour.

I likely haven't seen it because I'm only building osmesa - not the full build. It looks like those experiencing this behaviour are also building libgl-gdi. Nevertheless, the problem appears to be rooted in SCons which is probably where the behaviour I'm experiencing is rooted also. I say this because osmesa w/ llvm (swr on Ubuntu Linux 16.04 and llvmpipe on OSX Sierra both run to completion without issue.

Try your build without libgl-gdi - just osmesa. If you have a MSYS/MinGW env with the required pre-reqs installed, my script (https://github.com/trevorsandy/osmesa-install/blob/master/osmesa-install.sh) automates the build quite nicely. You tweak and run quite efficiently to better narrow down the cause of failure.

Cheers,

George,

One more point. I did not use the windows command environment. My toolchain is MSYS2/Mingw64. My command environment is Bash. Looking at your command output, it looks like you are using mingw64 under the native Windows command environment ?

I tried compiling just osmesa, but still got the same issue.

I've always had trouble using bash with python on windows (bash from Cygwin).

Regardless of whether I use python from the windows python distribution of python from Cygwin, I always run into trouble with windows vs linux paths.  If I use windows python, then bash does not understand the windows paths that python uses, and if I use the Cygwin python, then scons tries to append windows paths on top of Cygwin paths.

Which bash and which python are you using?

Thanks!

You can see in options listing of the logged output I posted all the component versions of my MSYS/MINGW dev env.

I'm using...
- MSYS bash at /usr/bin
- MSYS python at /usr/bin (#02 below)

For python, be careful. There are 3 instances available for MSYS/MinGW which can cause the native windows/posix confusion you describe. They are:

1. Native win32 python: From python.org, sys.platform == "win32", os.path.sep == "", os.name == "nt"
2. MSYS2 python: "msys2/python2" package  installed in /usr/bin/python, sys.platform == "msys", os.path.sep == "/", os.name == "posix"
3. mingw64 python:  "mingw64/mingw-w64-x86_64-python2" package, installed in /mingw64/bin/python, sys.platform == "win32", os.path.sep == "/", os.name == "nt"

If you use MSYS' package manager to setup your MSYS components, scons will be in base-devel so it will be deposited in /usr/bin so it will rightly use the python instance located there also - which happens to be the msys/posix instance. This instance of python will properly interpret your paths as unix paths. 

Native windows and mingw64 python will interpret paths as Windows paths. If you setup MinGW outside of MSYS like it is described here https://stackoverflow.com/questions/17871781/building-mesa-for-windows-7-mesa-9-1, your setup will likely not properly interpret unix paths even if you run it from a MSYS command shell.

MSYS2 Setup Details - FYI

MSYS2 Install:
• Download and install msys2-x86_64-<Date>.exe (see instructions at http://www.msys2.org/)
• Install to C:\msys64 (be sure to check 'Run MSYS2 now.' on the last install dialogue)
• Update package database: pacman -Syu pacman
• Close MSYS2 terminal and launch again MSYS2 MSYS (Shell) from Start Menu
• Update core system packages: pacman -Suu
• Install packages:
  o pacman -S base-devel git wget p7zip
  o pacman -S perl ruby python2 python2-mako mingw-w64-x86_64-toolchain
   • toolchain package options:  ^4-6 (exclude 4 - 6)
  o pacman -S mingw-w64-x86_64-python2-mako
  o pacman -S mingw-w64-x86_64-cmake
  o pacman -S scons
• Update your .bash_profile (setup your PATH)
  o Open .bash profile in your editor (C:\mingw64\home\<user>\.bash_profile

  # Set PATH so it includes MINGW_DIRS if they exists
  MINGW_PATHS=" \
   /mingw64/x86_64-w64-mingw32/bin \
   /mingw64/x86_64-w64-mingw32/include \
   /mingw64/x86_64-w64-mingw32/lib \
   /mingw64/bin \
   /mingw64/include \
   /mingw64/lib \
   "
  MINGW_DIRS=
  for dir in ${MINGW_PATHS}; do
   if [ -d "${dir}" ] ; then          
    MINGW_DIRS="${dir}:${MINGW_DIRS}"
   fi
  done
  export PATH="${MINGW_DIRS}:${PATH}"

• Setup MSYS2 terminal shortcut
  o Navigate to C:\msys\usr\bin and right-click mintty.exe
  o From your context menu, select 'pin to start'
  o In start menu, find shortcut and select 'pin to taskbar'
• OSMesa install script:
  o mkdir -p ~/projects; cd ~/projects
  o git clone https://github.com/trevorsandy/osmesa-install.git
  o mkdir -p osmesa-install/build
• Install script setup
  o mkdir -p /opt/llvm
  o mkdir -p /opt/osmesa
• Execute install script 
  o Launch MSYS2 terminal
  o Select Mingw-w64 64 bit at selection dialogue
  o cd ~/projects/osmesa-install/build
  o ../osmesa-install.sh
• Notes: 
• python2-mako is installed in 2 places, MSYS2 base (/usr/bin) and mingw-w64 (/mingw64/bin)
• Review well the osmesa-insall.sh script to understand the options available
• At script launch, review well the env options output before accepting to continue execution.
~~~fin~~~

Trevor, please try and get [ideally] all of the patches upstreamed. Be that in mesa[1] or the respective project. Frédéric Devernay may be able to lend a hand?

Any of your local patches may be changing Mesa is subtle ways, that people cannot foresee.

On the Travis/Appveyor part - yes, not everything is covered. Do send a patch when you get bored. Note that by default both libgl-gdi and osmesa are build, even if they are not explicitly listed on the command line.

On the issue in question - I'm suspecting that it's MINGW64 version/Windows specific, since cross-compiling llvmpipe/swr/osmesa on my Arch box has been fine for a while. I've been doing that for a while as part of the releasing process, even before 17.1.3 was out ;-)

Thanks
Emil
[1] https://www.mesa3d.org/submittingpatches.html

Ok, Thanks to George's patch (0001-mingw-fixes.patch), the behaviour reported in this ticket is fixed.

I applied the set below (excluding the conflict) to revision 89d4008ac85714bab8c49974377fd37970f6d66a of the master branch and the build is now running smoothly. 

# add_pi.patch \
# gallium-once-flag.patch \
# gallium-osmesa-threadsafe.patch \
# glapi-getproc-mangled.patch \
# install-GL-headers.patch \
# lp_scene-safe.patch \
# mesa-glversion-override.patch \
# osmesa-gallium-driver.patch \
# redefinition-of-typedef-nirshader.patch \
# scons25.patch \
# scons-llvm-3-9-libs.patch \
# swr-sched.patch \
#           scons-swr-cc-arch.patch \ (conflict)
# msys2_scons_fix.patch \
# 0001-mingw-fixes.patch \

You can see all the patches except George's at my github repo. I will add George's patch once he confirms it is released.

So for me, this issue is now resolved.

Cheers,

I wouldn't mark it as resolved yet.

Since it is filed as a mesa bug, it does reflect the status of mesa, not the status of any user's solution.

Re-opening until fix is checked in in mesa master, and due testing is performed.

Emil,

you said that cross compiling software drivers on arch works for you.

Have you tried running the generated binaries?  I've created a patch for Trevor, but for some reason the swr driver is not loading for me (llvmpipe works fine).

In addition, debug scons builds with mingw/msys2 don't include symbols.  Do you know if that is a known issue?

I don't want to submit the patch until swr runs on mingw/msys2.

(In reply to Trevor SANDY from comment #10)
> Ok, Thanks to George's patch (0001-mingw-fixes.patch), the behaviour
> reported in this ticket is fixed.
> 
> I applied the set below (excluding the conflict) to revision
> 89d4008ac85714bab8c49974377fd37970f6d66a of the master branch and the build
> is now running smoothly. 
> 
> # add_pi.patch \
> # gallium-once-flag.patch \
> # gallium-osmesa-threadsafe.patch \
> # glapi-getproc-mangled.patch \
> # install-GL-headers.patch \
> # lp_scene-safe.patch \
> # mesa-glversion-override.patch \
> # osmesa-gallium-driver.patch \
> # redefinition-of-typedef-nirshader.patch \
> # scons25.patch \
> # scons-llvm-3-9-libs.patch \
> # swr-sched.patch \
> #           scons-swr-cc-arch.patch \ (conflict)
> # msys2_scons_fix.patch \
> # 0001-mingw-fixes.patch \
> 
You really want to get these sorted and pushed upstream. While most devs will be happy to pull misc trees and play around, you're not doing yourself or them a favour.

That's enough harping from me on the topic.

George, I'm only build-testing those, I have no setup to actually test those.

No idea on the on the symbols part. In general scons or mingw (the w64 of course), Brian and Jose are the people to check with.

Hi Guys !

I would like to come back on this ticket because, for me, the problem I reported is completely fixed by the patch George Kyriazis provided. However, I noticed his patch is not applied to anything after 17.1.3. For example the latest release exactly reproduces the original issue.

Anyhow, I'm not sure what process must run before this fix can be applied but I can report that the issue is completely resolved when George's patch is applied to revision 89d4008ac85714bab8c49974377fd37970f6d66a - along with the patch set below. I'm not sure on what branch this revision is attached as it is no longer on the master/best-so-far branch. 

It would be great to have this patch on 17.2!

Patch set (https://github.com/trevorsandy/osmesa-install):
# add_pi.patch \
# gallium-once-flag.patch \
# gallium-osmesa-threadsafe.patch \
# glapi-getproc-mangled.patch \
# install-GL-headers.patch \
# lp_scene-safe.patch \
# mesa-glversion-override.patch \
# osmesa-gallium-driver.patch \
# redefinition-of-typedef-nirshader.patch \
# scons25.patch \
# scons-llvm-3-9-libs.patch \
# swr-sched.patch \
# msys2_scons_fix.patch \
# 0001-mingw-fixes.patch - George's patch is not in my repo - sorry:(

I can successfully run a quick osmesa rendering test for swr, lvmpipe and softpipe driver settings with the following MSYS2/MinGW DevEnv settings:

Mesa build options for platform MINGW64_NT-10.0:
- ((NOTE)) DEMO MODE - ALL BUILD LOGIC DISABLED!
- build date: 20/09/2017 03:38:51
- build and install mesa: No
- using existing Mesa 17.1.3.X 64 libraries in /opt/osmesa
- release, non-debug build
- non-mangled
- swr Gallium renderer
- build and install llvm: No
- using existing llvm 4.0.1 in /opt/llvm
- reuse built source at rebuild
- execute osmesa demo: Yes
- build and install glut: No
- using existing glut version: 3.0.0 (using freeglut)
- using existing libglu32.dll in /c/Windows/System32
- CC: gcc
- CXX: g++
- CFLAGS: -O3 -m64
- CXXFLAGS: -O3 -m64
- msys2 version: 2017.05-1
- mingw version: 2.28-2
- gcc version: 6.3.0-1
- cmake version: 3.9.1-1
- scons version: 2.5.1-1
- bison/yacc version: 3.0.4-1
- python2 version: 2.7.13-2
- python2-mako version: 1.0.7-1
- libxml2 version: 2.9.2-3
- interactive: Yes
- silent logging
- log file: /c/Users/Trevor/Projects/osmesa-mingw/osmesa-install_27.log
* copying binary files to demo location...
** copied libglu32.dll
** copied osmesa.dll
** copied swrAVX.dll
** copied swrAVX2.dll
env GAL_DRIVER=swr ./osdemo32 image.tga
channel sizes: 32 32 32 32
depth bits 24
osdemo, writing tga file 
all done
---------------------------------------------
* Demo mesa-demos-8.3.0 build and execution completed. Time Elapsed: 0hrs 0min 13sec
* Done!. osmesa-install ran to completion. Time Elapsed: 0hrs 0min 13sec

Cheers,
Trevor

Closing, since realistically we won't do any more work pre mesa-18.