New build for fixing unicode issues

[joepal]

We're trying to fix the long-standing issues with unicode path names. We've now come as far as having a test build available at http://download.tuxfamily.org/makehuman ... -win32.zip

We hope this will solve the problems arising in these two scenarios:

* A windows user has a username with unicode characters
* A windows user has placed Makehuman in a directory with unicode characters somewhere in the absolute full path

It would be helpful to get this stress tested a bit. To help out, download the build and try to make it fail by placing it in directories named with the weirdest characters you can think up. In the best of worlds it should work even if the directory name consist of a mix of emojis and chinese signs (but I haven't tested that specifically).

The relevant issue is discussed at http://bugtracker.makehumancommunity.org/issues/1029. If you managed to make MH crash, please post the name of the directory either there or as a comment to this post.

[Freehunter]

Not sure about using bugtracker so will post here.

Tried on Windows XP and Windows 8.1
MakeHuman v1.1.0 (r2046 c032a65a1ac9)
makehuman.exe MD5 = c1b9be5421fead70485d653ee05d3b96

Tried regular installations and created portable versions by installing regular portable and removing everything in makehuman folder and putting in new test release.
For each, installed in 1 folder with ASCII name and 2 folders including variations with Télé
Also created a character in each and saved with name Télé.

There were similar problems with both the regular and portable installations. Portable installation had additional problem loading a saved model due to path.

When portable folder name included Télé, the splash screen did not show the progress.

Both regular and portable installations failed, refusing to start, when folder was named TéléTélé.
Windows 8.1 displayed error:

Fatal Error
error loading Python DLL
(compressed path)/python27.dll
error code 126

Code: Select all

1 November 2016
MakeHuman v1.1.0 (r2046 c032a65a1ac9)
makehuman.exe  MD5 = c1b9be5421fead70485d653ee05d3b96

Windows XP SP3
REGULAR VERSION INSTALL

1) in makehuman launched OK, displays created model in Load OK
splash screen shows progress
saved character named Télé to default
closed then restart and try to load model and model displayed in File Chooser
load displays path as:
C:/Documents and Settings/username/My Documents/makehuman/v1/models
model loads

2) in MakeHumanTélé launched OK, displays created model in Load OK
splash screen shows progress
saved character named Télé to default
closed then restart and try to load model and model displayed in File Chooser
load displays path as:
C:/Documents and Settings/username/My Documents/makehuman/v1/models
model loads

3) in TéléTélé launched Fail

Task Manager shows:
makehuman.exe - runs then closes, no splash screen


PORTABLE VERSION INSTALL
( installed MH Portable then replace all files in makehuman folder with test version )

1) placed in MakeHumanPortable launched OK, displays created model in Load OK
splash screen displays progress
saved character named Télé to default
closed then restart and try to load model and model displayed in File Chooser
load displays path as:
D:/PortableApps/MakeHumanPortable/Data/makehuman/v1/models
model loads

2) placed in MakeHumanTélé launched OK, does not display created model in Load
splash screen does not display progress
saved character named Télé to default
closed then restart and try to load model but nothing shown in File Chooser, refresh does not change
load displays path as:
D:/PortableApps/MakeHumanTélé/Data/makehuman/v1/models
browse to model folder and shows up in file choser, path displayed as:
D:\PortableApps\MakeHumanTélé\Data\makehuman\v1\models
model loads

3) placed in TéléTélé launched Fail

Task Manager shows:
MakeHumanPortable.exe  - runs
makehuman.exe - runs then closes, no splash screen
MakeHumanPortable.exe  - closes


-------------------------------------


Windows 8.1

REGULAR VERSION INSTALL

1) in makehuman launched  OK, displays created model in Load OK
splash screen shows progress
saved character named Télé to default
closed then restart and try to load model and model displayed in File Chooser
load displays path as:
C:/Users/username/Documents/makehuman/v1/models
model loads

2) in MakeHumanTélé launched  OK, displays created model in Load OK
spash screen displays progress
saved character named Télé to default
closed then restart and try to load model and model displayed in File Chooser
load displays path as:
C:/Users/username/Documents/makehuman/v1/models
model loads

3) in TéléTélé launched  Fail

Task Manager shows:
makehuman.exe - runs (no splash screen)
popup error:

Fatal Error
error loading Python DLL
%path%/python27.dll
error code 126

close popup
makehuman.exe - closes


PORTABLE VERSION INSTALL
( installed MH Portable then replace all files in makehuman folder with test version )

1) placed in MakeHumanPortable launched OK, displays created model in Load OK
splash screen shows progress
saved character named Télé to default
closed then restart and try to load model and model displayed in File Chooser
load displays path as:
G:/PortableApps/MakeHumanPortable/Data/makehuman/v1/models
model loads

2) placed in MakeHumanTélé launched OK, does not display created model in Load
splash screen does not show progresss
saved character named Télé to default
closed then restart and try to load model but nothing shown in File Chooser, refresh does not change
load displays path as:
G:/PortableApps/MakeHumanTélé/Data/makehuman/v1/models
browse to model folder and shows up in file choser, path displayed as:
G:/PortableApps/MakeHumanTélé/Data/makehuman/v1/models
model loads
if close then restart and try to load model but nothing shown in File Chooser, refresh does not change
load displays path as:
G:/PortableApps/MakeHumanTélé/Data/makehuman/v1/models

3) placed in TéléTélé launched Fail

Task Manager shows:
MakeHumanPortable.exe  - runs
makehuman.exe - runs (no splash screen)
popup error:

Fatal Error
error loading Python DLL
%path%/python27.dll
error code 126

close popup
makehuman.exe - closes
MakeHumanPortable.exe  - closes



[loki1950]

Anything in the MH logs in /v1 that would be where one would actually see what caused the failure as is all you give binary info ie: it work or it did not windows it's self will not mention anything internal to MH.

Enjoy the Choice

[joepal]

Both regular and portable installations failed, refusing to start, when folder was named TéléTélé.

Do I read this right? It works if called "MakeHumanTélé" but not "TéléTélé"?

Huh.

Concerning having a saved model with an odd character in the name, yes that's still the case. I missed fixing that.

The same will most likely be true for any other file (clothes, skin, target...) which has deviant characters.

And thank you for your thorough testing. This is exactly the kind of testing which is needed.

I'll try to fix the model path thing and make a new build in the weekend.

[Freehunter]

Anything in the MH logs in /v1 that would be where one would actually see what caused the failure as is all you give binary info ie: it work or it did not windows it's self will not mention anything internal to MH.

According to http://www.makehumancommunity.org/wiki/FAQ:Where_are_my_MakeHuman_files_found_(where_is_my_HOME_directory)%3F logs are in the Base / Home folder.

The only one that fails completely is when placed in TéléTélé for both regular and portable installations.
No Base / Home folder is created, is there some place else for logs?

For portable, when placed in MakeHumanTélé, it runs and can create and save character. If close, restart, and try to load the character is not shown in File Chooser and path is shown as: G:/PortableApps/MakeHumanTélé/Data/makehuman/v1/models
If browse to folder, path shows correctly and model appears in File Chooser and can load model.
Would that be shown in the Logs?

Do I read this right? It works if called "MakeHumanTélé" but not "TéléTélé"?

Yes, it was weird. If regular or portable installed in "MakeHumanTélé" it works. If regular or portable installed in "TéléTélé" it does not work.
It was only a fluke that I tried "TéléTélé". Tried a similar name, "TéléTélél", got the same result. Weird!

Concerning having a saved model with an odd character in the name, yes that's still the case. I missed fixing that.

I'm not sure model name, Télé, was the problem would have to go back and check. From the way the path displayed, see above, I thought that was the problem.
If I browsed to the models folder it would display correctly in the path display and the model would show up in the File Chooser. Could select it and would load. Seemed the path was the problem.

Note on FAQ, the path shown for portable is X:\PortableApps\MakeHumanPortable\AppDate
it should be X:\PortableApps\MakeHumanPortable\Data

[joepal]

Jonas committed a bunch of new fixes to bitbucket now. I'll add my previous fixes to this and take another look at the filechooser.

There is no point in continued testing with the current build since it has known remaining issues that has since been fixed. But I'll try to take a closer look at the "TéléTélé" thing.

[Freehunter]

Concerning having a saved model with an odd character in the name, yes that's still the case. I missed fixing that.

I'm not sure model name, Télé, was the problem would have to go back and check. From the way the path displayed, see above, I thought that was the problem.
If I browsed to the models folder it would display correctly in the path display and the model would show up in the File Chooser. Could select it and would load. Seemed the path was the problem.

Confirm
With portable installed in MakeHumanTélé. Save character, to default, as either "Télé" or "Tele" and close. Restart and try to load saved character the File Chooser is empty and path is displayed as G:/PortableApps/MakeHumanTélé/Data/makehuman/v1/models.

Browse to models folder and path is displayed as G:/PortableApps/MakeHumanTélé/Data/makehuman/v1/models and model shows in File Choser. Select model and it loads.

[RobBaer]

According to http://www.makehumancommunity.org/wiki/FAQ:Where_are_my_MakeHuman_files_found_(where_is_my_HOME_directory)%3F logs are in the Base / Home folder.

and translates (for the regular MH version) to: c:\\Users\<username>\Documents\makehuman\v1\ with appropriate substitution for <username>.
You would know more than I about how the portable version works.

If you have ever had a regular version of MH, the Base folder probably already exists.

The only one that fails completely is when placed in TéléTélé for both regular and portable installations.
No Base / Home folder is created, is there some place else for logs?
...
Yes, it was weird. If regular or portable installed in "MakeHumanTélé" it works. If regular or portable installed in "TéléTélé" it does not work.
It was only a fluke that I tried "TéléTélé". Tried a similar name, "TéléTélél", got the same result. Weird!

The other thing that might be important, if this is really true, is your Windows Local settings, because different code pages can handle characters differently and I think this is the source of some of the problems in Windows environments. Just today, I encountered a situation where going from US English to British English locale and keyboard settings broke the command line syntax of another program. Perhaps in our efforts to replicate this behavior, it would be useful if you shared the regional settings you use (see control panel - Region dialog). And finally, thanks so much for the testing and specific example.

Same Test Windows 10
Finally, when I ran the new build in C:\TéléTélé on my Windows 10 system with US English settings, it started fine and ran without issue. However, there were a few things in the early portion of makehuman.log that might be useful. The handling of DLL Path makes unusual reference to some hex characters that look like they may stem from unicode translations, and it looks like line 415, in debug_dump might need fixing to accommodate unicode.

Code: Select all

[2016-11-02 10:11:39,900] logging->info():1622 -- MESSAGE -- Initialized logging
[2016-11-02 10:11:39,905] logging->debug():1630 -- DEBUG -- VERSION: 1.1.0
[2016-11-02 10:11:39,905] logging->debug():1630 -- DEBUG -- HG REVISION: r2046 (c032a65a1ac9) [data/VERSION static revision data]
[2016-11-02 10:11:39,905] logging->debug():1630 -- DEBUG -- SHORT VERSION: v1.1.0
[2016-11-02 10:11:39,905] logging->debug():1630 -- DEBUG -- BASEMESH VERSION: hm08
[2016-11-02 10:11:39,905] logging->debug():1630 -- DEBUG -- IS BUILT (FROZEN): Yes
[2016-11-02 10:11:39,905] logging->debug():1630 -- DEBUG -- IS RELEASE VERSION: Yes
[2016-11-02 10:11:39,907] logging->debug():1630 -- DEBUG -- WORKING DIRECTORY: C:\TéléTélé
[2016-11-02 10:11:39,907] logging->debug():1630 -- DEBUG -- HOME LOCATION: C:/Users/rbaer/Documents
[2016-11-02 10:11:39,907] logging->debug():1630 -- DEBUG -- PYTHON PATH: ./;./lib;./apps;./shared;./apps/gui;./core;C:\TLTL~1
[2016-11-02 10:11:39,907] logging->debug():1630 -- DEBUG -- DLL PATH: .;C:\TꭩTꭩ;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\QuickTime\QTSystem\;C:\Program Files\MiKTeX 2.9\miktex\bin\x64\;%USERPROFILE%\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Anaconda2;C:\Anaconda2\Scripts;C:\Anaconda2\Library\bin;C:\Program Files\Git\cmd;C:\Program Files\TortoiseGit\bin;C:\Program Files\TortoiseHg\
[2016-11-02 10:11:39,907] logging->error():1596 -- ERROR -- Could not create debug dump
Traceback (most recent call last):
  File "<string>", line 415, in debug_dump
  File "c:\jenkins\workspace\MH_unicode\buildscripts\win32\build\makehuman\out00-PYZ.pyz\debugdump", line 105, in reset
  File "c:\jenkins\workspace\MH_unicode\buildscripts\win32\build\makehuman\out00-PYZ.pyz\debugdump", line 81, in write
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 6: ordinal not in range(128)
[2016-11-02 10:11:40,292] OpenGL.acceleratesupport-><module>():16 -- MESSAGE -- No OpenGL_accelerate module loaded: No module named OpenGL_accelerate
[2016-11-02 10:11:40,298] OpenGL.arrays.formathandler->loadPlugin():54 -- MESSAGE -- Unable to load registered array format handler numeric
[2016-11-02 10:11:40,732] logging->debug():1630 -- DEBUG -- PYQT.VERSION: 4.8.6
[2016-11-02 10:11:40,857] logging->debug():1630 -- DEBUG -- PYQT.JPG_SUPPORT: supported
[2016-11-02 10:11:40,857] logging->debug():1630 -- DEBUG -- PYQT.SVG_SUPPORT: not supported
[2016-11-02 10:11:40,944] logging->debug():1630 -- DEBUG -- QT.PLUGIN_PATH: C:/TéléTélé/qt4_plugins;C:/TéléTélé
[2016-11-02 10:11:40,944] logging->debug():1630 -- DEBUG -- QT.PLUGIN_PATH_ENV:
[2016-11-02 10:11:40,944] logging->debug():1630 -- DEBUG -- QT.CONF: [Paths]
         Prefix = .
         Plugins = qt4_plugins
[2016-11-02 10:11:41,233] logging->debug():1630 -- DEBUG -- PYOPENGL.VERSION: 3.0.1
[2016-11-02 10:11:41,236] logging->debug():1630 -- DEBUG -- GL.VENDOR: NVIDIA Corporation
[2016-11-02 10:11:41,239] logging->debug():1630 -- DEBUG -- GL.RENDERER: Quadro K2000/PCIe/SSE2
[2016-11-02 10:11:41,242] logging->debug():1630 -- DEBUG -- GL.VERSION: 4.5.0 NVIDIA 353.82
[2016-11-02 10:11:41,243] OpenGL.extensions->getGLVersion():23 -- DEBUG -- OpenGL Version: 4.5.0 NVIDIA 353.82

...


[Freehunter]

Perhaps in our efforts to replicate this behavior, it would be useful if you shared the regional settings you use (see control panel - Region dialog)

Side note, I deleted the Base / Home folder before each test.

Settings:
XP
Region Options:
English (United States)

Languages
Default input languages
English (United States) - US

Installed services
English (United States)
--Advanced Text Services
----Ink Correction
--Keyboard
----US
------------------------------------------------------------------
Win 8.1
Region:
Format:
English (United States)
Match Windows display language (recommended)

Curent language for non-Unicode programs:
English (United States)

[RobBaer]

Hmmmm...

So you have the same locale settings as I and yet the new version doesn't work for you with Windows 8.1 and C:\TéléTélé but it does work for me with Windows 10. I am wondering if there could be something about the fact that the portable app version is likely also installed on your system and not on mine? [I'll ignore Windows XP for now since it is deprecated although I think it will likely work when other things work]

All this may be moot since the new r2080 bitbucket version seems to fix even the debug_dump log error I show below. Of course, not much testing on that yet