Page 1 of 2

Location of assets on Linux

PostPosted: Sat Jul 07, 2018 2:52 pm
by RobBaer
Although I'm not a Linux user myself, I'd like feedback from our Linux users on the location of the Makehuman user assets folder.

My understanding is that MakeHuman user assets are currently located by default in ~/makehuman/v1 on the Linux platform. The tilde indicates the user home directory, typically, /home/username

For Mac and Windows users, assets are typically located in a Documents folder. On the Mac, this is /Users/USERNAME/Documents/MakeHuman/v1/ On Windows 10, this is c:/user/%USERNAME%/Documents/makehuman/v1

A proposal was made to move the Linux Makehuman users assets folder one level down into a (the) Documents folder which would make it consistent across platforms and reduce "clutter" in the Linux user's home folder. The folder would now be located in the equivalent of ~/Documents/makehuman/v1

Any comments pro or con from our Linux base?

Re: Location of assets on Linux

PostPosted: Sat Jul 07, 2018 4:37 pm
by Aranuvir
There are some problems to consider: a) the Filesystem Hierarchy Standard defines the home folder as optional (Link^) and the content of the home folder is variable and not specified. If there is an equivalent to a Documents folder, its name often depends on the locale settings and there are no symbolic links by default folders, like for example on Windows systems.
However on Linux systems with a desktop environment (that can be assumed for most use cases of MakeHuman) that follows the freedesktop specifications (which is probably true for the most desktops in use^) the existence of a equivalent of a 'Documents' folder can be assumed and the configuration to locate the appropriate folder is stored in ~/.config/user-dirs.dirs. So a solution would be to parse the configuration file and to locate the Documents folder and in case of an failure it defaults to the current behavior.
BTW, the current concept of the code obliges to either hard code the location of the home folder, or to pass the information to the program on every startup via an environment variable or a command line option, which I consider an advanced use case. Letting the user choose the location of the home folder inside MakeHuman is probably not achievable without grater code refactoring, which, in my opinion, is not worth the effort.
Another thing to consider, is the fact that an user has to copy the current home folder to the new location. This can also be done by MakeHuman automatically, though the first start of MakeHuman can take while, depending on the size of the home folder to move (mine has grown to about 2G, though removing the git base-assets and the community assets cache could reduce some hundred MB of data).

Re: Location of assets on Linux

PostPosted: Sun Jul 08, 2018 6:47 pm
by punkduck
Hi there

I just migrated from MacOS back to Linux. One of the reasons, why I'm not as busy as always in this forum ... (not all applications are so easy to understand and migrate like MH :D ). I preferred a Xubuntu 18.04 version ...

Reason: my Apple was too small and furthermore I never understood why I should give all information who is using the box to Apple ... now I'm free again ;)

I also migrated all my MakeHuman stuff to Linux by simply using the backup device and copied it to Documents/MakeHuman.

You should take this into consideration: The Linux systems has a Documents directory. But it is using the native name (in German it is "Dokumente"), on a Mac you also see "Dokumente" in the file browser, but it is only an alias to Documents. In CLI mode using 'ls' you only will notice "Documents". On the Linux box it is the real directory name.

I copied the mac files on the new box and now "Documents" was created parallel to "Dokumente". I'm a lazy guy, so I linked makehuman to Documents/MakeHuman . It was nearly working ... the only thing not running is "settings.ini" in the v1-path. The pathnames loaddir, savedir, Scene_Editor_FileDlgPath and exportdir are absolute pathnames. I changed them one time before I started MH. That's all I've to do.

So if we can get rid off these absolute pathnames we can do both, copy the stuff to Documents/MakeHuman AND create the link. But the data must be migrated once and please do not put this code in MH directly. It will stay there, another dead branch, which will make the code confusing, when you look at it in 3 years.

But, hey, where is the problem. Linux users are able to do such a migration. I also had to fix the bug in module3d.py manually ... otherwise no export is running. And for Blender we've to use the non-debian version. Otherwise no Nvidia support is added. I guess we also may be able to enter a pathname during installation ... :mrgreen:

For Apple we always had the problem, that makeclothes creates its clothes in <HOME>/makehuman, so you've either to copy the clothes or change the export path in makeclothes.

As well as it might not be designed like that: working with a copy allowed me only to copy .mhclo and .obj file without destroying the .mhmat file again. I often had to do it because of small geometry changes in the end ...

There is one main aspect: a user should always know, where his data is located. A thing all Windows-, MacOS- or Linux users have in common. If he does backups without this data, he can start from scratch ... so don't hide this information. It is like not telling a pedestrian that on a road there could be cars ... :shock:

Re: Location of assets on Linux

PostPosted: Mon Jul 09, 2018 4:28 pm
by wolgade
Well, localization might be the big pitfall here as Aranuvir already mentioned.
punkduck wrote:In CLI mode using 'ls' you only will notice "Documents". On the Linux box it is the real directory name.

Punkduck and me have a quite similar setup with one difference. My home directory grew over years. Whenever I install a new distro, home just gets mounted. As a result I have a directory called "Dokumente". It's a real directory not an alias or symlink. Plain to see when using the terminal. In my opinion it's a big advantage not to hide real structures from the users. There are a lot of post on this forum from people on Windows not finding the right folders. I have to admit, that I run into the same trouble when I have to use Windows. These guys in Redmond are as good as it gets in keeping you away from understanding your system.
punkduck wrote:So if we can get rid off these absolute pathnames we can do both, copy the stuff to Documents/MakeHuman AND create the link. But the data must be migrated once and please do not put this code in MH directly.

Yes, please don't put this migration code anywhere. "mv" and "cp" have been around for a while and are easy to use. I have seen so many automagic stuff creating a perfect disaster.
Aranuvir wrote:BTW, the current concept of the code obliges to either hard code the location of the home folder, or to pass the information to the program on every startup via an environment variable or a command line option, which I consider an advanced use case.

If the current code can read the path from an environment variable or a command line argument, it shouldn't be too hard to get the path from a config file. This would fit anyone's needs. Just edit a single line and save your stuff wherever you want.

Re: Location of assets on Linux

PostPosted: Mon Jul 09, 2018 6:56 pm
by Aranuvir
We are not the only ones having to deal with the “Documents” folder on a Linux system. Freedesktop has created a solution for the problem. @Punkduck and @Wolgade: could you check if there is a ~/.config/user-dirs.dirs file on your system and check its contend?

wolgade wrote:
Aranuvir wrote:BTW, the current concept of the code obliges to either hard code the location of the home folder, or to pass the information to the program on every startup via an environment variable or a command line option, which I consider an advanced use case.

If the current code can read the path from an environment variable or a command line argument, it shouldn't be too hard to get the path from a config file. This would fit anyone's needs. Just edit a single line and save your stuff wherever you want.

The option already exists (it was introduced for the PortableApps (Windows Tm) experiment) and is a command line option for the current Python2. But it was too intrusive in the core code, making Jonas worrying about his headless branch. So it is now an environment variable in the current Python3 branch and if you want to move the makehuman folder forever, you can reconfigure bash.rc to set a “MH_HOME_LOCATION” environment variable ;). Honestly, I don’t think these are solutions for common users, and I do not want to give support for this advanced options.
Separating configuration files from the rest of the user’s content (for example a .makehuman.rc file or ./config/makehuman folder) would be the most flexible solution, allowing the user to store his content at any place of choice. Though, this will probably lead to greater code refactoring and to discussions, where to locate the config files on other platforms. IMHO, storing everything in one folder is the simplest solution for a multi platform application.
Currently there is already a working solution to move the makehuman folder to a “Documents” equivalent based on parsing user-dirs.dirs, with a fall back solution, if the “Documents” folder can’t be located.
I agree with punkduck, that moving the content of the user’s folder to a new location should be done by an external script, and we probably will need some cleanup actions, anyway, IIRC the Python3 code will crash on Python2 npz-files.

Robs intention was, to find out if anyone feels the need to move the makehuman folder to another location. Personally, I’m indifferent about this question, but if we agree to move the folder, I’d suggest the solution described above, that might not work for everyone.

BTW, would you mind giving the current development branch a try? I’d like to know if you experiance any OpenGl issues on your systems (the master branch currently will not move the location of the makehuman folder).
Some new features: plugins in the users folder and loading at run time, saving targets, weight estimation based on bsa, scripting console can make use of Jupyter/qtconsole (if existent on the system), sorted tags, fixed numpy exporting issue…
Most annoying known issues: OpenGl failures. Not working FBX exporter.

Re: Location of assets on Linux

PostPosted: Mon Jul 09, 2018 8:16 pm
by wolgade
Aranuvir wrote:could you check if there is a ~/.config/user-dirs.dirs file on your system and check its contend?

~/.config/user-dirs.dirs
Code: Select all
# This file is written by xdg-user-dirs-update
# If you want to change or add directories, just edit the line you're
# interested in. All local changes will be retained on the next run.
# Format is XDG_xxx_DIR="$HOME/yyy", where yyy is a shell-escaped
# homedir-relative path, or XDG_xxx_DIR="/yyy", where /yyy is an
# absolute path. No other format is supported.
#
XDG_DESKTOP_DIR="$HOME/Schreibtisch"
XDG_DOWNLOAD_DIR="$HOME/Downloads"
XDG_TEMPLATES_DIR="$HOME/Vorlagen"
XDG_PUBLICSHARE_DIR="$HOME/Öffentlich"
XDG_DOCUMENTS_DIR="$HOME/Dokumente"
XDG_MUSIC_DIR="$HOME/Musik"
XDG_PICTURES_DIR="$HOME/Bilder"
XDG_VIDEOS_DIR="$HOME/Videos"

There's another file called user-dirs.locale
Code: Select all
de_DE

Aranuvir wrote:if you want to move the makehuman folder forever, you can reconfigure bash.rc to set a “MH_HOME_LOCATION” environment variable ;). Honestly, I don’t think these are solutions for common users,

This is a quite uncomtortable solution, although you don't need to edit bash.rc. Makehuman gets started by a shell-skript. You could set MH_HOME_LOCATION there.
Aranuvir wrote:Separating configuration files from the rest of the user’s content (for example a .makehuman.rc file or ./config/makehuman folder) would be the most flexible solution,

This would be the Linux way to do it, but this is not portable at all. The Windows way to do it would be a registry entry. Does anyone want this?
Aranuvir wrote:I agree with punkduck, that moving the content of the user’s folder to a new location should be done by an external script, and we probably will need some cleanup actions, anyway,

Do we really need a script for "mv makehuman/ Dokumente/"? Cleaning up is a different story.
Aranuvir wrote:IIRC the Python3 code will crash on Python2 npz-files.

I already have a cleannpz.sh. It's part of the code you get from bitbucket.
Aranuvir wrote:Robs intention was, to find out if anyone feels the need to move the makehuman folder to another location. Personally, I’m indifferent about this question,

I don't feel the need to change the directory. If you consider a MH-asset to be a "document", then why don't you consider the photos you took to be "documents" as well. They should reside under /Documents/Pictures. This is nonsense, of course, but you might get my point. In my documents-folder you'll find, well, documents. The manual for the mainboard of my computer, letters I've written and some other stuff that's worth to keep a copy of.

Aranuvir wrote:BTW, would you mind giving the current development branch a try?

Sure. Is there a way to do so without messing up my working MH?

Re: Location of assets on Linux

PostPosted: Mon Jul 09, 2018 10:06 pm
by Aranuvir
[...] They should reside under /Documents/Pictures. This is nonsense, of course, but you might get my point. In my documents-folder you'll find, well, documents. [...]

Of course. The discussion came up on a user request (Link^) and if I get the specs of the Filesytem Hierarchy Standard correctly, an application should not create a folder in /home/$USER, without $USER's consent. And getting the consent is not so easy to achieve for several reasons. So, moving to the Documents folder would be the least bad solution...
This is a quite uncomfortable solution, although you don't need to edit bash.rc. Makehuman gets started by a shell-skript. You could set MH_HOME_LOCATION there.

Indeed. On the one hand I was just kidding. On the other, I must admit, I forgot about the shell script. Since about two years my code gets started by PyCharm.
This would be the Linux way to do it, but this is not portable at all. The Windows way to do it would be a registry entry. Does anyone want this?

IIRC I've seen applications creating a .config folder somewhere in AppData. Though, I cannot find an example, atm. And AppData seems to be the right place to put configuration files anyway. But as I stated above I do not vote for this option...
Do we really need a script for "mv makehuman/ Dokumente/"?

You won't need it, I won't need it, unfortunately there are many user who need something to click on with an icon (though, probably less on a Linux system)

Sure. Is there a way to do so without messing up my working MH?

get the code from github and place it in a folder of your choice. I'd recommend to use git for download:
Code: Select all
git clone https://github.com/makehumancommunity/makehuman.git

Install the dependencies: python3-numpy, python3-opengl, python3-pyqt5, python3-pyqt5.opengl, python3-pyqt5.qtsvg. Optional, if you like it: Jupyter and qtconsole. On first run the new version will create a v1py3 folder in ~/makehuman with a bunch of empty subfolders, despite some well known configuration and log files. For convenance, I'd recommend to create the v1py3 folder by hand and then copy the other stuff from v1 to v1py3, before first start. Maybe you should delete the old ini files. Missing folders will be created automatically on first run. That should be all to know.

Re: Location of assets on Linux

PostPosted: Mon Jul 30, 2018 9:35 pm
by punkduck
I have to give an additional hint.

I wrote, that I used a symbolic link to put the user files in another directory than $HOME/makehuman (which is not uncommon e.g. if you use an additional disc e.g. only for data or a NFS mounted storage).

It looked all good until I created the first new assets with my new box (yesterday, the riding equipment), Unfortunately the generation of .npz and .mhxpy files is creating funny pathnames in these files (going up and down the file system hierarchy). The next start of MH nothing of the new clothes really worked. (Now I created the link the other way round :? )

We really should put the absolute paths all into the settings.ini AND let the user decide where to put his data.
Best way is to ask while installation (maybe using a default entry like Documents/MakeHuman if the user is too lazy to select an own name).
Then the code should not go up and down the path, it should start from the pathnames given in the .ini file.

Unfortunately these python os-functions look all very simple and people use commonly relative paths. A common opinion is: configuration files with paths are not even needed.

Really? Maybe on a desktop. If you need data on another volume it is impossible to create this on a relative base. The worst I had seen in reality (and it is not really funny) was a python program writing mega-bytes of logfiles into a root volume (because the command resides there and the programmer used this relative path functions). Well, after this I urged the programmer to rewrite the code and to use an old-fashioned configuration file ... :evil:

Re: Location of assets on Linux

PostPosted: Sat Aug 04, 2018 10:24 am
by Aranuvir
The discussion has gone quite far away from its initial intention, though that's ok. I'd like to break down the discussion to two questions, now:


1) Do you think the homefolder for MakeHuman on Linux should be moved to another DEFAULT location and where should it be located (e.g. documents folder)?


2) Do you think the location of the home folder should be configurable? I'm providing some solutions that are IMHO suitable to achieve the goal. The behavior will be equal on all operating systems. Choose one of these options:

A) Always ask for the location on start up (starting MakeHuman from command line will be mandatory)

B) Scanning the whole filesystem, using the first folder matching a hard coded pattern as homefolder. Alternatively if it matches more than one folder: randomly selecting one of those folders

C) Breaking up configuration files, with using another configuration file whose hard coded location won't negotiable (e.g. for Linux somewhere in ~/.config) pointing to the homefolder. Alternatively: falling back to old behavior, if the configuration file cannot be found.

D) Leave everything as is, if someone feels the pressure to move the home location the env var option can be used...

Re: Location of assets on Linux

PostPosted: Sat Aug 04, 2018 12:57 pm
by wolgade
Aranuvir wrote:1) Do you think the homefolder for MakeHuman on Linux should be moved to another DEFAULT location and where should it be located (e.g. documents folder)?

Well, I already said, that I don't feel the need to move the folder and explained why. On the other hand: If FHS recommends "Documents", it's a good idea to have a program stick to standards. Keep in mind that "Documents" is for user generated data, not for user specific config files.
Aranuvir wrote:2) Do you think the location of the home folder should be configurable? I'm providing some solutions that are IMHO suitable to achieve the goal. The behavior will be equal on all operating systems. Choose one of these options:

A) Always ask for the location on start up (starting MakeHuman from command line will be mandatory)

The worst idea I can think of.
Aranuvir wrote:B) Scanning the whole filesystem, using the first folder matching a hard coded pattern as homefolder. Alternatively if it matches more than one folder: randomly selecting one of those folders

Sounds like some kind of folder lottery to me. Scanning the whole file system might take a while on my machine and anything but $HOME/* might be readable by MH, but MH lacks the rights to write there for very good reasons.
Aranuvir wrote:C) Breaking up configuration files, with using another configuration file whose hard coded location won't negotiable (e.g. for Linux somewhere in ~/.config) pointing to the homefolder. Alternatively: falling back to old behavior, if the configuration file cannot be found.

The only reasonable way to do it, BUT: If the configuration file can't be found (on a first run e.g), then create it and set default values.
Aranuvir wrote:D) Leave everything as is, if someone feels the pressure to move the home location the env var option can be used...

I consider it necessary to make the default folder configurable, even if we keep anything else the old way. It would have been great to have somthing like this when I tried to play around with the Python3 branch. I ended up creating a new user just for MH-Py3 only to force MH to leave my existing data alone.