Request for comments, SVN in deb files

[joepal]

Today we (again) ran into the quota roof at TuxFamily cause the distribution packages keep growing. Manuel has solved this by asking nicely to get the quota increased, but that's a temporary solution really.

I'm thinking to rewrite some of the nightly linux builds and break out the data section. Instead I'm thinking to include a postinst script in the deb file that automatically downloads the data from SVN. However, this kind of breaks the way deb files usually function, so I'd like some comments on it.

The logic would be:

* Deb files includes everything except data (data being targets, textures, meshes etc)
* Deb file gets a hard dependency on "subversion"
* At postinstall it checks if /usr/share/makehuman/data/.svn exists
* If it does it cd:s to the install directory and does a SVN update
* Else it checks out the data directory from google code

Upside is

a) distribution binary shrinks by up to 90%
b) network load on tuxfamily decreases
c) users only download what they need
d) more or less transparent to users

Downsides

a) Deb files usually don't work this way, so it might be considered a bit ugly
b) First time installation will be significantly longer
c) network connection is required in order to install (although it'd be pretty difficult to get the nightly without it in the first place)

So... comments?

[mflerackers]

For textures (which are really big) we recently made an automatic updater inside mh. The first time you access them they are downloaded, after that they are checked (by date and hash) whether they are uptodate. We might do something like this for the other data files too.

[Manuel]

For textures (which are really big) we recently made an automatic updater inside mh. The first time you access them they are downloaded, after that they are checked (by date and hash) whether they are uptodate. We might do something like this for the other data files too.

It's a good idea, but for nightly they should download from svn repository...

[joepal]

I've made a first attempt at using SVN to download the data dir, see

http://makehuman.blogspot.com/2011/08/i ... -debs.html

[joepal]

For reference, this is the postinst script:

Code: Select all

export currentdir=`pwd`

if [ -e "/usr/share/makehuman/data/.svn" ]
then
  echo "NOW UPDATING DATA FROM MAKEHUMAN SVN"
  echo
  cd /usr/share/makehuman/data
  svn update
else
  echo "NOW DOWNLOADING DATA FROM MAKEHUMAN SVN"
  echo
  rm -rf /usr/share/makehuman/data
  mkdir -p /usr/share/makehuman/data
  svn checkout http://makehuman.googlecode.com/svn/trunk/makehuman/data /usr/share/makehuman/data
fi

echo
echo "NOW DOWNLOADING SKINS"
echo

mkdir -p /usr/share/makehuman/data/skins
cd /usr/share/makehuman/data/skins
rm -f *
wget http://www.makehuman.org/download/skins/media.ini
dos2unix media.ini
for x in `cat media.ini`;
do wget http://www.makehuman.org/download/skins/$x;
done;

cd $currentdir


[Bolche]

I ran into an error while trying to do a fresh install on Ubuntu 11.04.
The error (in Portuguese) is:

dpkg (sub-processo): não foi possível executar script post-installation instalado (/var/lib/dpkg/info/makehuman-nightly.postinst): Erro no formato exec

The important part is the end: "Error on exec format".

I edited the file /var/lib/dpkg/info/makehuman-nightly.postinst and added a

Code: Select all

#!/bin/sh

And it appeared to do the trick.
Cheers

[joepal]

Ouch.

Thanks. I've fixed this now. Seems I forgot the #! thing on i386, it was there on amd64 which is why it worked for me.

This change will propagate tonight.

[cousteau]

Hmm, that kind of post-install downloaded/generated data usually goes on /var rather than /usr, if I recall correctly... anyway, it's OK this way.

Another alternative to this solution would have been to put the architecture-independent, rarely changed data on a separate deb package, so that both i386 and amd64 depend on that single package (which, in addition, wouldn't update very frequently, thus reducing the bandwidth usage). Although maybe you already considered this.

Would it be possible to make this _not_ depend on dos2unix, which is not available on Ubuntu Lucid? (I use Maverick, but I guess some people there still uses Lucid - anyway, if this doesn't affect the alphas, I guess it's ok)

[joepal]

For architecture-independent package... Problem is that the data is what changes most frequently. So that'd be a very big and often updated package, so there would not be much gain.

I agree with the /var thing. It's mostly that MH expect data to be a subdir of the installation folder. Maybe it could be solved with a symlink? I'll look into it.

For dos2unix, the annoyance comes from the format of the list of textures on the MH server, which very unfortunately is in a dos/windows format. The postinst script is written in bash and needs to parse this file, and bash really pukes on dos/windows text files. If not dos2unix, I'd have to include another dependency, possibly awk, sed, perl or some other text formatting tool. Dos2unix was simply the easiest solution since it was available on my debian box (which is why I assumed it'd also be available in ubuntu).

[cousteau]

Well, using a symlink would be a nice solution IMO.

About not using sed (which is present on virtually any Linux distribution) nor dos2unix (which exists for Ubuntu Maverick but not for Lucid) nor Python (which is already a dependency for makehuman but is not as handy for shell scripts), what about something like this?

Code: Select all

while read line; do
    something "${line%^M}"   # the code should contain an actual CR, not a literal "^M"
done

or, if _all_ the lines end with a CR, you could avoid writing a literal CR, which is kind of ugly, and just use

Code: Select all

"${line%?}"

which will strip the trailing character on the variable $line.
(This code works on bash, dash, and I think any Unix shell).