Archiv der Kategorie: ArchLinux DevStack

ArchLinux DevStack

Howto structure the Git-Repository (of


a little while ago, I felt the urge to propose a new git structure to my fellows at the team. I guess the explanation on the why could be helpful for others in the git community as well.

Currently we do have one repository with several packages in one git repository (server-core, server-extra, server-community). This was due to the fact, that the base-distribution (Arch Linux that is) uses SVN as their VCS and we were basically migrating their structure in an easy manner to our own systems.

Right now, we have one branch for each named release (e.g. redgum). Since we are planning to maintain at least two named releases (e.g. redgum and spruce) in parallel, and we have also a testing phase in each release, I suggested two branches for a named release (redgum, redgum-testing). This suggestion was accepted in the mailing list.

I have looked, how other distros with a named release are doing it, as well as for best practices in the git world (well, I really did this for the company I do work for). Fedora is using Git as their repository as well, and git is quite popular, so there is a wealth of information on this topic out there.


I propose to restructure the git repositories to the following structure:

server-core.git/kernel26-lts –> server-core/kernel26-lts.git
server-core.git/aif –> server-core/aif.git

In each of those newly created repositories we will then create at least two branches (e.g. redgum and redgum-testing) for each release. The master-Branch is a special branch, in that it is a development branch. Changes in there will go into release branches, but packages from the master-branches will never be put into a release-version directly. Packages always need to get created from the corresponding release branch, to establish a clean workflow for sign-offs of packages.

To let this work, we will need to adapt our dbscripts slightly, but this should not be too much of a hassel. The git repository could be restructured pretty much automatically and the current already available history could be migrated as well.

Furthermore I suggest to use a tool like repo or fedpkg, adapt it to our needs and implement this into our workflow. Package Updates can then be still done via the usual git commands, but to get a friendlier workflow, this tool can be used.


Let me first explain, why do we want to separate each package into its own repository like suggested by git and other distros like e.g. Fedora.

Git suggests to use a single repository for each project/package (a short explantion of this issue can be found on StackOverflow). Basically it comes to the point, that the usage of a repository per package is a best-practice in the git world, and best-pratices do have a sense, like shown in the previous link.

So where is our benefit in splitting repositories (server-core, server-extra and server-community) into several smaller ones (e.g. server-core/kernel26-lts.git)? IMHO the benefit is ease-of-use.

A TU (Trusted User) can then easily get one package, switch to the correct branch (e.g. redgum-testing) make the changes and push these changes back to the main repository. As soon as the package is out of the testing phase, the TU or somebody else can sign-off the changes and merge the package into the release branch (e.g. redgum). We could even say, that only signed packages are allowed to cross the border of the branches, I am unsure, if this could be done by git itself, but it could be a step in the workflow.

So, where is the difference now to the current structure?

Right now, as soon as a TU has made changes and pushed them into the right branch, other TUs will most probably work on other (or even the same) packages, meaning that the TU cannot easily go to the package and merge it into the release branch, he would merge all changes in the whole repository into the branch. The TU has the burden to look up the right commit-ID in the log of the whole repository (remember, git looks on the whole repository and not on sub-folders) and cherry-pick this change (even worse: changes). This does not seem to be right IMHO and it is against the best-practice.

If we are going to split all our repos (server-core, …) into several smaller git repositories, the look and feel of our repository handling will change slightly. To provide a consistent approach for all repositories (e.g. make a new branch for a new release), we will have to provide some tool to make the life of the TUs easier, thats what google is doing with repo and Fedora is doing with fedpkg (see: These tools offer some functionality not possible with git (at least as far as I know of, see: StackOverflow) like branching of several sub-projects, using the same branch for all sub-projects, committing changes in several sub-projects, etc. Also, there does exsits another alternatives (like git subtree, see: Git Subtree Documentation), these do not fit our needs very well.

Another advantage of this new structure is, that it could be used for automatic builds of packages as well. Right now, for example, starts a new build of a package by request. In the software engineering world there is something that is called Continuuous Integration (CI), which builds a package as soon as a change in the repository appears. The tool I am using for this right now (Hudson/Jenkins) does support the build of unix packages as well, so in the long run, it could be worthwhile to investigate this for our use-case as well.

So, WDYT? What are your thoughts on this one? I guess, that this is the way to go, even though it is not for RC4 or even redgum, but soon afterwards, I would suggest is the latest to implement this approach.

Hudson / Jenkins – Butler wars

Today I found a nice article about the split of Hudson and Jenking. It has a nice title and is really well written. Furthermore it shows some of the possible implications of this „war“.

Butler Wars.

Read on.

For more information about this story please also visit:

Hudson and Jenkins now on GitHub
Hudson vs. Jenkins – more thoughts
Hudson vs. Jenkins

Hudson and Jenkins now on GitHub

I think, the whole fork stuff with Hudson and Jenkins gets even more hilarious. In a vote, the Hudson community decided to move (take your breath) to

See InfoQ for more details about this movement. Now both Hudson as well as Jenkins are on github.

Jason (van Zyl) seem to have made already quite some changes to Hudson so, I would expect some major changes coming up for the next Hudson release. Of course these changes are related to (guess what?) Maven. These changes are going to diverge Hudson and Jenkins, and I guess that in the long run these two projects are going to diverge so much, that there is no „drop-in“ of Jenkins Plugins into Hudson and Vice-versa. On the other hand, Sonatype is known for their strong commitment to backward compatibility (see Maven 2/3), but is the same true for the Jenkins guys? I do hope so.

Still, new features developed for one of these forks will not directly appear in the other version. Since Sonatype seems to be doing some major restructuring of the hudson core, I doubt, that features on Hudson will easily be transportable to Jenkins, and vice versa.

Lets see whats going to happen 😉

For more information about this story please also visit:

Hudson vs. Jenkins – more thoughts
Hudson vs. Jenkins

Hudson vs. Jenkins – more thoughts

After having released the last post about Hudson vs. Jenkins, I have read quite a lot about this issue. Sonatype seems to have started a new „campaign“ on their relationship to and investments into hudson (see their latest banner on their homepage, where they state: „Maven + Hudson = a heart is shown here“).

Of course Sonatype (which is a heavily risk capital backed company) would like to earn money with this move. Some of the reasons are IMHO really well explained on Thomas Ferris Nicolaisen’s blogpost. Right now it looks like the whole community is in an real outcry about this move and especially the involvement of Sonatype and seem to be slightly annoyed by this move (see: Jerome’s Coffebreak).

Not all the critics of Jason on Hudson are 100% right, but he has a point. The only real question here is, why he (Sonatype that is) has not taken Benson (Sonatypes own Hudson Fork, without a real Homepage or anything like that, so it does not seem to „active“) and put it into the Hudson code? There could be several reasons, e.g. Koshuke was not really into making Hudson a totally Maven focused application.

Most probably Jason and Sonatype like to push Maven 3 into the market quite heavily. They are not the only company doing this, though. Hudson is a very good fit in Sonatypes software stack. So Sonatype is building up their software stack with Maven 3, Nexus, Hudson and Eclipse kind of stuff. Like Thomas Ferris Nicolaisen points out, there is still the option, that Oracle is going to buy-out Sonatype and that would most probably a good fit for Sonatypes investors 😉 (The whole discussion seems to be money related, doesn’t it?)

The least we can say is, that Hudson and especially Sonatype is now receiving a lot of attention due to this move, and they should have this attention, because otherwise Hudson would be dead (long live Jenkins ;-)).

On a sidenote, why did Jason and Sonatype not support Continuum instead of Hudson, Jason seemed to be a contributor to this project as well? The answer stays unsure, but I guess, Jason is going to put his bets all on market leaders to become a market leader on its own (well, with maven he is already, but what about Sonatype?). There are some strong competitors in the ALM market (is this the right name for this niche?) like e.g. the guys from MaestroDev, but they do not have this momentum as Sonatype has right now. The product portfolio seems to be (at least at a first glance) quite similar, only the selected/made tools are different. Maestro totally focusses on Apache Projects, whereas Sonatype implements its own products (at least this is true for Nexus).

So the question, who will win the race for the most usage in the CI market is still open. I strongly believe that there is a place for Hudson as well as Jenkins in the market. Oracle is going to offer professional support to Hudson and Sonatype is doing it already, something, Jenkins does not have right now (at least not that I am aware of). I and the company I work for am not a real contributor to the success of Sonatype and neither Oracle through buying their products or support, but we will stick to Hudson at least in the near future. Lets see the outcome of all this. Probably I re-decide on this one, depending who fixes the well known GZIP Problem first 😉

Another side note is some missing pieces in the Sonatype Software stack, after they do have already the whole Development Cycle in their portfolio (pretty close to IntelliJ and Atlassian, I guess). One piece of software still missing the from Sonatypes stack would be Sonar (IMHO), which is used in the IT industry quite often as well. So, lets see, if Sonar stays with the current company or if some other company is going to buy this piece of software as well, it would fit perfectly into the Sonatype offerings though.

So, the whole Java Open Source market, especially in the Development and Apllication Lifecycle management, seems to change right now. Sonatype seems to be one of the leaders in this market, next to Atlassian, which has a slightly different target group. It is nowadays all about earning money. Hopefully the support into Open Source projects will stay as it is right now from these kind of companies 😉

For more information about this story please also visit:

Hudson and Jenkins now on GitHub
Hudson vs. Jenkins

Hudson vs. Jenkins

In my last blog post, I mentioned, that we are still using Hudson instead of the new community fork Jenkins. You can ask now why we are doing this. This is mainly a personal decision, and it is not about Oracle and their „relationship“ to open source communities, but more about Sonatype and their involvement in the future of Hudson (see blog post here, here and here). I think that if the leader of Maven (Jason van Zyl) is investing time and resources into the hudson project, it is definitly worthwhile to stay there. Lets see if this is going to work out.

For more information about this story please also visit:

Hudson and Jenkins now on GitHub
Hudson vs. Jenkins – more thoughts

Maven CI Best Practices

Yesterday our Admin came to me and told me, that the Harddrive on our CI Server is nearly full. This is due to the fact that our Nexus Repository as well as our Hudson CI Server are installed on the same machine. Hudson is using it’s own locale repository and therefor all Snapshots are filling up the diskspace.

Hudson has its own local repository due to the fact, that we are following the best practices mentioned by Jason Van Zyl on this blog post.

What I did now, was to purge the local repository of Hudson. Unfortunately we are not following all Maven best practices and therefor one project (the master pom for all other projects) is then missing from the local repository. I hope that I can clean this up during the next couple of days (hey, we are in a pretty tough project right now, and therefor the time for this is not really there). This means, that we do have to build and „mvn install“ this project (base-build) directly after the purge. Furthermore I am cleaning up the Backups we are doing using ThinBackup (a small Hudson Plugin). Therefor we are using the following script:

#!/bin/sh -e
# fetches all changes in all repositories and sync it to redmine
cd /var/cache/hudson/mvn-repository
rm -rf *
cd /var/lib/hudson/jobs/base-build/workspace
su hudson -c 'mvn clean install' > /dev/null
cd /var/cache/hudson/backup
find . -mtime +5 -exec rm {} \;

Furthermore we are now using a scheduled Task in nexus to purge the Nexus Repository from any Snapshots older then 5 days, but keeping at least the latest snapshot (see another blog post).

After some more cleaning of e.g. log files and more, the disk is now half full (or half empty, depending on your point of view). This was more then enough for me.

Hope this helps.

IRC: nick/channel temporarily unavailable

I am using the public ZNC (an IRC bouncer) from for my access to the freenode IRC network. I have had a minor problem when I connected to my ZNC. Since my away nick was not registered to my account (which is triplem btw), I have had to „login“ each time when I connected to the ZNC and my nick was changed from triplem[a] to triplem. So, I started my IRC client (XChat btw) and forgot to type in my password again, this lead to a problem that my nick was temporarily unavailable. To resolve this I looked on the freenode page and found the following easy solution:

  1. Registered my away nick with /nick triplem[a] and /msg nickserv group (of course I needed to login to my account, which was not a problem, since several nicks are already grouped to my main nick and I just chose one of those).
  2. I released the original nick (triplem) with the following command: /msg nickserv release triplem , and everything was fine.

Hope this helps you as well. And if you would like to meet me on IRC, you can take a look into the following channels:

  • #shellium
  • #globalshellz
  • #netburst
  • #archserver

Probably we meet us there 😉

More thoughts on the ALM Stack

I am just thinking, that just providing some Toolset, like the ALM Stack is not enough. There is some documentation needed, not only for configuration, maintance and usage of the ALM Stack, but also some hints and best practices for the handling of projects.

Since we are already providing a wiki in our toolbox (Redmine contains a wiki), we will most probably provide an initial project in Redmine which will be configured the „right“ way and contain some documentation.

Most of these best practices are already documented elsewhere, so we will mostly provide links to these resources, as well as additional resources we can provide ourselves.

If you have any input on this, do not hesitate to contact me anytime.

My ALM thoughts

Yesterday we released a new ALM Stack based on ArchServer, which replaces the beforehand already known „Arch Linux Development Stack„. I guess the name change was needed due to the fact that the underlying architecture changed from Arch Linux to ArchServer and furthermore the ALM in the name shows more appropriate what the purpose of this stack really is.

The goal of this software stack is really, to release a full working stack of tools needed for a Application Lifecycle Management. The selection we chose (Maven, Hudson, Nexus, Sonar and Redmine) is pretty much a best-of-breed solution. One of the main ALM-Software components (IDE) is still missing though. Since this is not a server-side component, it would be really hard to provide this as well. But I guess we should provide an instance of Eclipse with all needed Plugins pre-installed (Mylyn with Redmine Plugin, Git, m2eclipse and probably more…). The same could be done for IntelliJ Idea, which I do like a lot as well, lately.

Something, which should be implemented as well, is most probably ReviewBoard or Gerrit for Reviewing code changes. This is IMHO not needed in all cases of software development, but in some (eg. the Android development and/or the KDE developers).

A real nice thing would be to provide a kind of „Installation procedure“ to install all components in one step with some personalizations as well (e.g. Hostname of the ALM Stack host). Furthermore some real documentation is needed, so that the „customer“ will find the answers to all their questions.

If you would like to participate, do not hesitate to write an email to me 😉

ArchServer – ALM Stack

Please take a closer look onto the newest addition of the ArchServer distribution. I have just published a ALM (Application Lifecycle Management) Stack for this distro with all ALM related tools already installed.

The stack contains the following components:

You can find all scripts to install such a stack (I named the stack Archlinux Development Stack earlier on, but it should now be called ArchServer ALM Stack), on Github. There is also additional information about this stack. In the near future I hope to add more components as well as more documentation. The documentation should then be also put into the wiki.

The Virtualbox image itself can be found on our mirror at The image is lzma-compressed, so you need lzma to decompress the image.