Latest Posts

Topic: Daily-build PPA and disk space usage

hjd

Topic Opener
Joined: 2011-06-12, 19:24
Posts: 164
Ranking
At home in WL-forums
Location: bugs.launchpad.net/widelands
Posted at: 2013-07-20, 11:46

Hi,

I've been looking at the daily-build PPA a bit recently, and I've noticed that the packages page routinely reports more than 100% disk space usage. Since I don't think we should be using more disk space than we have available, I've looked into how to improve this situation:

  1. I've already deleted the old packages for Ubuntu 11.04 and 11.10 since those releases had reached End of Life quite some time ago and new packages weren't built anyways. This helped some of course, but not enough.

  2. Most of the time, the PPA reports it's using 4.1 out of 4.0 GiB. That doesn't seem too bad, but sometimes it reports way higher numbers (I've seen > 6 GiB ). I believe this is when the new packages have been uploaded and the older ones are still present, waiting to be deleted some time later. If it is indeed due to the older packages, I don't know how serious it is that a double set of packages exceed the storage limit for a short period of time. I guess we need to ask the Launchpad people about this. That doesn't change the fact that we are above the limit when things are normal, though.

  3. Looking at the PPA for stable builds, I notice that it only has 2 GiB available, which from what I could find is the default size. Since the daily-build archive has twice that amount already, have we applied for increased storage space in the past?

A simple fix would of course be to remove one of the older series to get below the storage limit. However, with an upcoming Ubuntu release in October, there will be a new series created and if we were to add that we would be back to where we are today. On the other hand, if we keep the current series, we will not have the space to add a new one regardless.

Suggestions or input? I guess we could ask the Launchpad people how to deal with this, the documentation mentions that if you need more space, you can ask them. I would first like to know whether we have already asked for increased storage space in the past. If we did, it might be relevant to look at that request before asking again, especially since I don't know what they would say to a second request.


Ships!

Top Quote
QCS

Joined: 2009-12-29, 21:47
Posts: 256
Ranking
Tribe Member
Posted at: 2013-07-20, 12:16

I would like to understand more how this comes (this is mostly new to me).

First of all it would be good to know how this much space (I am currently reading "25 binary packages (2.6 GiB)" which is probably not regular since a lot of builds failed lately) is claimed/distributed. Is there a way to see?

Edit: What I think is always possible: Create several PPAs for distributions (say, widelands-daily for quantal/raring/saucy, widelands-daily-lts for lucid/precise)

Edited: 2013-07-20, 12:21

CMake is evil.

Top Quote
hjd

Topic Opener
Joined: 2011-06-12, 19:24
Posts: 164
Ranking
At home in WL-forums
Location: bugs.launchpad.net/widelands
Posted at: 2013-07-20, 12:44

QCS: Click the triangle next to a series to expand it; it will list details and the package files. I can't get the numbers to add up when adding together the listed sizes there though, it's not enough to reach 2.6.

You might be on to something here, because if we compare with the stable PPA, it's four series are using a lot less.

I've thought about multiple PPAs as well, but I would rather avoid it if we can. It is a lot easier to say "want a development version, add this" rather than trying to figure out which one to use.


Ships!

Top Quote
QCS

Joined: 2009-12-29, 21:47
Posts: 256
Ranking
Tribe Member
Posted at: 2013-07-20, 13:09

Looking at the stable PPA, I can clearly see where the numbers come from. Add the numbers and you reach exactly the repository size.

Now looking at the daily...

Binaries: - a data _all.deb of 175MB - a debug executable _i386.deb of 25MB - a debug executable _amd64.deb of 25MB - an executable _i386.deb of 2MB - an executable _amd64.deb of 2MB

makes roughly 230MB per series for the binary packages - times five for the five series makes up for 1.1GB.

Sources: Only a sources.tar.gz of 180MB - times five for the five series makes up for 900MB.

So, from what I see, the reason must really be "old" package files which have not been deleted yet. Useful package files add up to about 2GB, and we should be fine with 4GB repo size. I guess the cleaning process is quite slow and may not be fit for a really daily update.


CMake is evil.

Top Quote
QCS

Joined: 2009-12-29, 21:47
Posts: 256
Ranking
Tribe Member
Posted at: 2013-07-20, 13:27

Ah well... checking the server:

http://ppa.launchpad.net/widelands-dev/widelands-daily/ubuntu/pool/main/w/widelands/

It clearly shows there are "old" files (bzr6643, bzr6638, bzr6637, bzr6632).

So this is now solved. Now to hjd's second question... do we have to ask Launchpad for more space, or do we not? face-wink.png


CMake is evil.

Top Quote
hjd

Topic Opener
Joined: 2011-06-12, 19:24
Posts: 164
Ranking
At home in WL-forums
Location: bugs.launchpad.net/widelands
Posted at: 2013-07-23, 20:20

And now it's down to 2.1 GiB, which is well within the limits. Maybe the bot responsible for deleting older builds came back from vacation? I'll keep an eye out in case the disk usage starts rising again, but it looks fixed for now.

Thanks for checking QCS, that definitely confirms it was the older builds sticking around. How did you come across that url?


Ships!

Top Quote
hjd

Topic Opener
Joined: 2011-06-12, 19:24
Posts: 164
Ranking
At home in WL-forums
Location: bugs.launchpad.net/widelands
Posted at: 2014-02-16, 13:32

I asked the Launchpad people about this earlier today, and they increased the size limit to 8 GiB. With this increase we should be safely within the limit for quite some time.


Ships!

Top Quote
SirVer

Joined: 2009-02-19, 14:18
Posts: 1445
Ranking
One Elder of Players
Location: Germany - Munich
Posted at: 2014-02-16, 14:09

Awesome! Thanks for that. And cool that they increased our quota - they are really helpful there at canonical.


Top Quote