Aria2

From Mandriva Community Wiki

Jump to: navigation, search
Aria2 was integrated as download manager in urpmi for 2 reasons
  • Increase the download speed by selecting the user's best mirror
  • Fallback when errors occur on a mirror (Out of sync, unreachable, etc)


Contents

Advantages

The actual svn version of aria2 includes a new feature which allows to store statistics on download servers. These statistics are parsed by a URI Selector, which provides URLs to the Download Engine of Aria2. A new URI Selector, named "Adaptive" was developed to improve download performance for small files like RPM. (The first use of aria2 was to download big files like ISOs)


Difference between aria2 and a classic download manager

First of all, aria2 is a segmented download manager, allowing to download the same file from multiple servers at the same time. It also supports the metalink files. With this kind of file, we are able to specify for each file to download one or more mirrors. By adding some sources for each file, aria2 is able to select a faster mirror for the user and to fall back when an error occurs on one of them.


Examples

With a classic download manger (like curl), you specify your favorite mirror (for example ftp://ftp.free.fr). Each file required by urpmi will be retrieved from this server. But some problems can occur :

  • The mirror is down
  • The mirror is out of sync
  • The mirror is slow

For each of these cases, aria2 has a solution :

  • If the mirror is down or out of sync, aria2 will automatically fall back to the next best server for the user.
  • Each mirror is tested, and statistics are reported in order to select the best performance mirror on the next download.


Mirrors evaluation

For each file download, aria2 uses between 1 or 3 connections depending on the file size. As said before, aria2 is a segmented download manager. So, for each file to download, this one is segmented into 'n' segments of 1Mb. The number of connections used by aria2 is never more than the number of segments of the file.

For example, a file of 2Mb is split into 2 segments of 1Mb, so aria2 will use at most 2 connections. So :

  • file size <= 1Mb : One connection
  • 1Mb < file size <= 2Mb : Two connections
  • 2Mb < file size : Three connections

The first connection uses one of the best mirrors according to the statistics stored in the aria2's server stat file. The others (connection 2 and 3) are used to test mirrors which need to be tested again, or which have not been tested yet. These connections allow aria2 to update statistics in the cache file.


How it works

First, in our case, Aria2 needs metalink files to work correctly. This job is done by urpmi thanks to the information contained in the mirrorlist file.

Urpmi

Mirrorlist files can be found on this type of URL : http://api.mandriva.com/mirrors/basic.cooker.i586.list


continent=EU,zone=FR,country=France,city=Paris,latitude=48.87,longitude=2.33,version=cooker,arch=i586,type=distrib,url=ftp://ftp.free.fr/mirrors/ftp.mandriva.com/MandrivaLinux/devel/cooker/i586
continent=EU,zone=FR,country=France,city=Paris,latitude=48.87,longitude=2.33,version=cooker,arch=i586,type=distrib,url=http://ftp.free.fr/mirrors/ftp.mandriva.com/MandrivaLinux/devel/cooker/i586
...

As you can see, it contains some information for each mirror like coordinates. Urpmi retrieves geolocalisation information of the user from his timezone (e.g.: France/Paris) and computes the proximity between him and each mirror. A random is applied on each proximity, the list is sorted and stored by urpmi in its cache file (/var/cache/urpmi/mirrors.cache).

Each time urpmi needs to download some files, it will build a metalink file using the information contained in the mirrors.cache file. It takes the 8 best mirrors for that. The proximity is converted into a value between 0 and 100 and this is used as the preference field of metalink file.



<?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" generator="URPMI" xmlns="http://www.metalinker.org/">
<files>
	<file name="drakxtools-curses-11.35-1mdv2009.0.i586.rpm">
            <resources>
		<url type="http" preference="99" location="fr">http://ftp.free.fr/mirrors/ftp.mandriva.com/MandrivaLinux/devel/cooker/i586/media/main/release/drakxtools-curses-11.35-1mdv2009.0.i586.rpm</url>
		<url type="http" preference="98" location="fr">http://ftp.u-picardie.fr/mirror/ftp.mandriva.com/MandrivaLinux/devel/cooker/i586/media/main/release/drakxtools-curses-11.35-1mdv2009.0.i586.rpm</url>
		<url type="ftp" preference="97" location="fr">ftp://ftp.u-picardie.fr/mirror/ftp.mandriva.com/MandrivaLinux/devel/cooker/i586/media/main/release/drakxtools-curses-11.35-1mdv2009.0.i586.rpm</url>
		<url type="ftp" preference="96" location="fr">ftp://ftp.ciril.fr/pub/linux/mandrakelinux/devel/cooker/i586/media/main/release/drakxtools-curses-11.35-1mdv2009.0.i586.rpm</url>
		<url type="ftp" preference="95" location="fr">ftp://fr2.rpmfind.net/linux/Mandrakelinux/devel/cooker/i586/media/main/release/drakxtools-curses-11.35-1mdv2009.0.i586.rpm</url>
		<url type="ftp" preference="94" location="fr">ftp://ftp.u-strasbg.fr/pub/linux/distributions/Mandriva/devel/cooker/i586/media/main/release/drakxtools-curses-11.35-1mdv2009.0.i586.rpm</url>
		<url type="http" preference="93" location="fr">http://fr2.rpmfind.net/linux/Mandrakelinux/devel/cooker/i586/media/main/release/drakxtools-curses-11.35-1mdv2009.0.i586.rpm</url>
		<url type="ftp" preference="92" location="nl">ftp://ftp.surfnet.nl/pub/os/Linux/distr/Mandrakelinux/devel/cooker/i586/media/main/release/drakxtools-curses-11.35-1mdv2009.0.i586.rpm</url>
	    </resources>
        </file>
	<file name="harddrake-11.35-1mdv2009.0.i586.rpm">
            <resources>
		<url type="http" preference="99" location="fr">http://ftp.free.fr/mirrors/ftp.mandriva.com/MandrivaLinux/devel/cooker/i586/media/main/release/harddrake-11.35-1mdv2009.0.i586.rpm</url>
		<url type="http" preference="98" location="fr">http://ftp.u-picardie.fr/mirror/ftp.mandriva.com/MandrivaLinux/devel/cooker/i586/media/main/release/harddrake-11.35-1mdv2009.0.i586.rpm</url>
		<url type="ftp" preference="97" location="fr">ftp://ftp.u-picardie.fr/mirror/ftp.mandriva.com/MandrivaLinux/devel/cooker/i586/media/main/release/harddrake-11.35-1mdv2009.0.i586.rpm</url>
		<url type="ftp" preference="96" location="fr">ftp://ftp.ciril.fr/pub/linux/mandrakelinux/devel/cooker/i586/media/main/release/harddrake-11.35-1mdv2009.0.i586.rpm</url>
		<url type="ftp" preference="95" location="fr">ftp://fr2.rpmfind.net/linux/Mandrakelinux/devel/cooker/i586/media/main/release/harddrake-11.35-1mdv2009.0.i586.rpm</url>
		<url type="ftp" preference="94" location="fr">ftp://ftp.u-strasbg.fr/pub/linux/distributions/Mandriva/devel/cooker/i586/media/main/release/harddrake-11.35-1mdv2009.0.i586.rpm</url>
		<url type="http" preference="93" location="fr">http://fr2.rpmfind.net/linux/Mandrakelinux/devel/cooker/i586/media/main/release/harddrake-11.35-1mdv2009.0.i586.rpm</url>
		<url type="ftp" preference="92" location="nl">ftp://ftp.surfnet.nl/pub/os/Linux/distr/Mandrakelinux/devel/cooker/i586/media/main/release/harddrake-11.35-1mdv2009.0.i586.rpm</url>
          </resources>
      </file>
</files>
</metalink>


Urpmi launches aria2 with this metalink file. Here are some important aria2 parameters:


/usr/bin/aria2c
  --metalink-enable-unique-protocol=false
  --max-tries 1
  --timeout 3
  --metalink-servers 3
  --uri-selector=adaptive
  --server-stat-of=/root/.aria2-adaptive-stats
  --server-stat-if=/root/.aria2-adaptive-stats
  --max-file-not-found 3
  file.metalink


Aria2

The aria2 version used by urpmi is actually the svn version revision 685, with a patch applied to add the URI Selector named "adaptive". First, I will explain each option listed above :

  • metalink-enable-unique-procotol=false allows aria2 to download from the same server on multiple procotols listed in the metalink file.
  • max-tries=1 allows only one other try on a server which has failed.
  • timeout=3 allows aria2 to switch to the next mirror if the current one does not answer after 3 seconds.
  • metalink-servers 3 allows aria2 to download a file from 3 servers at the same time at most.
  • uri-selector=adaptive indicates to aria2 to use the Adaptive URI Selector (more info about it below)
  • server-stat-if=/root/.aria2-adaptive-stats indicates where aria2 can find server statistics
  • server-stat-of=/root/.aria2-adaptive-stats indicates where aria2 will store updated server statistics
  • max-file-not-found=3 aborts download after not finding a file on 3 servers

What is a URI Selector

The URI Selector is the part of aria2 which selects which mirrors of the metalink file will be used for the download. In most cases, URI Selectors make these choices thanks to Server Statistics:

The Adaptive URI Selector

This URI Selector was written after many hours of brainstorming with Pixel ;), and certainly still needs some improvement. The RC1 release of Mandriva 2009.0 will permit us to evaluate the current algorithm.

Statistics reports

Each time aria2 downloads from a mirror, it reports download performance in the server stat file. The goal is to maintain some statistics for best mirror selection on the next launch.

Personal tools
In other languages