mirrors/poky
1
0
Fork 0
mirror of https://git.yoctoproject.org/poky synced 2026-05-30 12:29:55 +00:00
Code Issues Projects Releases Wiki Activity

bitbake: user-manual-fetching.xml: Re-write of "File Download Support" chapter.

Browse Source 
Basic re-write to clean up text and flow.

(Bitbake rev: 0f82dc4c22ce015bef87fd5aae0b6fa3e429016e)

Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Scott Rifenbark
2014-02-11 20:00:02 -06:00
committed by Richard Purdie
parent b4b3bf56f6
commit f3caa2b27e
1 changed files with 270 additions and 160 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bitbake/doc/user-manual/user-manual-fetching.xml
+270 -160
View File
@@ -2,84 +2,110 @@
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"> "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
<chapter> <chapter>
<title>File download support</title> <title>File Download Support</title>
<para>
BitBake's <filename>fetch</filename> and
<filename>fetch2</filename> modules support downloading
files.
This chapter provides an overview of the fetching process
and also presents sections on each of the fetchers BitBake
supports.
<note>
The original <filename>fetch</filename> code, for all
practical purposes, has been replaced by
<filename>fetch2</filename> code.
Consequently, the information in this chapter does not
apply to <filename>fetch</filename>.
</note>
</para>
<section id='file-download-overview'> <section id='file-download-overview'>
<title>Overview</title> <title>Overview</title>
<para> <para>
BitBake provides support to download files. When BitBake starts to execute, the very first thing
This procedure is called fetching and is handled by the it does is to fetch the source files needed.
fetch and fetch2 modules. This section overviews the process.
At this point, the original fetch code is considered to For some additional information on fetching source files, see the
be replaced by fetch2 and this manual is only related "<link linkend='source-fetching-dev-environment'>Source Fetching</link>"
to the fetch2 codebase. section.
</para> </para>
<para> <para>
The <filename>SRC_URI</filename> is normally used to When BitBake goes looking for source files, it follows a search
tell BitBake which files to fetch. order:
The next sections describe the available fetchers and <orderedlist>
their options. <listitem><para><emphasis>Pre-mirror Sites:</emphasis>
Each fetcher honors a set of variables and per BitBake first uses pre-mirrors to try and find source
URI parameters separated by a “;” consisting of a key and files.
a value. These locations are defined using the
The semantics of the variables and parameters are <link linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
defined by the fetcher. variable.
BitBake tries to have consistent semantics between the </para></listitem>
different fetchers. <listitem><para><emphasis>Source URI:</emphasis>
If pre-mirrors fail, BitBake uses
<link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>.
</para></listitem>
<listitem><para><emphasis>Mirror Sites:</emphasis>
If fetch failures occur using <filename>SRC_URI</filename>,
BitBake next uses mirror location as defined by the
<link linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
variable.
</para></listitem>
</orderedlist>
</para> </para>
<para> <para>
The overall fetch process first attempts to fetch from
<filename>PREMIRRORS</filename>.
If these fail, the original <filename>SRC_URI</filename>
is attempted.
If that fails, BitBake falls back to
<filename>MIRRORS</filename>.
Because cross-URLs are supported, it is possible to mirror Because cross-URLs are supported, it is possible to mirror
a Git repository on an HTTP server as a tarball. a Git repository on an HTTP server as a tarball.
Here are some examples that show commonly used mirror Here are some examples that show commonly used mirror
definitions: definitions:
<literallayout class='monospaced'> <literallayout class='monospaced'>
PREMIRRORS ?= "\ PREMIRRORS ?= "\
bzr://.*/.* http://somemirror.org/sources/ \n \ bzr://.*/.* http://somemirror.org/sources/ \n \
cvs://.*/.* http://somemirror.org/sources/ \n \ cvs://.*/.* http://somemirror.org/sources/ \n \
git://.*/.* http://somemirror.org/sources/ \n \ git://.*/.* http://somemirror.org/sources/ \n \
hg://.*/.* http://somemirror.org/sources/ \n \ hg://.*/.* http://somemirror.org/sources/ \n \
osc://.*/.* http://somemirror.org/sources/ \n \ osc://.*/.* http://somemirror.org/sources/ \n \
p4://.*/.* http://somemirror.org/sources/ \n \ p4://.*/.* http://somemirror.org/sources/ \n \
svk://.*/.* http://somemirror.org/sources/ \n \ svk://.*/.* http://somemirror.org/sources/ \n \
svn://.*/.* http://somemirror.org/sources/ \n" svn://.*/.* http://somemirror.org/sources/ \n"
MIRRORS =+ "\ MIRRORS =+ "\
ftp://.*/.* http://somemirror.org/sources/ \n \ ftp://.*/.* http://somemirror.org/sources/ \n \
http://.*/.* http://somemirror.org/sources/ \n \ http://.*/.* http://somemirror.org/sources/ \n \
https://.*/.* http://somemirror.org/sources/ \n" https://.*/.* http://somemirror.org/sources/ \n"
</literallayout> </literallayout>
</para> </para>
<para> <para>
Non-local downloaded output is placed Any source files that are not local (i.e. downloaded from
into the directory specified by the the Internet) are placed into the download directory,
<filename>DL_DIR</filename> variable. which is specified by
For non local archive downloads, the code can verify <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>.
sha256 and md5 checksums for the download to ensure </para>
the file has been downloaded correctly.
These can be specified in the following forms <para>
for md5 and sha256 checksums, respectively: For non-local archive downloads, the fetcher code can verify
sha256 and md5 checksums to ensure
the archives have been downloaded correctly.
You can specify these checksums by using the
<filename>SRC_URI</filename> variable with the appropriate
varflags as follows:
<literallayout class='monospaced'> <literallayout class='monospaced'>
SRC_URI[md5sum] SRC_URI[md5sum]
SRC_URI[sha256sum] SRC_URI[sha256sum]
</literallayout> </literallayout>
You can also specify them as parameters on the You can also specify the checksums as parameters on the
<filename>SRC_URI</filename>: <filename>SRC_URI</filename> as shown below:
<literallayout class='monospaced'> <literallayout class='monospaced'>
SRC_URI="http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d" SRC_URI="http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d"
</literallayout> </literallayout>
If <filename>BB_STRICT_CHECKSUM</filename> is set, any download If
without a checksum will trigger an error message. <link linkend='var-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
In cases where multiple files are listed in is set, any download without a checksum triggers an error message.
In cases where multiple files are listed using
<filename>SRC_URI</filename>, the name parameter is used <filename>SRC_URI</filename>, the name parameter is used
assign names to the URLs and these are then specified assign names to the URLs and these are then specified
in the checksums using the following form: in the checksums using the following form:
@@ -89,142 +115,226 @@ MIRRORS =+ "\
</para> </para>
</section> </section>
<section id='local-file-fetcher'> <section id='bb-fetchers'>
<title>Local file fetcher</title> <title>Fetchers</title>
<para> <para>
The URN for the local file fetcher is file. As mentioned in the previous section, the
The filename can be either absolute or relative. <filename>SRC_URI</filename> is normally used to
If the filename is relative, tell BitBake which files to fetch.
<filename>FILESPATH</filename> and failing that And, the fetcher BitBake uses depends on the how
<filename>FILESDIR</filename> will be used to find the <filename>SRC_URI</filename> is set.
appropriate relative file. </para>
The metadata usually extend these variables to include
variations of the values in <filename>OVERRIDES</filename>. <para>
Single files and complete directories can be specified. These next few sections describe the available fetchers and
<literallayout class='monospaced'> their options.
Each fetcher honors a set of variables URI parameters,
which are separated by semi-colon characters and consist
of a key and a value.
The semantics of the variables and parameters are
defined by the fetcher.
BitBake tries to have consistent semantics between the
different fetchers.
</para>
<section id='local-file-fetcher'>
<title>Local file fetcher</title>
<para>
The URN for the local file fetcher is file.
</para>
<para>
The filename can be either absolute or relative.
If the filename is relative,
<link linkend='var-FILESPATH'><filename>FILESPATH</filename></link>
is used.
Failing that,
<link linkend='var-FILESDIR'><filename>FILESDIR</filename></link>
is used to find the appropriate relative file.
</para>
<para>
The metadata usually extend these variables to include
variations of the values in
<link linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>.
Single files and complete directories can be specified.
<literallayout class='monospaced'>
SRC_URI = "file://relativefile.patch" SRC_URI = "file://relativefile.patch"
SRC_URI = "file://relativefile.patch;this=ignored" SRC_URI = "file://relativefile.patch;this=ignored"
SRC_URI = "file:///Users/ich/very_important_software" SRC_URI = "file:///Users/ich/very_important_software"
</literallayout> </literallayout>
</para> </para>
</section> </section>
<section id='cvs-fetcher'> <section id='cvs-fetcher'>
<title>CVS fetcher</title> <title>CVS fetcher</title>
<para> <para>
The URN for the CVS fetcher is cvs. The URN for the CVS fetcher is cvs.
This fetcher honors the variables <filename>CVSDIR</filename>, </para>
<filename>SRCDATE</filename>, <filename>FETCHCOMMAND_cvs</filename>,
<filename>UPDATECOMMAND_cvs</filename>.
<filename>DL_DIR</filename> specifies where a
temporary checkout is saved.
<filename>SRCDATE</filename> specifies which date to
use when doing the fetching (the special value of "now"
will cause the checkout to be updated on every build).
<filename>FETCHCOMMAND</filename> and
<filename>UPDATECOMMAND</filename> specify which executables
to use for the CVS checkout or update.
</para>
<para> <para>
The supported parameters are module, tag, date, This fetcher honors the variables <filename>CVSDIR</filename>,
method, localdir, rshand scmdata. <filename>SRCDATE</filename>, <filename>FETCHCOMMAND_cvs</filename>,
The module specifies which module to check out, <filename>UPDATECOMMAND_cvs</filename>.
the tag describes which CVS TAG should be used for The
the checkout. <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
By default, the TAG is empty. variable specifies where a
A date can be specified to override the temporary checkout is saved.
<filename>SRCDATE</filename> of the The
configuration to checkout a specific date. <link linkend='var-SRCDATE'><filename>SRCDATE</filename></link>
The special value of "now" will cause the checkout to be variable specifies which date to
updated on every build. use when doing the fetching.
method is by default pserver. The special value of "now" causes the checkout to be
If ext is used the rsh parameter will be evaluated updated on every build.
and <filename>CVS_RSH</filename> will be set. The <filename>FETCHCOMMAND</filename> and
Finally, localdir is used to checkout into a special <filename>UPDATECOMMAND</filename> variables specify the executables
directory relative to <filename>CVSDIR</filename>. to use for the CVS checkout or update.
<literallayout class='monospaced'> </para>
<para>
The supported parameters are as follows:
<itemizedlist>
<listitem><para><emphasis>"module":</emphasis>
Specifies the module to check out.
</para></listitem>
<listitem><para><emphasis>"tag":</emphasis>
Describes which CVS TAG should be used for
the checkout.
By default, the TAG is empty.
</para></listitem>
<listitem><para><emphasis>"date":</emphasis>
Specifies a date.
If no "date" is specified, the
<link linkend='var-SRCDATE'><filename>SRCDATE</filename></link>
of the configuration is used to checkout a specific date.
The special value of "now" causes the checkout to be
updated on every build.
</para></listitem>
<listitem><para><emphasis>"method":</emphasis>
By default <filename>pserver</filename>.
If "method" is set to "ext", BitBake examines the "rsh"
parameter and sets <filename>CVS_RSH</filename>.
</para></listitem>
<listitem><para><emphasis>"localdir":</emphasis>
Used to checkout force into a special
directory relative to <filename>CVSDIR</filename>.
</para></listitem>
<listitem><para><emphasis>"rsh"</emphasis>
Used in conjunction with the "method" parameter.
</para></listitem>
<listitem><para><emphasis>"scmdata":</emphasis>
I need a description for this.
</para></listitem>
</itemizedlist>
Following are two examples using cvs:
<literallayout class='monospaced'>
SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext" SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext"
SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat" SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat"
</literallayout> </literallayout>
</para> </para>
</section> </section>
<section id='http-ftp-fetcher'> <section id='http-ftp-fetcher'>
<title>HTTP/FTP fetcher</title> <title>HTTP/FTP fetcher</title>
<para> <para>
The URNs for the HTTP/FTP fetcher are http, https, and ftp. The URNs for the HTTP/FTP fetcher are http, https, and ftp.
This fetcher honors the variables </para>
<filename>FETCHCOMMAND_wget</filename>.
<filename>FETCHCOMMAND</filename> contains the command used <para>
for fetching. This fetcher honors the variables
“${URI}” and “${FILES}” will be replaced by the URI and <filename>FETCHCOMMAND_wget</filename>.
basename of the file to be fetched. The <filename>FETCHCOMMAND</filename> variable
<literallayout class='monospaced'> contains the command used for fetching.
“${URI}” and “${FILES}” are replaced by the URI and
the base name of the file to be fetched.
<literallayout class='monospaced'>
SRC_URI = "http://oe.handhelds.org/not_there.aac" SRC_URI = "http://oe.handhelds.org/not_there.aac"
SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac" SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac"
SRC_URI = "ftp://you@oe.handheld.sorg/home/you/secret.plan" SRC_URI = "ftp://you@oe.handheld.sorg/home/you/secret.plan"
</literallayout> </literallayout>
</para> </para>
</section> </section>
<section id='svn-fetcher'> <section id='svn-fetcher'>
<title>SVN Fetcher</title> <title>SVN Fetcher</title>
<para> <para>
The URN for the SVN fetcher is svn. The URN for the SVN fetcher is svn.
</para> </para>
<para> <para>
This fetcher honors the variables This fetcher honors the variables
<filename>FETCHCOMMAND_svn</filename>, <filename>FETCHCOMMAND_svn</filename>,
<filename>SVNDIR</filename>, <filename>SVNDIR</filename>,
and <filename>SRCREV</filename>. and
<filename>FETCHCOMMAND</filename> contains the <link linkend='var-SRCREV'><filename>SRCREV</filename></link>.
subversion command. The <filename>FETCHCOMMAND</filename> variable contains the
<filename>SRCREV</filename> specifies which revision <filename>subversion</filename> command.
to use when doing the fetching. The <filename>SRCREV</filename> variable specifies which revision
</para> to use when doing the fetching.
</para>
<para> <para>
The supported parameters are proto, rev and scmdata. The supported parameters are as follows:
proto is the Subversion protocol, rev is the <itemizedlist>
Subversion revision. <listitem><para><emphasis>"proto":</emphasis>
If scmdata is set to “keep”, the “.svn” directories will The Subversion protocol.
be available during compile-time. </para></listitem>
<literallayout class='monospaced'> <listitem><para><emphasis>"rev":</emphasis>
The Subversion revision.
</para></listitem>
<listitem><para><emphasis>"scmdata":</emphasis>
Set to "keep" causes the “.svn” directories
to be available during compile-time.
</para></listitem>
</itemizedlist>
Following are two examples using svn:
<literallayout class='monospaced'>
SRC_URI = "svn://svn.oe.handhelds.org/svn;module=vip;proto=http;rev=667" SRC_URI = "svn://svn.oe.handhelds.org/svn;module=vip;proto=http;rev=667"
SRC_URI = "svn://svn.oe.handhelds.org/svn/;module=opie;proto=svn+ssh;date=20060126" SRC_URI = "svn://svn.oe.handhelds.org/svn/;module=opie;proto=svn+ssh;date=20060126"
</literallayout> </literallayout>
</para> </para>
</section> </section>
<section id='git-fetcher'> <section id='git-fetcher'>
<title>GIT Fetcher</title> <title>GIT Fetcher</title>
<para> <para>
The URN for the GIT Fetcher is git. The URN for the Git Fetcher is git.
</para> </para>
<para> <para>
The variable <filename>GITDIR</filename> will be used as the The variable <filename>GITDIR</filename> is used as the
base directory where the Git tree is cloned to. base directory in which the Git tree is cloned.
</para> </para>
<para> <para>
The parameters are tag, protocol, and scmdata. The supported parameters are as follows:
The tag parameter is a Git tag, the default is “master”. <itemizedlist>
The protocol tag is the Git protocol to use and defaults to “git” <listitem><para><emphasis>"tag":</emphasis>
if a hostname is set, otherwise it is “file”. The Git tag.
If scmdata is set to “keep”, the “.git” directory will be available The default is "master".
during compile-time. </para></listitem>
<literallayout class='monospaced'> <listitem><para><emphasis>"protocol":</emphasis>
The Git protocol.
The default is "git" when a hostname is set.
If a hostname is not set, the Git protocol is "file".
</para></listitem>
<listitem><para><emphasis>"scmdata":</emphasis>
When set to “keep”, the “.git” directory is available
during compile-time.
</para></listitem>
</itemizedlist>
Following are two examples using git:
<literallayout class='monospaced'>
SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1" SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http" SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http"
</literallayout> </literallayout>
</para> </para>
</section>
</section> </section>
</chapter> </chapter>