Commit Graph
100 Commits
Author SHA1 Message Date
Paul Eggleton 720c926271 recipetool: create: fix falling back to declared license for npm packages
Fix two problems falling back to the "license" field from package.json
when no license file is present:
1) The function that was supposed to return the license field value was
   always explicitly returning None, and this was never noticed (because
   the test cases never exercised the fallback as they provided license
   files for each module).
2) Fix the main package not falling back because it had a default of an
   empty list, which evaluates to '' instead of 'Unknown'.

(From OE-Core rev: 59381a9450949ce6b4b03adb717e950b999830f3)

(From OE-Core rev: 2d96460f2dcac4263f43ebcb7556722ce55c9918)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-29 19:35:56 +01:00
Paul Eggleton e89f4e531f recipetool: create: fix picking up false npm package directories
It is possible for a Node.js module to have node_modules subdirectories
that contain no package.json file (e.g. iotivity-node has such a
directory). It appears these should simply be ignored, or else with the
way the current code works we will get errors later.

(From OE-Core rev: 8c522f1f536270e195c8c73f5c72801495e7b33b)

(From OE-Core rev: 8da9185a1c68c8274269841d0867d7d4abf426f0)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-29 19:35:56 +01:00
Paul Eggleton 519600c607 devtool: sdk-update: fix handling of UNINATIVE_CHECKSUM changes
If UNINATIVE_CHECKSUM changes over an SDK update, bitbake within the
extensible SDK will be broken because it will see that the matching
uninative tarball doesn't exist and if there is a default value of
UNINATIVE_URL it will attempt to download the file and will then fail
because the checksums don't match up; alternatively if no UNINATIVE_URL
is set then it'll also fail with an error about misconfiguration. To fix
this, add some logic to devtool sdk-update to download the matching
uninative tarball(s) for the checksum(s) in the newly fetched SDK
configuration.

Fixes [YOCTO #9301].

(From OE-Core rev: 14ff58ad98a5afac08db77068d80f152d8875766)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-19 21:11:24 +01:00
Paul Eggleton c219c6d582 buildtools-tarball: fix perl being included when building with ipk
Due to some logic within opkg, a package with the name matching a
dependency will always win over a package with that name in RPROVIDES -
even if there is an RCONFLICTS (which is silently ignored), higher feed
priority and version. The end result is that buildtools gets perl
installed instead of the nativesdk-buildtools-perl-dummy package and
that perl (with missing dependencies) gets used in preference to the
host one, which is precisely what we were trying to avoid.

This is almost certainly a bug in opkg, especially as the other
package's dependencies aren't properly installed under these
circumstances either. However, specifying RREPLACES works around this,
and with no apparent side-effects is probably the safest solution for
now.

At the same time I noticed that in prepending to SDK_PACKAGE_ARCHS we
were actually ending up with a low priority for the dummy package feed
rather than a high one, so change to append it instead. This has no
effect on the packages that get installed at the moment, but should be
done in case the package manager behaviour changes to factor in the feed
priority in future.

Fixes [YOCTO #9469].

(From OE-Core rev: b18134ddaf949b4f001a1613ab876aed6324040a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-18 16:28:22 +01:00
Paul Eggleton 60a8719e6e devtool: upgrade: handle recipes where source is not first entry in SRC_URI
It is unusual but not impossible to find recipes whose first entry is
not the main source URL but instead some patch or other local file, for
example python-cryptography in meta-python (which sets SRC_URI before
inheriting pypi). There's nothing inherently wrong with this, and we
shouldn't assume that the first entry is the main source URL, so just
take the first non-local entry instead.

(From OE-Core rev: c868198c1f6006789707b497c2ae34d7cc5e706f)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-15 06:59:43 +01:00
Paul Eggleton 8353557193 devtool: update-recipe: handle where SRC_URI is appended to with +=
If a recipe sets SRC_URI and then appends more items to it with +=
(such as the current rpm recipe in OE-Core), the code in
patch_recipe_file() was failing with a traceback. Work around the
problem for now by dropping the existing lines if we understand the
operation, else just set the value outright at the end. This leaves
something to be desired as it either doesn't respect the existing
structure or leaves a mess but it's better than the current
breakage.

We'll need to come up with a better solution later. Part of the problem
is the existing code structure doesn't allow for patch_recipe_file() to
know what's being added or removed - it only knows the final value that
the caller wants set.

Fixes [YOCTO #9458].

(From OE-Core rev: da5cd84794e1ab5c4efcd6250d3b93a46f610744)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-15 06:59:43 +01:00
Paul Eggleton b13184cccc classes/buildhistory: fix grammar in comments
Fix a minor grammatical error in the comments here.

(From OE-Core rev: ddc60aea113e587f27df03645620d6eb677ff28a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-05 14:33:57 +01:00
Paul Eggleton e5c0a9f067 classes/buildhistory: fix filtering of depends-nokernel.dot
For images we produce a number of filtered dependency .dot files for
readability, the first of which is depends-nokernel.dot which filters
out just the kernel itself (not kernel modules). Unfortunately the
filter specifications hadn't been updated for the dash-to-underscore
removal or the 4.x kernel upgrade, thus the filtering wasn't actually
doing anything.

(From OE-Core rev: ec1077bed0c1538084bceeafb957abe725b13b76)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-05 14:33:57 +01:00
Paul Eggleton 4d364f27e7 classes/buildhistory: optimise getting package size list
Invoking oe-pkgdata-util in turn for every package in the list was slow
with a large image. Modify oe-pkgdata-util's read-value command to take
an option to read the list of packages from a file, as well as prefix
the value with the package name; we can then use this to get all of the
package sizes at once. This reduces the time to gather this information
from minutes to just a second or two.

Fixes [YOCTO #7339].

(From OE-Core rev: 51c24904cc1bc823bccc3f179b8d7a192dace168)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-05 14:33:57 +01:00
Paul Eggleton 4b7790915f devtool: modify: call shutdown on tinfoil when done
Strictly speaking we ought to explicitly shut down a tinfoil instance
when we're done with it. This doesn't affect modify's operation but is
important if you want to be able to call into modify() from another
plugin (though anyone doing so should be advised that the function is
by no means a stable API and is subject to change in future releases).

(From OE-Core rev: 626dbadf22b57a22a8f8b9d1957937120f4ba4d5)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-31 23:01:37 +01:00
Paul Eggleton 43da71258c toolchain-shar-extract.sh: ensure all_proxy is allowed through
all_proxy is referred to by oe-git-proxy so ensure it is allowed through
into the installer environment in case the extensible SDK install
process needs to query a remote git repository.

(From OE-Core rev: 65efec4cca9926b217b07c429a130088a9b8e2f6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-31 23:01:37 +01:00
Paul Eggleton 2aec71e028 oe-publish-sdk: exclude sstate-cache if publishing minimal SDK
If SDK_EXT_TYPE is set to "minimal" then the SDK won't contain many
sstate artifacts, and you're required to set up an sstate mirror in this
case anyway so there's no point publishing the "stub" sstate-cache
directory from within the SDK since it won't be useful for update
purposes and may be confused with the real sstate-cache.

There is however a possibility that people might publish the real
sstate-cache directory under the same output directory provided to
oe-publish-sdk, thus deleting it after extracting (as we were doing with
other files we wanted to clean up at the end) would be problematic,
besides which extracting it and then deleting it is wasteful. Thus,
introduce a "-p" command line option to the SDK installer that we can
use to tell tar not to extract the items we don't want when publishing.
This has the added benefit of mostly keeping references to these in the
place they belong i.e. in populate_sdk_ext.bbclass.

(From OE-Core rev: 774b85d42db1d81936d4e4af4f6fb2c57cb51d2c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-31 23:01:37 +01:00
Paul Eggleton 8ef70163fc oe-publish-sdk: prevent specifying a directory for the SDK argument
The SDK argument is expected to be an installer .sh file; if a directory
is specified we can get an ugly failure later on; best to check up
front.

Fixes [YOCTO #9065].

(From OE-Core rev: ce71f5c2fb8a7b473988da30bbb9bec95e8a6f5e)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-31 23:01:37 +01:00
Paul Eggleton 591b97c6be classes/populate_sdk_ext: support setting vars from environment at build time
When running bitbake you may pass in values of variables from the
external environment (making use of BB_ENV_EXTRAWHITE), and you may
choose to do this when building the extensible SDK, for example:

  MACHINE=qemuarm bitbake -c populate_sdk_ext core-image-minimal

You would naturally expect those settings to be reflected in the
extensible SDK itself; however they were not, since we were only
considering local.conf and auto.conf. Check the variables mentioned in
BB_ENV_EXTRAWHITE to see if any are different than the values set in
local.conf/auto.conf and add lines setting them in the SDK's local.conf
if so.

Fixes [YOCTO #9339].

(From OE-Core rev: 2bfed75c48a6f6596ded9cb64cb96f00510f914e)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-31 23:01:36 +01:00
Paul Eggleton 39170fe6f1 classes/sanity: use proper multi-line string literals
A python string literal isn't meant to be split over multiple lines
unless it is explicitly multi-line. Use three quotes to indicate that.

(From OE-Core rev: 8cdb505bb3d9748be1744f36cbbfb4f62fa20b67)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-28 15:55:48 +01:00
Paul Eggleton 2b992f3e56 site.conf.sample: fix reference to oe-git-proxy script
The name of the script is oe-git-proxy, not oe-git-proxy.sh.

(From meta-yocto rev: c10150d94e8830c82937f6b229672c194ef270bf)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-26 07:34:59 +00:00
Paul Eggleton 13a31b15ec oe-publish-sdk: fix remote publishing
A typo in the command prevented the last part of the publish from
succeeding.

(From OE-Core rev: 5d51bd1f02fb20e02d0de0ac9874d7f9a6f5ddc6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:14 +00:00
Paul Eggleton 9926425bbd oe-publish-sdk: improve help output slightly
This was just copy-and-pasted from devtool - set a reasonable
description and tweak the sdk parameter help text.

Also add a copyright statement and drop the opening comment describing
the command-line syntax that duplicated the help output (with at least
one mistake in it).

(From OE-Core rev: b18f9cb62e12d068fb840fd7ab9f35b4f039e649)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:13 +00:00
Paul Eggleton 905286ce46 oe-publish-sdk: drop SDK installer file from published output
We don't really need the SDK installer in the published output, for two
reasons:

1) The directory produced is for the consumption of devtool sdk-update,
   and the installer isn't used by that at all
2) It wouldn't really make sense to point users at the update directory
   to download the SDK installer because it contains a bunch of things
   that aren't meant for manual download, so it wouldn't be very tidy.

Leaving the file present can mislead you into thinking the opposite of
both of the above.

(From OE-Core rev: a9ec72b7897ffc0b901c1ccbcbe3cabbc7ac41ee)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:13 +00:00
Paul Eggleton 0523378082 devtool: add: create git repository if URL specified as positional argument
When we add from a fetched URL we are supposed to turn the resulting
source tree into a git repository (if it isn't already one). However, we
were using the older deprecated option name here instead of the
positional argument, so "devtool add -f <url>" resulted in the repo
being created but "devtool add <url>" didn't, which was wrong.

Also update the oe-selftest tests to check that this worked.

(From OE-Core rev: a7b6b1f8cc1c096724f794ac9dee312b0f771f66)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:13 +00:00
Paul Eggleton 11c1d30152 devtool: add: delete externalsrc files on npm recipe do_install
The npm class just installs whatever is in ${S}; if you're using
externalsrc in conjunction with it the symlinks (oe-workdir and oe-logs
by default) plus singletask.lock will end up in the final package, which
isn't really right. Introduce a variable so we know the path the files
will be installed into within npm.bbclass, and append to do_install
within the workspace bbappend to delete the files.

(From OE-Core rev: 766845e06db9d7d595e836ea1364c16fa132a413)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:13 +00:00
Paul Eggleton 552a68ad1c devtool: configure-help: fix error if do_configure not already run
The code here for running do_configure if it hadn't already been run was
using the wrong string substitution parameters; fix it and test it.

(From OE-Core rev: b2677a4448dbc42e523c731b953b44006749252c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:13 +00:00
Paul Eggleton eab3f06e39 bitbake.conf: whitelist proxy variables in config hash
We don't need these variables incorporated in the config hash (i.e. we
don't need to reparse if they change).

This fixes an issue with the extensible SDK where the cache generated
at installation time not to be used afterwards due to the way we're
allowing through the proxy variables into the install environment (such
that they get set to "" if not set externally).

(From OE-Core rev: 3dc7c58b7b3224f3ed2d59d95313e70e4c57d7c4)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:13 +00:00
Paul Eggleton 58d2e5625f classes/populate_sdk_ext: parse metadata on minimal SDK install
Instead of skipping the build system preparation step within the
extensible SDK install process when SDK_EXT_TYPE is "minimal", run
bitbake -p so that the cache is populated ready for the first time
devtool is run.

(From OE-Core rev: 6b38a991a3475fb82889428b94563968c7570473)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:13 +00:00
Paul Eggleton 06845723d4 devtool: sdk-install: add option to allow building from source
By default the sdk-install subcommand expects to restore the requested
items from sstate and fails if it can't. If the user is OK with building
from source, add a -s/--allow-build option to allow them to do that. In
the process, ensure we show the status output while we're installing.

Also add the missing header to the top of the file.

(From OE-Core rev: a86b426cdd465ec5cb08bb5fa7729e4e673d94bb)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:13 +00:00
Paul Eggleton 50addfb97b classes/distutils*: don't hide logs when setup script fails
We need to see the output of the setup script in the error message
because what we are having bbfatal print here doesn't completely
describe what went wrong, thus we should use bbfatal_log here and not
bbfatal.

(From OE-Core rev: 970a02061b474a238f57cd0cc4db7b62e36dfa3d)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:13 +00:00
Paul Eggleton 0ec30c7a68 classes/packagegroup: drop complementary -ptest if ptest not in DISTRO_FEATURES
If ptest isn't in DISTRO_FEATURES, there won't be any -ptest packages
and thus no need for the -ptest package for the packagegroup either.

(From OE-Core rev: 52f9b12601c02f1585438f890f847b4596170362)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:13 +00:00
Paul Eggleton d96ea299b3 classes/packagegroup: fix dbg/dev/ptest complementary packages
OE-Core commit 8981ed20234c42dc1f5cdef802ebe7214bd55238 (from 2013)
caused a regression where -dbg/-dev/-ptest packages weren't generated
for packagegroups because we weren't setting ALLOW_EMPTY for them.
Break out the code for adding complementary groups to its own
conditional so we can set ALLOW_EMPTY at the same time.

(From OE-Core rev: 340e9d619c881f320070474d5c4674dcaeabad35)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:13 +00:00
Paul Eggleton 9cc65ed020 Revert "selftest: Added MACHINE = "qemux86" to tests that use runqemu"
This is not the correct way to handle this - it significantly increases
the time taken to run oe-selftest anywhere MACHINE is set to some other
value (for example "qemux86-64"), because all of the artifacts for
qemux86 need to then be built as well when running the test. If we need
to skip these tests on non-QEMU machines, the devtool test already
demonstrates how to do that.

This reverts commit 169e1eaa4fc5ed03e2307b68686a7f5b1db37a36.

(From OE-Core rev: f60da6d5be5469ed5b834759c3822471f81fcdd2)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-20 23:12:26 +00:00
Paul Eggleton 600b7007f6 recipetool: create: don't create extra files directory unconditionally
The extra directory next to the recipe should only be created if there
are files to put into it; currently only the npm plugin does this. I
didn't notice the issue earlier because the test was actually able to
succeed under these circumstances if the recipe file came first in the
directory listing, which was a fault in my original oe-selftest test;
apparently on some YP autobuilder machines the order came out reversed.

With this change we can put the oe-selftest test that highlighted the
issue back to the way it was, with an extra check to reinforce that only
a single file should be created.

(From OE-Core rev: b8b778345eb0997c2cd952a1f61fdd2050b6b894)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-20 23:12:25 +00:00
Paul Eggleton 0eb52b91a7 classes/buildhistory: record a few more variables for extensible SDK
We've recently added SDK_EXT_TYPE and SDK_RECRDEP_TASKS and these will
have an effect on the size and functionality of the SDK, so record them
in sdk-info.txt.

(From OE-Core rev: 1be0ae80f97bdee758232ec5397c09d73d57e1e0)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-12 22:11:48 +00:00
Paul Eggleton c50bdb300b bitbake: fetch2/npm: add missing URL argument to ParameterError
Without this you get a rather odd traceback instead of the proper
exception message.

(Bitbake rev: 2fe1826d3077eeda6cde433d3a1e6620f74e08dd)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:45:16 +00:00
Paul Eggleton fbf27c4c5d bitbake: fetch2/npm: properly handle npm dependencies
The output of "npm view dependencies" isn't entirely JSON if there are
multiple results, but the code here was just discarding the output if
the entire thing didn't parse as JSON. Split the output into lines and
iterate over it, parsing JSON fragments as we find them; this way we end
up with the last package's dependencies since it'll be last in the
output.

Digging further, it seems that the dependencies field reported by "npm
view" also includes optional dependencies. That wouldn't be a problem
except some of these optional dependencies may be OS-specific; for
example the "chokidar" module has "fsevents" in its optional
dependencies, but fsevents only works on MacOS X (and is only needed
there). If we erroneously pull in fsevents, not only is it unnecessary
but it causes "npm shrinkwrap" to throw a tantrum. In the absence of a
better approach, look at the os field and discard the module (along with
any of its dependencies) if it isn't for Linux.

As part of this, we can reduce the calls to npm view to one per package
since we get the entire json output rather than querying twice for two
separate fields. Overall the time taken has probably increased since we
are being more thorough about dependencies, but it's not quite as bad as
it could have been.

(Bitbake rev: 436d67fe7af89ecfbd11749a6ae1bc20e81f2cc8)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:45:16 +00:00
Paul Eggleton ef6a4518e4 bitbake: fetch2/npm: fix errors with some version specifications
"2 || 3" is a valid version specification for a dependency in an npm
package.json file, but of course that looks like something else when
sent to a shell. Quote the version value to avoid this.

(Bitbake rev: bea0246831a46d943d2e27d6b38f6e498bd3413c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:45:16 +00:00
Paul Eggleton bc0e99d2b1 recipetool: create: shrinkwrap and lockdown npm modules
"npm shrinkwrap" creates a file that ensures that the exact same
versions get fetched the next time the recipe is built. lockdown is
similar but also includes sha1sums of the modules thus validating they
haven't changed between builds. These ensure that the build is
reproducible.

Fixes [YOCTO #9225].

(From OE-Core rev: 277377f13b2b771915eb853e336ca24b84523ed1)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 17:00:29 +00:00
Paul Eggleton 309b2e6c57 recipetool: create: support creation of additional files by plugins
Allow plugins to create additional files to go alongside the recipe. The
plugins don't know what the output filename is going to be, so they need
to put the files in a temporary location and add them to an "extrafiles"
dict within extravalues where the destination filename is the key and
the temporary path is the value.

devtool add was also extended to ensure these files get moved in and
preserved upon reset if they've been edited by the user.

(From OE-Core rev: 334b9451111b7e3efbb43b3a4eecebcab8ec6f0e)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 17:00:29 +00:00
Paul Eggleton 2279eb2a4c recipetool: create: check if npm available if npm:// URL specified
If the user specifies an npm:// URL then the fetcher needs npm to be
available to run, so check if it's available early rather than failing
later.

(From OE-Core rev: a08d12ad867c292f7474731a0fe5e51e712446d6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 17:00:29 +00:00
Paul Eggleton 91455005b6 recipetool: create: split npm module dependencies into packages
Rather than rolling all of an npm module's dependencies into the same
package, split them into one module per package, setting the SUMMARY and
PKGV values from the package.json file for each package. Additionally,
mark each package with the appropriate license using the license
scanning we already do, falling back to the license stated in the
package.json file for the module if unknown. All of this is mostly in
aid of ensuring all modules and their licenses now show up in the
manifests for the image.

Additionally we set the main LICENSE value more concretely once we've
calculated the per-package licenses, since we have more information at
that point.

(From OE-Core rev: 8226805f83d21e7c1d2ba21969f3e8ee4b137496)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 17:00:29 +00:00
Paul Eggleton d46827cfd3 recipetool: create: add license file crunching
Matching license texts directly to md5sums only goes so far. Some
licenses make the copyright statement an intrinsic part of the license
statement (e.g. MIT) which of course varies between projects. Also,
people often seem to take standard license texts such as GPLv2 and
reformat them cosmetically - re-wrapping lines at a different width or
changing quoting styles are seemingly popular examples. In order to
match license files to their actual licenses more effectively, "crunch"
out these elements before comparing to an md5sum. (The existing plain
md5sum matching has been left in since it's a shortcut, and our list of
crunched md5sums isn't a complete replacement for it.)

As always, this code isn't providing any guarantees (legal or otherwise)
that it will always get the license correct - as indicated by the
accompanying comments the LICENSE values it writes out to the recipe are
indicative and you should verify them yourself by looking at the
documentation supplied from upstream for the software being built if you
have any concerns.

(From OE-Core rev: 553bb4ea5d51be5179e7d8c019740cf61ece76ea)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 17:00:29 +00:00
Paul Eggleton 3fd244b94f recipetool: create: match *LICENSE* as a license file
For example, this picks up a file named MIT-LICENSE.txt.

(From OE-Core rev: 103b4d26b340cbdf70bf43906e293f3497671fdc)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 17:00:29 +00:00
Paul Eggleton 2b6a35212d recipetool: create: improve mapping for autotools program macros
Make the following improvements to mapping items specified in
AC_CHECK_PROG, AC_PATH_PROG and AX_WITH_PROG to recipes/classes:

* Produce a map of native recipe -> binary for all binaries currently in
  STAGING_BINDIR_NATIVE and use this when mapping items
* Add some more entries to the class map
* Ignore autotools binaries since they are covered by the inherit of
  autotools
* Ignore coreutils-native since that would almost always be a bogus
  dependency

(From OE-Core rev: 5614c5ae6a004d4367eccc34dd3cc7ee61fb7e57)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 17:00:29 +00:00
Paul Eggleton 1607fac521 recipetool: create: be more tolerant of spacing in configure.ac
Allow for whitespace in appropriate places, and ensure we match all
whitespace chars not just the space character.

(This fixes extracting dependencies from tmux's configure.ac, for
example.)

(From OE-Core rev: 63524ac8093b734aa4f29f4ea47bcc036f748314)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 17:00:29 +00:00
Paul Eggleton 2d0933ce31 conf/distro/include: drop old recipes
These recipes have been removed (some a very long time ago).

(From meta-yocto rev: aaac6caf880c977c1385515580743f0e01800f73)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 16:58:14 +00:00
Paul Eggleton 1fd183e0b8 bblayers.conf.sample: remove BBLAYERS_NON_REMOVABLE
Hob was the only thing paying attention to this, and now Hob itself has
been removed we can remove this as well.

(From meta-yocto rev: 31f937ebfe916aadcd0beb5f8088d88feadc2a98)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 16:58:14 +00:00
Paul Eggleton a54cebe929 bitbake: fetch2/npm: ignore unknown headers in tarballs
Tarballs that are fetched down via npm repositories seem to often have
unknown headers. This doesn't affect our ability to extract the contents
though so we don't really care to see those warnings.

(Bitbake rev: b38975103e52a0c25e9ad9032c8cca1c47cbdcc2)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:03 +00:00
Paul Eggleton 0cd1be1f09 bitbake: fetch2/npm: handle alternative dependency syntax
npm allows you to specify a dependency as a Github username+path, or
omit the version entirely. You can hit these if you don't use a
shrinkwrap file, with the result that the code later fails due to the
output of "npm view" being empty; so handle this lazily by just ignoring
this part of the dependency if it's not really a version.

(Bitbake rev: 7b7a65c44dbdd5ba9366d4e2093f76df8758d546)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:03 +00:00
Paul Eggleton d9999279d9 bitbake: fetch2/npm: fix indentation
No code changes, just fix to use four spaces.

(Bitbake rev: 66a9ee7d54ca9c25209f72da079f260ccdcc872a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:02 +00:00
Paul Eggleton e43fcdf1e8 scripts/hob: drop
The Hob UI is gone from BitBake, so we should remove this shortcut
script.

(From OE-Core rev: 414f0d1ef07bb782ffa15ffd9db648ee2b2d1448)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:17:43 +00:00
Paul Eggleton 59b4cefe29 classes/packageinfo: remove
This class was only used by Hob, and since Hob has now been removed we
can drop it as well.

(From OE-Core rev: b8db070926a7ec294816bc6648eb12db7f126f26)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:17:43 +00:00
Paul Eggleton bbf2a5d35f conf/documentation.conf: remove BBLAYERS_NON_REMOVABLE
Hob was the only thing paying attention to this, and now Hob itself has
been removed we can remove this as well.

(From OE-Core rev: 270830fcfebf2fa1304a0dd15b218c24080b742e)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:17:43 +00:00
Paul Eggleton afb1d09ced recipetool: create: fix support for AX_CHECK_LIBRARY
Clearly I didn't test this part of the code - lists don't have an "add"
method. Needless to say I have tested it now.

(From OE-Core rev: 063ed9058a14775f77e7875d4f6ef5719fa03f18)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 00:11:38 +00:00
Paul Eggleton 25d9c4e02a devtool: add build-sdk subcommand
Add a build-sdk command which is only available within the extensible
SDK that builds a derivative extensible SDK. The idea is recipes in the
workspace become a part of the new SDK - for example, this allows taking
a vendor provided SDK, adding a few libs and then producing a new SDK
with those included.

When normally building the extensible SDK, the workspace is excluded;
here we need to copy into the new SDK (renaming it in the process); the
recipes' task signatures become locked and thus the sources are no
longer needed, so they are removed along with the workspace bbappends
which would interfere with the locked signatures. Additionally we need
to just copy the configuration files (i.e. local.conf and auto.conf)
rather than filtering and appending to them since that work has already
been done when constructing the original SDK. The extra sstate artifacts
from workspace recipes are also determined and copied into the new SDK
in minimal mode (on the assumption that you won't set up a new sstate
mirror).

This reuses some code from build-image, so that needed to be
generalised to allow that.

Implements [YOCTO #8892].

(From OE-Core rev: 59e207ff6dd4b50a8905e14bc9292cf2794f4e7a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 00:11:38 +00:00
Paul Eggleton 41eb36dc4c devtool: build-image: rename module
Hyphens aren't allowed in python identifiers, so you shouldn't use them
in module names or they are more difficult to import.

(From OE-Core rev: e8f6db9436dfc923e236f2cbc08f357e3b24fd5d)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 00:11:37 +00:00
Paul Eggleton 5c5c13d777 recipetool: create: add basic support for new npm fetcher/class
Add detection for npm modules and support for extracting the name and
version from package.json as is usually part of an npm module contents.

Note: this will likely only produce a buildable recipe if you use an
npm:// URL; simply pointing to a node.js source repository isn't going
to fetch the module's dependencies. It also doesn't set up the
shrinkwrap/lockdown automatically, so there is some room for improvement
later.

Implements [YOCTO #8690].

(From OE-Core rev: 41d0e4d75f13b53a6c1b6a8df9be4742be7534e0)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 23:08:51 +00:00
Paul Eggleton 2be37a9373 recipetool: create: add basic support for generating linux kernel recipes
Add support for detecting a Linux kernel source tree and generating a
basic kernel recipe using meta-skeleton's linux-yocto-custom recipe as a
base.

Implements [YOCTO #8981].

(From OE-Core rev: 39cab544b80ca4450106c9ede3180929ba24703c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 23:08:51 +00:00
Paul Eggleton 5cf15ffecc recipetool: create: add support for out-of-tree kernel modules
Detect kernel modules by looking for #include <linux/module.h>, and
handle the various styles of Makefile that appear to be used. I was able
to use this code to successfully build a number of external kernel
modules I found.

Implements [YOCTO #8982].

(From OE-Core rev: a85604f2eb2438b4caf0832c2ea15b5822f7e9a1)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 23:08:51 +00:00
Paul Eggleton 70153b4782 classes/externalsrc: fix symlinking if symlink exists pointing to another path
If the oe-workdir / oe-logs symlink exists and points to a different
path then the symlink needs to be removed before calling os.symlink() or
it'll fail.

(From OE-Core rev: cb8f064e48c24dcb1a15a32cef3399f35e549bdc)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:39:45 +00:00
Paul Eggleton 695cc45804 classes/populate_sdk_ext: prepend to PATH rather than appending
The rest of the environment setup script prepends to PATH, so when we
add the path to run devtool we should be prepending as well. This
also ensures that when you run the environment setup script from
extensible SDK installation A and then in the same shell session run the
environment setup script from installation B, and then run devtool, that
you're running B's devtool and not A's.

Fixes [YOCTO #9046].

(From OE-Core rev: b7ac987274f7aeaeacc217488cf4fe639de81b76)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:39:43 +00:00
Paul Eggleton b145480d44 classes/module: allow substitution of the modules_install target name
Quite a few external kernel modules I've found floating around don't
have a modules_install target, but they do have an install target that
basically differs only in name. To make it easier to build these just
make this a MODULES_INSTALL_TARGET variable that you can set from the
recipe - the alternative would be copy-and-paste the do_install
definition from this class which is potentially fragile.

(From OE-Core rev: effa6ce777540c5557e5cf904b48cc3369ee3f9f)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:39:43 +00:00
Paul Eggleton bf4d380b97 oe-selftest: devtool: add an additional test for devtool upgrade
Add a test for devtool upgrade with a recipe pointing to a git
repository, since this uses several different code paths.

(From OE-Core rev: d3ec74ff4db248ea11a568d5214708d6a1757012)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:33:01 +00:00
Paul Eggleton 4bae2f25b8 oe-selftest: devtool: rework devtool upgrade test
* Use a more real-world test of a recipe pointing to a remote file
* The cleanup tracking / teardown commands need to be added towards the
  top, or they won't have the desired effect of cleaning up if the test
  fails.
* Check that a versioned subdirectory gets renamed to match the new
  version
* Ensure the recipe contents gets changed as we expect it to
* Check that the recipe directory is deleted by devtool reset at the end

(From OE-Core rev: d0ae258e963f9bafffc4ca43c87497d27e57c127)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:33:01 +00:00
Paul Eggleton 10290f2245 devtool: upgrade: print new recipe name
Make this consistent with "devtool add" so that the user knows where to
find the new recipe.

(From OE-Core rev: 28fea4a104ba85aded2ecfe891e9bf997d72a103)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:33:01 +00:00
Paul Eggleton 5cd3be38f2 devtool: upgrade: drop PR on upgrade
The PR value should be reset to the default when upgrading, so we need
to drop it from the newly created file.

(From OE-Core rev: 4acf8a6912106dd3da5ceee9e30ccce86211d04b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:33:00 +00:00
Paul Eggleton e6f684b10c devtool: upgrade: eliminate unnecessary datastore copy
We aren't modifying the datastore copy here, so we don't need a copy at
all.

(From OE-Core rev: 9f05bf98d4dbcb991b684a2ab480c6638c6be292)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:33:00 +00:00
Paul Eggleton 860574edb2 devtool: upgrade: fix several issues with extraction of new source
Fix several issues when extracting the new version source over the top
of the old one (when the recipe is not fetching from a git repo):

* Delete the old source first so we ensure files deleted in the new
  version are deleted. This also has the side-effect of fixing any
  issues where files aren't marked writeable in the old source and thus
  overwriting them failed (harfbuzz 1.1.3 contains such files).
* Fix incorrect variable name in abspath statement that made it a no-op

(From OE-Core rev: 4f7595a514661f0e2418393286849a1cb6c46f0d)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:33:00 +00:00
Paul Eggleton 66a781c7e1 devtool: upgrade: fix constructing new branch from tarball releases
When we do an upgrade from one tarball version to another we want to:

1) Check out the old version as a new branch
2) Record the changes between the old and new versions as a commit
3) Check out the old version with patches applied
4) Rebase that onto the new branch

Where we went wrong was step #1 where instead we checked out the old
version with patches applied as the new branch, which meant the rebase
didn't do anything and any changes made by the patches to files still in
the new version were wiped out.

(From OE-Core rev: a4f3e6d5f4c549e36cce1e18801a565a3a3c9b0a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:33:00 +00:00
Paul Eggleton d30cc76039 devtool: upgrade: fix renaming of recipe if PV is not in name
If the actual value of PV isn't in the name of the recipe (for example,
a git or svn recipe) there's no point trying to rename it. Additionally,
we already have the original filename, there's no need to guess it -
just pass it in.

(From OE-Core rev: e144db3650b4a7c0a59066621905f34a5400303e)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:33:00 +00:00
Paul Eggleton 75eeeabe3c devtool: upgrade: fix moving version-specific files directory
We were trying to move this from the current directory instead of the
path. Let's just use shutil.move() instead of shelling out to mv.

(From OE-Core rev: 60454a0ba154d6c777e0c2b05b887b4e4fcde986)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:33:00 +00:00
Paul Eggleton 81ebb0b61e devtool: upgrade: fix version argument checking
For recipes that specify SRCREV, the code here wasn't quite doing the
right thing. If the recipe has a SRCREV then that needs changing on
upgrade, so ensure that the user specifies it. If it doesn't, then it'll
be "INVALID" not None since the former is the actual default, so handle
that properly as well. Additionally an unset variable was being
erroneously passed when raising the error about the version being the
same leading to a traceback, so fix that as well.

(From OE-Core rev: 1d0f821371d1cb93e30fad86f0c20e38cb93b54b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:33:00 +00:00
Paul Eggleton e953b5711f devtool: upgrade: drop superfluous call to validate_pn
The recipename argument to devtool upgrade specifies an existing recipe,
so by definition the name will be valid (or it won't exist) - we don't
need to validate it ourselves, that's only needed for situations like in
devtool add where we're creating a new recipe.

(From OE-Core rev: 1e2bc7d861555a04350a87f19047efdc717046be)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:33:00 +00:00
Paul Eggleton 492b1eb1d2 devtool: upgrade: make source tree path optional
Make devtool upgrade consistent with devtool add/modify in defaulting to
sources/<recipename> under the workspace if no source tree path is
specified.

(From OE-Core rev: 8a952407b192313515e91570632446b6dff01665)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:33:00 +00:00
Paul Eggleton 942ae25279 devtool: modify: fix source tree default name when mapping virtuals
If you for example ran devtool modify virtual/libusb0 without specifying
a source tree path, the default was <workspace>/sources/virtual/libusb0
which isn't correct - it should be using the mapped name i.e.
libusb-compat (in the default OE-Core configuration). Reorder some of
the code to ensure that the mapped name is used.

(From OE-Core rev: c51736df17da8e6e561dd5b7ce59cb08254da870)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:33:00 +00:00
Paul Eggleton e2334e126f devtool: add: tweak auto-determining name failure message
As suggested by Khem Raj.

(From OE-Core rev: 36cc6b81d0281348a0f241a80ddd427745a6a678)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:32:59 +00:00
Paul Eggleton 4883ccce1b classes/populate_sdk_ext: add a better config extension mechanism
The sdk_extraconf() method of setting the configuration was awkward
since you needed to set it in a class and then inherit that class since
function definitions aren't allowed in conf files. It seemed to me the
a neater way to do this was to read the extra lines from an additional
conf file sdk-extra.conf (which can be located in a conf/ directory
anywhere along BBPATH as with other configuration files).

(From OE-Core rev: b53edb86c65ad375df153017f245244ef97f3932)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-21 09:32:43 +00:00
Paul Eggleton 524ee088b3 recipetool: create: improve CMake package mapping
* Package names are actually case sensitive near as I can tell, so we
  shouldn't be lowercasing them everywhere.
* Look for CMake packages in pkgdata and map those back to recipes,
  so we aren't dependent on the hardcoded mappings (though those are
  still preserved).
* Avoid duplicates in the unmapped package list

(From OE-Core rev: 2ddad52ccca07245eea43d9b844c6c7d4b667ca3)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-21 09:32:43 +00:00
Paul Eggleton 7b6e5b025e recipetool: create: add additional extension mechanisms
Add a means of extending the dependency extraction for autotools and
cmake.

Note: in order to have this work, you need to have an __init__.py in the
lib/recipetool directory within your layer along with the module
implementing the handlers, and the __init__.py needs to contain:

  # Enable other layers to have modules in the same named directory
  from pkgutil import extend_path
  __path__ = extend_path(__path__, __name__)

(From OE-Core rev: 915dea9f89cd737e5ba167c384e8d314c5c23c49)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-21 09:32:43 +00:00
Paul Eggleton b2d44729e9 devtool: modify: tweak help description for behaviour change
I should have adjusted this in OE-Core commit
80a44e52609a89d9ffe816181ae193af491c06ac where the behaviour changed.

(From OE-Core rev: 13409a2b899bb74b8060c840b8c7ef8759d099cb)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-21 09:32:43 +00:00
Paul Eggleton a8e0e5ecd3 devtool: deploy-target: preserve existing files
If files would be overwritten by the deployment, preserve them in a
separate location on the target so that they can be restored if you
later run devtool undeploy-target.

At the same time, also check for sufficient space before starting the
operation so that we avoid potentially failing part way through.

Fixes [YOCTO #8978].

(From OE-Core rev: a2da55712691bcf1066c53d813107d75213d6b10)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-21 09:32:43 +00:00
Paul Eggleton 2059a344b6 devtool: undeploy-target: support undeploying all recipes
If you want to put the target device back to exactly how it was before
devtool deploy-target started poking things into it, then it would make
things easier if you didn't have to figure out which recipes were
deployed. Now that we have the list stored on the target we can
determine this reliably, so add a -a/--all option to undeploy-target to
undeploy everything that has been deployed.

One of the side-effects of this is that the dry-run functionality for
undeploy-target had to be reimplemented to actually run the script on
the target, since we have no way of knowing what's been deployed from
the host side. We don't need to do the same for deploy-target though
since we know exactly which files will be deployed without referring to
the target.

(From OE-Core rev: 41fed83060f0041e14e455d1446397bda277d953)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-21 09:32:42 +00:00
Paul Eggleton b95c72c4dc devtool: deploy-target: write deployed files list to target
When running devtool deploy-target, we save a list of deployed files,
and this list is used by devtool undeploy-target (or the next time
deploy-target is run if the list is present, in case any files have been
renamed or deleted since the first time). We were writing this file to
the host, but it makes more sense to write the list to the target
instead, so that if we for example swap in a different board, or switch
hosts, things will work as expected.

In order to do this properly we have to construct a shell script and
ship it over to the target so we can run it. The manifest is written out
to a hidden directory in the root (/.devtool).

Fixes [YOCTO #7908].

(From OE-Core rev: a16a0c9334b785e2df896266c8911a2c7a1806b8)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-21 09:32:42 +00:00
Paul Eggleton 62989efe6d devtool: sdk-update: tweak command-line handling of updateserver
Get the default value for updateserver from the configuration file and
show it in the help; also only make the parameter optional if it's
specified. This means we can also drop the check in the function as
argparse will then ensure it's specified if there's no config setting.

(From OE-Core rev: 82497bde58fc8bdae8d8acfbf025a1a90b14cc3e)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-21 09:32:42 +00:00
Paul Eggleton cada5a83c7 devtool: (un)deploy-target: add help descriptions
Add a long description used when running --help on the specific command.

(From OE-Core rev: eb7787d1652fd84a149fd394969f4f1099406051)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-21 09:32:42 +00:00
Paul Eggleton 6bd88e6171 scripts/lib/argparse_oe: tweak title above options
Naming these as "optional arguments" is perhaps slightly confusing since
some of the positional arguments might also be optional; in addition
it's rare (though possible) for options to be mandatory - up until
recently we had a recipetool option (-o) that was mandatory. It's not
perfect, but change it to "options" so it's at least a bit more
appropriate.

(From OE-Core rev: 55e675de6191bf7eccd26df29189f2a6faa40a20)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-21 09:32:42 +00:00
Paul Eggleton 32ef523898 devtool: categorise and order subcommands in help output
The listing of subcommands in the --help output for devtool was starting
to get difficult to follow, with commands appearing in no particular
order (due to some being in separate modules and the order of those
modules being parsed). Logically grouping the subcommands as well as
being able to exercise some control over the order of the subcommands
and groups would help, if we do so without losing the dynamic nature of
the list (i.e. that it comes from the plugins). Argparse provides no
built-in way to handle this and really, really makes it a pain to add,
but with some subclassing and hacking it's now possible, and can be
extended by any plugin as desired.

To put a subcommand into a group, all you need to do is specify a group=
parameter in the call to subparsers.add_parser(). you can also specify
an order= parameter to make the subcommand sort higher or lower in the
list (higher order numbers appear first, so use negative numbers to
force items to the end if that's what you want). To add a new group, use
subparsers.add_subparser_group(), supplying the name, description and
optionally an order number for the group itself (again, higher numbers
appear first).

(From OE-Core rev: e1b9d31e6ea3c254ecfe940fe795af44761e0e69)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-21 09:32:42 +00:00
Paul Eggleton 9f7df76eb4 devtool: update-recipe: don't show workspace recipe warning if no update
If we didn't make any changes to the file then there's no point warning
the user that we have done.

(From OE-Core rev: 391b9ba30d802ac420ddf382588e03e718861c01)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-21 09:32:42 +00:00
Paul Eggleton 51972edbb0 devtool: reset: fix preserving patches/other files next to recipes
If files had been created next to the recipe (for example devtool add,
edit the source and commit and then devtool update-recipe), running
devtool reset failed to preserve those files and gave an error due
to trying to rmdir the directory containing them which wasn't empty.
Fix the preservation of files in the "attic" directory properly so
we catch anything under the directory for the recipe, and replicate
the same structure in the attic directory rather than slightly
flattening it as we were before.

(From OE-Core rev: bbe63eb97ae7f78959f117d6066ef821c4da1c77)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-21 09:32:42 +00:00
Paul Eggleton e54f9c159d devtool / recipetool: use common code for launching editor
Looking at Chris Larson's code for starting the user's editor for
"recipetool newappend" it was slightly better than what I wrote for
"devtool edit-recipe" in that it checks VISUAL as well as EDITOR and
defaults to vi if neither are set, so break this out to its own function
and call it from both places. The broken out version passes shell=True
however in case it's a more complicated command rather than just a name
of an executable.

(From OE-Core rev: 184a256931e8cdc7bea97a905c4e67a435964de0)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-21 09:32:42 +00:00
Paul Eggleton dd35f69340 devtool: minor fix for error message
There is no -N/--name option for devtool, that's a recipetool option -
with devtool you just specify the name as a positional argument.

(From OE-Core rev: d2bc0cba5ca8a7220ffe1ef96acf856fe972ce7c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-21 09:32:42 +00:00
Paul Eggleton 2eb19cc723 classes/buildhistory: fix for python function parsing change
Variable expressions are no longer expanded in python functions as of
BitBake commit 8bf33a8e92c0e188fa392030025756196c96fcbb, so we've now
got to do this explicitly here.

(From OE-Core rev: 60fd4ff61a4ad240a89d48553002901c10e93178)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-15 16:28:45 +00:00
Paul Eggleton e64ce73b7e oe-selftest: devtool: add another devtool add test
Add a test so we validate the following recently implemented/fixed
aspects of "devtool add":

* Adding from a local directory with no name specified
* Auto-detecting the name and version from autoconf files
* Setting SRC_URI and S from the local git repo
* Showing the recipe file path in "devtool status" if it has been
  created within the workspace

Incidentally also tests:

* LICENSE and LIC_FILES_CHKSUM detection (though just for this piece of
  software)
* Extracting a dependency from autoconf (though just for this narrow
  case since there's only one)

(From OE-Core rev: d991fe7f17861dc5c21f7eac30d8c912b4d5dbed)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 12:33:03 +00:00
Paul Eggleton a5095d146d recipetool: create: set S when we set SRC_URI from local git repo
If you specify a local directory which happens to be a git repository
with an origin remote (and it is in fact remote), we can use that for
SRC_URI as implemented by OE-Core revision
b143d414846854dc8b3e0a47358daf5646eded38, however we also need to set S
if the recipe is going to be of any use fetching from that SRC_URI
later.

(From OE-Core rev: 8bbbd2d63f1bc752f9a30054a089dc2caf5fd84c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 12:33:03 +00:00
Paul Eggleton ca5a36cbd1 recipetool: create: convert http git URLs that don't end in .git but contain /git/
When recipetool create is given a URL that starts with http(s):// and
contains /git/, such as the URLs at git.yoctoproject.org, it's fairly safe
to assume it's a git repository and not something that should be fetched
with wget, so rewrite the URL.

(From OE-Core rev: 3ca04757a670e8b6f78799cc0454d75691809ac4)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 12:33:03 +00:00
Paul Eggleton 4c71afb378 recipetool: create: ensure URL parameters don't make it into the name
When auto-detecting the name for a recipe from the URL, strip off any
parameters (";name=value...") before parsing the URL, otherwise this
just ends up in the recipe name which is undesirable.

(From OE-Core rev: d3c46b5d0abd56bcadd4f2f1ef985f13d67f605b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 12:33:03 +00:00
Paul Eggleton 86f3464b47 devtool: add: fix adding from a local source directory
Fix a regression introduced in in OE-Core revision
aedfc5a5db1c4b2b80a36147c9a13b31764d91dd where specifying a local source
tree without specifying a name resulted in a traceback.

Fixes [YOCTO #9086].

(From OE-Core rev: 67ea9d21f20a371a0ee443b46326018cb1527b62)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 12:33:03 +00:00
Paul Eggleton fa50153bc2 devtool: modify: make -x the default behaviour
It's going to be more common for users not to have the prepared source
tree for a recipe already, so the default behaviour ought to be to
extract it for them from the recipe. Change the default to extract
(effectively making the -x option a no-op) and add a --no-extract/-n
option to disable it. Later we can look at trying to be smart and
reusing an existing source tree instead of erroring out if it exists;
for now this is just the default reversal.

(From OE-Core rev: 80a44e52609a89d9ffe816181ae193af491c06ac)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 12:33:03 +00:00
Paul Eggleton f767757b3e recipetool: create: determine name/version from github/bitbucket URLs
Both BitBucket and GitHub provide "release" downloads that unfortunately
don't include the actual project name. For recipetool this means we have
to have special handling for these URLs or else the current code assumes
the name of the project is something like "v0.9.1". Borrow and adapt
some code from autospec to do this, and put it in its own function for
clarity.

(From OE-Core rev: e8435fde7f82c39cc0efe1d4bf043b29f63898e9)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 12:33:03 +00:00
Paul Eggleton d94c7e367c recipetool: create: support cmake find_library directive
CMake supports a find_library() directive to find named libraries, so
detect dependencies from this.

(From OE-Core rev: d0265bc67f797ee4b7760cf37335994133809abf)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 12:33:03 +00:00
Paul Eggleton ddfe74447b devtool: commit for extra tasks that modify source when extracting
When extracting source for a recipe, if there are additional custom
tasks run that make changes to the source, create a commit in the
generated git branch so they are contained. This is particularly
useful for tasks that come before do_patch since otherwise the changes
might get incorporated in the first applied patch, but otherwise it
helps avoid the tree being dirty at any point.

Fixes [YOCTO #7626].

(From OE-Core rev: 997a77d9b20af1778b804778e5d8c8a7424f7582)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 12:33:02 +00:00
Paul Eggleton e36cb6c718 classes/externalsrc: create symlinks for workdir and logs
Auto-create symlinks in the source directory to the work directory
(${WORKDIR}) and logs directory (${T}) so that they are easier for the
user to find. This is particularly useful within the extensible SDK
where the user is less likely to be familiar enough with the structure
of the build system to know where to find these things, but otherwise
they are a useful shortcut for anyone.

(From OE-Core rev: 58adb3904c18acefd0da319e32f66ebca72eeaac)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 12:33:02 +00:00
Paul Eggleton 20034c3f7a classes/externalsrc: disable rm_work when active
If you're using externalsrc, it's very likely that you're going to want
to examine the intermediate build results even if the recipe builds
successfully; therefore you won't want rm_work to delete those.

(From OE-Core rev: c0867a425166086624bcffa231a1497291f860cb)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 12:33:02 +00:00