mirrors/meta-arm
1
0
Fork 0
mirror of https://git.yoctoproject.org/meta-arm synced 2026-01-11 15:00:39 +00:00
Code Issues Projects Releases Wiki Activity

arm/oeqa: Merge all OEFVP*Target classes

Browse Source 
The differences between OEFVPTarget, OEFVPSSHTarget and
OEFVPSerialTarget are not obvious and there is a lot of duplication.

Merge all the logic into one OEFVPTarget (again). This has the following
features:
 * Run SSH commands
 * Run serial console assertions
 * Lazily await a Linux login prompt while running test cases (only when
   self.target.run is called).

Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Jon Mason <jon.mason@arm.com>
This commit is contained in:
Peter Hoyes
2023-07-17 19:56:23 +01:00
committed by Jon Mason
parent cf0baaee0b
commit ea762113ce
7 changed files with 73 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files

48
documentation/oeqa-fvp.md
View File

@@ -4,32 +4,36 @@ OE-Core's [oeqa][OEQA] framework provides a method of performing runtime tests o
Tests can be configured to run automatically post-build by setting the variable `TESTIMAGE_AUTO="1"`, e.g. in your Kas file or local.conf.
There are two main methods of testing, using different test "targets". Both test targets generate an additional log file with the prefix 'fvp_log' in the image recipe's `${WORKDIR}/testimage` containing the FVP's stdout.
## OEFVPTarget
This runs test cases on a machine using SSH. It therefore requires that an SSH server is installed in the image.
In test cases, the primary interface with the target is, e.g:
```
(status, output) = self.target.run('uname -a')
```
which runs a single command on the target (using `ssh -c`) and returns the status code and the output. It is therefore useful for running tests in a Linux environment.
For examples of test cases, see meta/lib/oeqa/runtime/cases in OE-Core. The majority of test cases depend on `ssh.SSHTest.test_ssh`, which first validates that the SSH connection is functioning.
Example machine configuration:
meta-arm provides the OEFVPTarget which must be set up in the machine configuration:
```
TEST_TARGET = "OEFVPTarget"
TEST_SERVER_IP = "127.0.0.1"
TEST_TARGET_IP = "127.0.0.1:8022"
IMAGE_FEATURES:append = " ssh-server-dropbear"
FVP_CONFIG[bp.virtio_net.hostbridge.userNetPorts] ?= "8022=22"
FVP_CONSOLES[default] = "terminal_0"
FVP_CONSOLES[tf-a] = "s_terminal_0"
```
## OEFVPSerialTarget
The test target also generates a log file with the prefix 'fvp_log' in the image recipe's `${WORKDIR}/testimage` containing the FVP's stdout.
This runs tests against one or more serial consoles on the FVP. It is more flexible than OEFVPTarget, but test cases written for this test target do not support the test cases in OE-core. As it does not require an SSH server, it is suitable for machines with performance or memory limitations.
OEFVPTarget supports two different test interfaces - SSH and pexpect.
## SSH
As in OEQA in OE-core, tests cases can run commands on the machine using SSH. It therefore requires that an SSH server is installed in the image.
This uses the `run` method on the target, e.g:
```
(status, output) = self.target.run('uname -a')
```
which executes a single command on the target (using `ssh -c`) and returns the status code and the output. It is therefore useful for running tests in a Linux environment.
For examples of test cases, see meta/lib/oeqa/runtime/cases in OE-Core. The majority of test cases depend on `ssh.SSHTest.test_ssh`, which first validates that the SSH connection is functioning.
## pexpect
To support firmware and baremetal testing, OEFVPTarget also allows test cases to make assertions against one or more consoles using the pexpect library.
Internally, this test target launches a [Pexpect][PEXPECT] instance for each entry in FVP_CONSOLES which can be used with the provided alias. The whole Pexpect API is exposed on the target, where the alias is always passed as the first argument, e.g.:
```
@@ -39,16 +43,6 @@ self.assertNotIn(b'ERROR:', self.target.before('tf-a'))
For an example of a full test case, see meta-arm/lib/oeqa/runtime/cases/linuxboot.py This test case can be used to minimally verify that a machine boots to a Linux shell. The default timeout is 10 minutes, but this can be configured with the variable TEST_FVP_LINUX_BOOT_TIMEOUT, which expects a value in seconds.
The SSH interface described above is also available on OEFVPSerialTarget to support writing a set of hybrid test suites that use a combination of serial and SSH access. Note however that this test target does not guarantee that Linux has booted to shell prior to running any tests, so the test cases in OE-core are not supported.
Example machine configuration:
```
TEST_TARGET="OEFVPSerialTarget"
TEST_SUITES="linuxboot"
FVP_CONSOLES[default] = "terminal_0"
FVP_CONSOLES[tf-a] = "s_terminal_0"
```
[OEQA]: https://docs.yoctoproject.org/test-manual/intro.html
[FVP]: https://developer.arm.com/tools-and-software/simulation-models/fixed-virtual-platforms
[RUNFVP]: runfvp.md

2
meta-arm-bsp/conf/machine/corstone1000-fvp.conf
View File

@@ -8,7 +8,7 @@ TFA_TARGET_PLATFORM = "fvp"
TFM_PLATFORM_IS_FVP = "TRUE"
# testimage config
TEST_TARGET = "OEFVPSerialTarget"
TEST_TARGET = "OEFVPTarget"
TEST_SUITES = "linuxboot"
# FVP Config

2
meta-arm-bsp/conf/machine/corstone500.conf
View File

@@ -32,7 +32,7 @@ WKS_FILE_DEPENDS:append = " ${EXTRA_IMAGEDEPENDS}"
WKS_FILE ?= "core-image-minimal.corstone500.wks"
TEST_TARGET = "OEFVPSerialTarget"
TEST_TARGET = "OEFVPTarget"
TEST_SUITES = "linuxboot"
FVP_PROVIDER ?= "fvp-corstone500-native"

2
meta-arm-bsp/conf/machine/fvp-baser-aemv8r64.conf
View File

@@ -29,7 +29,7 @@ MACHINE_FEATURES:append = " efi"
MACHINE_EXTRA_RRECOMMENDS += "ssh-pregen-hostkeys"
# testimage configuration
TEST_TARGET = "OEFVPSerialTarget"
TEST_TARGET = "OEFVPTarget"
TEST_SUITES = "linuxboot"
TEST_TARGET_IP ?= "127.0.0.1:8022"
TEST_SERVER_IP ?= "127.0.1.1"

2
meta-arm-bsp/conf/machine/tc1.conf
View File

@@ -6,7 +6,7 @@
require conf/machine/include/tc.inc
TEST_TARGET = "OEFVPSerialTarget"
TEST_TARGET = "OEFVPTarget"
TEST_SUITES = "linuxboot"
# FVP Config

120
meta-arm/lib/oeqa/controllers/fvp.py
View File

@@ -6,11 +6,14 @@ from oeqa.core.target.ssh import OESSHTarget
from fvp import runner
class OEFVPSSHTarget(OESSHTarget):
class OEFVPTarget(OESSHTarget):
"""
Base class for meta-arm FVP targets.
Contains common logic to start and stop an FVP.
For compatibility with OE-core test cases, this target's start() method
waits for a Linux shell before returning to ensure that SSH commands work
with the default test dependencies.
"""
DEFAULT_CONSOLE = "default"
def __init__(self, logger, target_ip, server_ip, timeout=300, user='root',
port=None, dir_image=None, rootfs=None, bootlog=None, **kwargs):
super().__init__(logger, target_ip, server_ip, timeout, user, port)
@@ -19,25 +22,61 @@ class OEFVPSSHTarget(OESSHTarget):
basename = pathlib.Path(rootfs)
basename = basename.name.replace("".join(basename.suffixes), "")
self.fvpconf = image_dir / (basename + ".fvpconf")
self.bootlog = bootlog
if not self.fvpconf.exists():
raise FileNotFoundError(f"Cannot find {self.fvpconf}")
def _after_start(self):
pass
self.bootlog = bootlog
self.terminals = {}
self.booted = False
def start(self, **kwargs):
self.fvp_log = self._create_logfile("fvp")
self.fvp = runner.FVPRunner(self.logger)
self.fvp.start(self.fvpconf, stdout=self.fvp_log)
self.logger.debug(f"Started FVP PID {self.fvp.pid()}")
self._after_start()
self._setup_consoles()
def await_boot(self):
if self.booted:
return
# FVPs boot slowly, so allow ten minutes
boot_timeout = 10*60
try:
self.expect(OEFVPTarget.DEFAULT_CONSOLE, "login\\:", timeout=boot_timeout)
self.logger.debug("Found login prompt")
self.booted = True
except pexpect.TIMEOUT:
self.logger.info("Timed out waiting for login prompt.")
self.logger.info("Boot log follows:")
self.logger.info(b"\n".join(self.before(OEFVPTarget.DEFAULT_CONSOLE).splitlines()[-200:]).decode("utf-8", errors="replace"))
raise RuntimeError("Failed to start FVP.")
def stop(self, **kwargs):
returncode = self.fvp.stop()
self.logger.debug(f"Stopped FVP with return code {returncode}")
def run(self, cmd, timeout=None):
self.await_boot()
return super().run(cmd, timeout)
def _setup_consoles(self):
with open(self.fvp_log.name, 'rb') as logfile:
parser = runner.ConsolePortParser(logfile)
config = self.fvp.getConfig()
for name, console in config["consoles"].items():
logfile = self._create_logfile(name)
self.logger.info(f'Creating terminal {name} on {console}')
port = parser.parse_port(console)
self.terminals[name] = \
self.fvp.create_pexpect(port, logfile=logfile)
# testimage.bbclass expects to see a log file at `bootlog`,
# so make a symlink to the 'default' log file
if name == 'default':
default_test_file = f"{name}_log{self.test_log_suffix}"
os.symlink(default_test_file, self.bootlog)
def _create_logfile(self, name):
if not self.bootlog:
return None
@@ -54,71 +93,6 @@ class OEFVPSSHTarget(OESSHTarget):
os.symlink(fvp_log_file, fvp_log_symlink)
return open(fvp_log_path, 'wb')
class OEFVPTarget(OEFVPSSHTarget):
"""
For compatibility with OE-core test cases, this target's start() method
waits for a Linux shell before returning to ensure that SSH commands work
with the default test dependencies.
"""
def __init__(self, logger, target_ip, server_ip, **kwargs):
super().__init__(logger, target_ip, server_ip, **kwargs)
self.logfile = self.bootlog and open(self.bootlog, "wb") or None
# FVPs boot slowly, so allow ten minutes
self.boot_timeout = 10 * 60
def _after_start(self):
with open(self.fvp_log.name, 'rb') as logfile:
parser = runner.ConsolePortParser(logfile)
config = self.fvp.getConfig()
self.logger.debug(f"Awaiting console on terminal {config['consoles']['default']}")
port = parser.parse_port(config['consoles']['default'])
console = self.fvp.create_pexpect(port)
try:
console.expect("login\\:", timeout=self.boot_timeout)
self.logger.debug("Found login prompt")
except pexpect.TIMEOUT:
self.logger.info("Timed out waiting for login prompt.")
self.logger.info("Boot log follows:")
self.logger.info(b"\n".join(console.before.splitlines()[-200:]).decode("utf-8", errors="replace"))
raise RuntimeError("Failed to start FVP.")
class OEFVPSerialTarget(OEFVPSSHTarget):
"""
This target is intended for interaction with the target over one or more
telnet consoles using pexpect.
This still depends on OEFVPSSHTarget so SSH commands can still be run on
the target, but note that this class does not inherently guarantee that
the SSH server is running prior to running test cases. Test cases that use
SSH should first validate that SSH is available, e.g. by depending on the
"linuxboot" test case in meta-arm.
"""
DEFAULT_CONSOLE = "default"
def __init__(self, logger, target_ip, server_ip, **kwargs):
super().__init__(logger, target_ip, server_ip, **kwargs)
self.terminals = {}
def _after_start(self):
with open(self.fvp_log.name, 'rb') as logfile:
parser = runner.ConsolePortParser(logfile)
config = self.fvp.getConfig()
for name, console in config["consoles"].items():
logfile = self._create_logfile(name)
self.logger.info(f'Creating terminal {name} on {console}')
port = parser.parse_port(console)
self.terminals[name] = \
self.fvp.create_pexpect(port, logfile=logfile)
# testimage.bbclass expects to see a log file at `bootlog`,
# so make a symlink to the 'default' log file
if name == 'default':
default_test_file = f"{name}_log{self.test_log_suffix}"
os.symlink(default_test_file, self.bootlog)
def _get_terminal(self, name):
return self.terminals[name]

4
meta-arm/lib/oeqa/runtime/cases/linuxboot.py
View File

@@ -5,9 +5,7 @@ from oeqa.runtime.case import OERuntimeTestCase
class LinuxBootTest(OERuntimeTestCase):
"""
This test case is only compatible with the OEFVPSerialTarget as it uses
the pexpect interface. It waits for a Linux login prompt on the default
console.
This test waits for a Linux login prompt on the default console.
"""
def setUp(self):