Age | Commit message (Collapse) | Author | Files | Lines |
|
First user will be the upcoming handover_2G/handover.py test in
I0b2671304165a1aaae2b386af46fbd8b098e3bd8.
Change-Id: Id799b3bb81eb9c04d13c26ff611e40363920300e
|
|
Allow enriching the junit output with arbitrary subtasks within a test.
The current aim is, for handover tests, to not just show that a test
failed, but to show exactly which steps worked and which didn't, e.g.:
handover.py/01_bts0_started PASSED
handover.py/02.1_ms0_attach PASSED
handover.py/02.2_ms1_attach PASSED
handover.py/02.3_subscribed_in_msc PASSED
handover.py/03_call_established PASSED
handover.py/04.1_bts1_started FAILED
In this case it is immediately obvious from looking at the jenkins
results analyzer that bts1 is the cause of the test failure, and it is
visible which parts of the test are flaky, over time.
First user Will be the upcoming handover_2G suite, in
I0b2671304165a1aaae2b386af46fbd8b098e3bd8.
Change-Id: I4ca9100b6f8db24d1f7e0a09b3b7ba88b8ae3b59
|
|
In a test, I called print() on a multi-line string and saw the log
showing each line 0.2 seconds apart. redirect.stdout seems to be pretty
inefficient.
Instead, put a print() function into the testenv, to directly call log()
on the strings passed to print().
The initial idea for redirect_stdout was that we could print() in any
deeper functions called from a test script. But we have no such nested
print() anywhere, only in test scripts themselves.
As a result of this, a multi-line print() in test scripts now no longer
puts the log prefix (timestamp, test name...) and suffix (backtrace /
source position) to each single line, but prints the multiline block
between a single log prefix and suffix -- exactly like the log()
function does everywhere else.
I actually briefly implemented adding the log prefix to each separate
line everywhere, but decided that it is not a good idea: in some places
we log config file snippets and other lists, and prepending the log
prefix to each line makes pasting such a snippet from (say) a jenkins
log super cumbersome. And the log prefix (backtrace) attached on each
separate line makes multiline blocks very noisy, unreadable.
Change-Id: I0972c66b9165bd7f2b0b387e0335172849199193
|
|
...because it otherwise causes confusing selftest log changes in
upcoming I0972c66b9165bd7f2b0b387e0335172849199193
Change-Id: I1eec31c5baba5ef33d0c8441b1e94a00d79af9bf
|
|
test.Test() overrides name() in order to provide source line number
information. However, overriding name() is the wrong place for that, as
name() is also often used for identifying an object - when listing the
tests of a suite, the line number should not appear in the test name.
For example, the line number sometimes ends up in the test results in
jenkins, making 'foo.py' and 'foo.py:23' two distinct report items.
Instead, add a separate function Origin.src() that defaults to name(),
but specific classes can override src() if they wish to provide more
detailed information with the object name.
Override src() in Test, not name().
Use src() in backtraces.
The suite_test.ok shows that the backtracing in the log remains
unchanged, but the place where the test name is printed is corrected:
I am 'test_suite' / 'hello_world.py:23'
becomes
I am 'test_suite' / 'hello_world.py'
(Notice that "[LINENR]" in suite_test.ok is a masking of an actual
number, done within the selftest suite)
Change-Id: I0c4698fa2b3db3de777d8b6dcdcee84e433c62b7
|
|
Remove ARFCNs as a concept from resource pool, assign a fixed ARFCN to
each BTS and TRX in the resource pools.
Using ARFCNs on specific bands as resources was an idea that is hard to
implement, because specific BTS dictate selection of bands which
influences which ARFCNs can be picked. That means reserving ARFCN
resources is only possible after reserving specific BTS resources, but
the tester is currently not capable of such two-stage resolution.
Writing handover tests, I got the problem that both BTS in a scenario
attempt to use the same ARFCN.
The by far easiest solution is to assign one fixed ARFCN to each BTS and
TRX. If ever needed, a scenario modifier can still configure different
ARFCNs.
(Due to uncertainty about OC2G operation stability, I prefer to leave
OC2G on ARFCN 50, as it happened to end up being configured before this
patch.)
Change-Id: I0a6c60544226f4261f9106013478d6a27fc39f38
|
|
On non-debug log level, show something like this at the beginning of
each suite:
03:45:49.439720 tst handover:sysmo+secondbts-trx-b200: RESERVED RESOURCES for handover:
bts
sysmoBTS 1002
Ettus B200
ip_address
10.42.42.2
10.42.42.3
10.42.42.4
10.42.42.5
10.42.42.6
10.42.42.7
modem
sierra_1st
sierra_2nd
Change-Id: Ic23556eafee654c93d13c5ef405028da09bd51d7
|
|
Makes for saner log output in stderr files.
Change-Id: I2755beffd18d6779367c04e4aaaed119d66eb3be
|
|
Before this patch, almost everything was in place to support concurrent
osmo-gsm-tester instances sharing a common state dir. However, during
resource reservation, if the reservation couldn't be done due to too
many resources being in use, osmo-gsm-tester would fail and skip the
test suite.
With this patch, OGT will wait until some reserved resources are
released and then try requesting the reservation again.
Change-Id: I938602ee890712fda82fd3f812d8edb1bcd05e08
|
|
tests can now use 'tenv.test().set_kpis(some_dict)' to set any kind of
data as KPIs, which will be presented in the junit report.
The representation of KPIs in the xml file doesn't follow the junit
format, mainly because it has no support for per-test properties.
Change-Id: I00e976f65a202e82d440bf33708f06c8ce2643e2
|
|
Change-Id: I8cad02abe776cc00b513113dbaf3c948ea7956cd
|
|
Timeout value can be specified by test in suite.conf:
config:
suite:
<suite_name>:
<test_name>:
timeout: 2 # 2 seconds timeout
Change-Id: I522f51f77f8be64ebfdb5d5e07ba92baf82d7706
|
|
The dictionary is generated once during test creation at suite startup.
Change-Id: I4de511d64ccf65d2ec6eec45dcf28529b74c5d26
|
|
This feature is not really implemented and maybe never was. In any case,
it makes sense to have that working per-test so we can specify different
values per test in case it's needed.
Change-Id: I3c1b95c10e974da87ec9abd25578d8bcc0bc55a3
|
|
Change-Id: I10fdfc5b2e3bf4c96941a3fae53230131f384de9
|
|
This way tests which require a very specific config file can override
specific template files used by object classes.
Change-Id: I65d1b1e826d2d430ee83810d998b98d0ccaa07cd
|
|
Change-Id: Iff1a4dafcbf2193e95b1fb6e5a6712f1bcdd6309
|
|
Change-Id: Id7576d21d0f9de29dd9bc4211eaf9b9ec634c990
|
|
tgz files in trials can be categorized in subdirectories, allowing to
select different bianry files at runtime based on the target run node
which is going to run them. This way for instance one can have a binary
linked against libs for eg. CentOS under run_label "centos/" or an ARM
target under "arm", and then use "run_label: arm" on the resource using
it.
Change-Id: Iaf2e97da3aff693395f44f0e93b184d4846cf6da
|
|
This allows inheriting suites or scenarios from eg. sysmocom/ dir, while
still allowing to apply new suites and scenarios on top.
Change-Id: Icecdae32d400a6b6da2ebf167c1c795f7a74ae96
|
|
Change-Id: I998c8674a55531909bfeac420064c3f238cea126
|
|
Change-Id: Ia4141001d084f690897dbdff5eae6c69ff2e521c
|
|
containing it
It has been notified that current configuration system is difficult to
understand and to use, so it has been envisioned to refactor it a bit.
The idea is that the user passes a -c path/to/main.conf file, which in
turn contains whatever osmo-gsm-tester main settings supports (basically
what old paths.conf used to be, plus some files harcoded to the same -c
directory are now configurable through the main configuration file).
Change-Id: Ieca65b71b543c44cfcec8e83efd0fe053c432e55
|
|
This covers some unit tests for scenario module (Scenario class).
Change-Id: I4c80047bb03ae8254c192057007fa7df84478605
|
|
Change-Id: Ia029de7ecda4c8dc3d0b4c412e4c9c0a65cf0185
|
|
Environment variable OSMO_GSM_TESTER_CONF is marked as deprecated and an
error is logged each time it's used. Same feature is available through
"-c" command line parameters, so having the envvar only makes things
more complex for no good reason. It cannot yet be completely dropped
since some environemnt still make use of it. Give some time to users to
adapt their setups.
Other environment variables setting some config apths can be dropped
since they are not being really used in any setup.
Change-Id: I7eb69f870d0dcb5906d45ae067d6bed1aabf5862
|
|
Change-Id: Idbe99a35993d193cd97059feb980e61ff14c67ad
|
|
Change-Id: Ia21b0f184bfd454b3835774949f581a27860a80a
|
|
Change-Id: Id21cda19df131e24402e1a593036e1e33a914920
|
|
Change-Id: I167bce29d7b22956c82053dcae5634c23d405b3f
|
|
On older versions apparently upon child termination due to SIGINT
subprocess.poll() returned 1. On new python versions (such as 3.8.2),
-2 is returned, according to documentation:
A negative value -N indicates that the child was terminated by signal N (Unix only).
Let's catch the SIGINT in the child process and exit with a known 42
value to fix different behavior.
Change-Id: I7949ff2b435e91e890061e6840b0f411f8b0a817
|
|
If using python 3.8.2, the trailing comma at the end of parameters is
not longer there (probably got fixed, since it's not needed). That
change breaks expected output.
Change-Id: I6da3024f946c0e761099058e812c0eacf3d6071f
|
|
Change-Id: I777ec59ffd83b861ddc09e85e52d7a8896264b8c
|
|
Change-Id: Ic9a4c90c190e9d8e1829fbef83a9243f3c6f655b
|
|
Change-Id: If4ab39be7a97d33e82c5a34e2a10dfec38613a4e
|
|
Change-Id: Ibb7fa5ab40bcf1e59705bdd2c2c5a76025b2b544
|
|
Change-Id: I0f36a5e61ff0ef14d08165c9c184acdf2d6e6901
|
|
This way we benefit from:
* knowing which attributes are used/required by each object class and
subclass
* Having validation function definitions near the class going to use them
Change-Id: I8fd6773c51d19405a585977af4ed72cad2b21db1
|
|
Change-Id: I0b2963cf00acd1f9036dca2e87aa3b87a0f8066d
|
|
tests
Change-Id: I9aec6c55ccd71894182057d36e0025b69925d314
|
|
If class Test calls get_run_dir(), it will generate the run dir from
Suite.get_run_dir() + append, and Suite.get_run_dir() in turn will do
the same with Trial.get_run_dir(). In suite_test.py, however, we are
passing a plain log.Origin object (parent of Trial), which doesn't
implement this function. We need to add it since tests will start using
get_run_dir() to write their own log files.
Generated tmp files are deleted on success and left there on error, in
order to be able to debug stuff easily.
Change-Id: Ic8326659a9ced0351e1e7e57ef21ce36f777729a
|
|
This avoids extra unneeded logging about killing with signal when
actually no signal is being sent.
Change-Id: I5b5dd78fe3301d8eef2ab93da3b37029268ae198
|
|
Change-Id: Icf1ac6774ea11880542012fd6c6ac73302bb74f5
|
|
It's always been the default anyway, so no need to explicitly state it.
Furthermore, recently (osmobsc 167cb828665b91ed7d28007abd4a1862ebbb78c9)
force-combined-si was enabled by default for nanoBTS due to issues with
some MS not registering when it's not enabled, so let's osmo-bsc decide
were should it be enabled.
Related: OS#3063
Change-Id: I97b10406a4a0048c66a47aa7d32a113f0c0366d1
|
|
VTY cmd "logging level all" is deprecated in favour of "logging level
set-all", but the required behaviour in this case is "logging level
set-all".
Change-Id: Ice54fca6be86f05a02ce4ae783e6e42e6958b95d
|
|
* This commit is a preparation for future commits to add support for
different osmo-trx devices and backends like osmo-trx-lms.
* Drop deprecated osmo-trx-* cmd line params and use VTY cfg to set them.
* As number of osmo-trx related osmo-gsm-tester attributes grow, group
them togther in an "osmo_trx" dictionary.
Change-Id: I77d29413c9e3b600b796627ba366f80c3281b7e1
|
|
Before this patch, scenarios were only used to select resources with
specific attributes. This commit introduces "modifiers" in scenarios,
which allows setting or modifing config attributes of resources once
they have been reserved. This way same test can be run selecting same
resources but modifying its configuration, allowing for instance running
different number of TRX, different timeslot configuration, etc.
Modifiers are described by placing a "modifiers" dictionary in any
scenario file, similar to the current "resources" one used to select
requird resources. The "modifiers" dictionary is overlaid on top of the
"resources" one resulting from combining all the "resources" dictionary
of all scenario files.
Change-Id: If8c422c67d9a971d9ce2c72594f55cde2db7550d
|
|
We are using the CDF to decide which percentage of the jobs should
be running at a given point. The x-axis is time and the y-axis the
percentage of how many jobs should be running.
There are three functions to do this. The first one is a constant
which would result in everything being started right now, one to
start them linearly and the last (formula from Qt/3rdparty) to first
accelerate and decelerate slowly.
Change-Id: I9e3064f4c3c4c7af5d3491f850090516e541f4d3
|
|
Since libosmocore doesn't support XOR at all, it seems weird to use it in the
test, even though it's just a selftest without libosmocore involved...
Change-Id: I51edec255e7ef277907817b3187c2f492465467f
|
|
Change-Id: Ie65d7d33322c775a1040b5c81f367d882a7c2c5b
|