| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
If class Test calls get_run_dir(), it will generate the run dir from
Suite.get_run_dir() + append, and Suite.get_run_dir() in turn will do
the same with Trial.get_run_dir(). In suite_test.py, however, we are
passing a plain log.Origin object (parent of Trial), which doesn't
implement this function. We need to add it since tests will start using
get_run_dir() to write their own log files.
Generated tmp files are deleted on success and left there on error, in
order to be able to debug stuff easily.
Change-Id: Ic8326659a9ced0351e1e7e57ef21ce36f777729a
|
|
This avoids extra unneeded logging about killing with signal when
actually no signal is being sent.
Change-Id: I5b5dd78fe3301d8eef2ab93da3b37029268ae198
|
|
Change-Id: Icf1ac6774ea11880542012fd6c6ac73302bb74f5
|
|
It's always been the default anyway, so no need to explicitly state it.
Furthermore, recently (osmobsc 167cb828665b91ed7d28007abd4a1862ebbb78c9)
force-combined-si was enabled by default for nanoBTS due to issues with
some MS not registering when it's not enabled, so let's osmo-bsc decide
were should it be enabled.
Related: OS#3063
Change-Id: I97b10406a4a0048c66a47aa7d32a113f0c0366d1
|
|
VTY cmd "logging level all" is deprecated in favour of "logging level
set-all", but the required behaviour in this case is "logging level
set-all".
Change-Id: Ice54fca6be86f05a02ce4ae783e6e42e6958b95d
|
|
* This commit is a preparation for future commits to add support for
different osmo-trx devices and backends like osmo-trx-lms.
* Drop deprecated osmo-trx-* cmd line params and use VTY cfg to set them.
* As number of osmo-trx related osmo-gsm-tester attributes grow, group
them togther in an "osmo_trx" dictionary.
Change-Id: I77d29413c9e3b600b796627ba366f80c3281b7e1
|
|
Before this patch, scenarios were only used to select resources with
specific attributes. This commit introduces "modifiers" in scenarios,
which allows setting or modifing config attributes of resources once
they have been reserved. This way same test can be run selecting same
resources but modifying its configuration, allowing for instance running
different number of TRX, different timeslot configuration, etc.
Modifiers are described by placing a "modifiers" dictionary in any
scenario file, similar to the current "resources" one used to select
requird resources. The "modifiers" dictionary is overlaid on top of the
"resources" one resulting from combining all the "resources" dictionary
of all scenario files.
Change-Id: If8c422c67d9a971d9ce2c72594f55cde2db7550d
|
|
We are using the CDF to decide which percentage of the jobs should
be running at a given point. The x-axis is time and the y-axis the
percentage of how many jobs should be running.
There are three functions to do this. The first one is a constant
which would result in everything being started right now, one to
start them linearly and the last (formula from Qt/3rdparty) to first
accelerate and decelerate slowly.
Change-Id: I9e3064f4c3c4c7af5d3491f850090516e541f4d3
|
|
Since libosmocore doesn't support XOR at all, it seems weird to use it in the
test, even though it's just a selftest without libosmocore involved...
Change-Id: I51edec255e7ef277907817b3187c2f492465467f
|
|
Change-Id: Ie65d7d33322c775a1040b5c81f367d882a7c2c5b
|
|
Change-Id: I01485c5d74e5fe62d0ffea9eb1fad29041426eef
|
|
Make the code more reachable for newcomers, as well as more organized
for people who work a lot with it.
SuiteRun in suite.py is already quite big, and having the Test class in
there make it unnecessarily more big, and makes it difficult to find
stuff.
At the same time, having a test.py which does actually not contain the
Test class but other stuff, makes it even more confusing.
Change-Id: I9c8d67f598466ba52a4827ff77027b9eae85929a
|
|
Preparation to move current Test class into its own module test from
suite module.
Change-Id: I520bd046cb09042b5567d967f951f050e4e02e85
|
|
Change-Id: If4559b945a3e71f3a36fd7ac760cb094278f6b39
|
|
Change-Id: I654484b21ca2e950ff3019ee20133e9cb50296bb
|
|
Change-Id: I532f66585ab6f7be1a71853190791feeecbf263c
|
|
... instead of using the one from from osmo vty directly.
This way we avoid having multiple word attribute value and we can skip
using quotes in the conf files.
Change-Id: I5265cc9990dd5e99dba1f6262b3a8c597a3e958d
|
|
Now that combination and matching of lists works properly, update the
suite test to check a more complex scenario.
Change-Id: Ib0b82dc0c2a7f27cd6b2ffc32eed5f1287692ddc
|
|
We want to handle lists in the same way as we handle them in combine().
Without this commit, reserve()->find() failed to match objects
containing dictionaries inside lists correctly (such as trx configs).
A few attributes are added to trx_list of some resources in
suite_test/resources.conf to show a case in which resource reservation
would fail without this patch. It failed because before this patch,
dictionaries inside lists are compared to be equal instead of being
compared element by element to see if one dictionary is a subset of the
other one (for each element in the lists).
Change-Id: I8588d5b788b9f74a9cc84b8bdcb049921788bb48
|
|
This commit fixes combination of resources containing lists.
For lists containing complex types, it has been decided to handle them
as sorted list, where position in list matters. In this case, combine is
called recursively for each element in dest and src sharing position in
the list, and assumes that if one list is shorter than the other, then
it has to be combined against empty set for that tye.
For instance this is useful when defining trx_list properties, where a
BTS can have a different amount of TRX but we may be interested in
restricting the first TRX and don't care about extra TRX.
For lists containing simple types (eg. integers or strings), we just want
to merge both lists and we only need to check if the value is already there,
ie. handle them as unsortered sets. This case won't work if we call combine
for each element of the list because for a simple case it will just end up
checking if a[i] == b[i].
This kind of operation for simple types is needed in later commits where
cipher attribute is introduced. Without this patch, having following 2
scenarios and trying them to use together "-s foosuite:cipher-a50+ciphera51"
will fail:
cipher_a50.conf:
bts:
- ciphers:
- 'a5 0'
cipher_a51.conf
bts:
- ciphers:
- 'a5 1'
ValueError: cannot combine dicts, conflicting items (values 'a5 0' and 'a5 1')
Change-Id: Ib7a38f10eb9de338a77bf1fa3afceb9df1532015
|
|
As suite.conf and scenarios need to match 1-to-1 in lists, it's important to
extend the dictionaries by replicating the objects with a 'times' values
higher than 1 in order to match the objects correctly.
Since dictionanries are expanded at combine time, there's no need to
expand them during reserve() time because they are already expanded.
As a result, this commit reworks the kind of schema applied in each
place (and takes the change to start validating scenario files, which
were neglected previously).
Two unit tests are added as a show case. Unfortunately output showing
scenario dictionaries needs to be ignored while verifying because it was
encountered that different versions of python print dictionary elements
in different order.
Change-Id: I25eb639c7e3cf3b4c67a205422808bffbdd791e6
|
|
Before this commit, only max_power_red was specified and it could only
be used as a defaults and could not be set per object. Together with
nominal_power, it can be useful to be able to set them to different
values for different objects, as for example different osmo-bts-trx
objects can require different values.
Change-Id: I472742e98052cc39686d38c945be76d7f50eeebd
|
|
Do not pass os.O_TRUNC to open(), that immediately empties out the lock file.
Fix lock_test to be able to catch this error. So far things were happening too
fast to notice it.
Caught by running 'make check' on a main unit APU that has different timing
behavior.
Change-Id: I872a3d2548ac84097ac7acf13cb12c36822e076e
|
|
As described in https://osmocom.org/issues/2380 and related gerrit
patch https://gerrit.osmocom.org/#/c/3306 the config files currently
contain timers with a 'zero' value, leading to issues.
The config file templates used in osmo-gsm-tester suffer actually
from exactly the same issue: They basically contain values that were
written back to the file based on the default initialization in libbsc,
which unfortunately was zero in some cases.
Let's remove those values, they are the (bad) default anyway.
libbsc patches in the pipeline will introduce reasonable defaults
and forbid the use of zero for timers.
Change-Id: I8cde29a597a17c2659b3b87268be4b12975f4bef
|
|
This is useful while debugging and trying to check events across other
outputs such as pcap files, process logs, etc.
Change-Id: I43bb5c6e9977189251802bc2b078c52eb046bab8
|
|
As defined in [1], the different related actors are implemented in this
commit: ESME and SMSC.
SMSC: In Osmocom, the SMSC is currently implemented inside the NITB or
the MSC. A new Smsc abstract class is created to shared code between the
NITB and the MSC, and also makes it easier for later when the SMSC is
splitted. ESMEs can be dynamically added to its configuration in a
similar way to how the BTSs are added.
ESME: A new class Esme is created which can be used by tests to control
an ESME to interact with the SMSC. The ESME functionalities are
implemented using python-smpplib. Required version of this library is at
least 43cc6f819ec76b2c0a9d36d1d439308634716227, which contains support
for python 3 and some required features to poll the socket.
This commit already contains a few tests which checks different
features and tests the API. Extending tested features or scenarios can be
later done quite easily.
The tests are not enabled by default right now, because there are several
of them in a suite and the ip_address resources are not freed after every
tests which ends up in the suite failing due to missing reserved
resources. All the tests run alone work though. When the issue is fixed
they can then be added to the default list of tests to be run.
[1] http://opensmpp.org/specs/SMPP_v3_4_Issue1_2.pdf
Change-Id: I14ca3cb009d6d646a449ca99b0200da12085c0da
|
|
This is kept separate to not clutter up previous patch
I5f9b53150f2bb6fa9d63ce27f0806f0ca6a45e90.
Change-Id: I0ce50375fdb028da96c2159d577d8ed1967d4fe6
|
|
With the recent fix of the junit report related issues, another issue arose:
the 'with log.Origin' was changed to disallow __enter__ing an object twice to
fix problems, now still code would fail because it tries to do 'with' on the
same object twice. The only reason is to ensure that logging is associated with
a given object. Instead of complicating even more, implement differently.
Refactor logging to simplify use: drop the 'with Origin' style completely, and
instead use the python stack to determine which objects are created by which,
and which object to associate a log statement with.
The new way: we rely on the convention that each class instance has a local
'self' referencing the object instance. If we need to find an origin as a new
object's parent, or to associate a log message with, we traverse each stack
frame, fetching the first local 'self' object that is a log.Origin class
instance.
How to use:
Simply call log.log() anywhere, and it finds an Origin object to log for, from
the stack. Alternatively call self.log() for any Origin() object to skip the
lookup.
Create classes as child class of log.Origin and make sure to call
super().__init__(category, name). This constructor will magically find a parent
Origin on the stack.
When an exception happens, we first escalate the exception up through call
scopes to where ever it is handled by log.log_exn(). This then finds an Origin
object in the traceback's stack frames, no need to nest in 'with' scopes.
Hence the 'with log.Origin' now "happens implicitly", we can write pure natural
python code, no more hassles with scope ordering.
Furthermore, any frame can place additional logging information in a frame by
calling log.ctx(). This is automatically inserted in the ancestry associated
with a log statement / exception.
Change-Id: I5f9b53150f2bb6fa9d63ce27f0806f0ca6a45e90
|
|
A bit of refactoring to fix logging and error reporting, and simplify the code.
This transmogrifies some of the things committed in
0ffb41440661631fa1d520c152be4cf8ebd4c46b "Add JUnit XML reports; refactor test
reporting", which did not fully match the code structuring ideas used in
osmo-gsm-tester. Also solve some problems present from the start of the code
base.
Though this is a bit of a code bomb, it would take a lot of time to separate
this into smaller bits: these changes are closely related and resulted
incrementally from testing error handling and logging details. I hope it's ok.
Things changed / problems fixed:
Allow only a single trial to be run per cmdline invocation: unbloat trial and
suite invocation in osmo-gsm-tester.py.
There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun.
SuiteDefinition had a list of tests, which was modified by the SuiteRun to
record test results. Instead, have only the test basenames in the
SuiteDefinition and create a new set of Test() instances for each SuiteRun, to
ensure that no state leaks between separate suite runs.
State leaking across runs can be seen in
http://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run/453/
where an earlier sms test for sysmo succeeds, but its state gets overwritten by
the later sms test for trx that fails. The end result is that both tests
failed, although the first run was successful.
Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once,
skipping the second entry. The problem there is that we'd still __exit__ twice
or more, popping the Origin off the stack even though it should still remain.
We could count __enter__ recurrences, but instead, completely disallow entering
a second time.
A code path should have one 'with' statement per object, at pivotal points like
run_suites or run_tests. Individual utility functions should not do 'with' on a
central object. The structure needed is, in pseudo code:
try:
with trial:
try:
with suite_run:
try:
with test:
test_actions()
The 'with' needs to be inside the 'try', so that the exception can be handled
in __exit__ before it reaches the exception logging.
To clarify this, like test exceptions caught in Test.run(), also move suite
exception handling from Trial into SuiteRun.run_tests(). There are 'with self'
in Test.run() and SuiteRun.run_tests(), which are well placed, because these
are pivotal points in the main code path.
Log output: clearly separate logging of distinct suites and test scripts, by
adding more large_separator() calls at the start of each test. Place these
separator calls in more logical places. Add separator size and spacing args.
Log output: print tracebacks only once, for the test script where they happen.
Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and
suite.test_skipped_ctr, instead add SuiteRun.count_test_results().
For test failure reporting, store the traceback text in a separate member var.
In the text report, apply above changes and unclutter to achieve a brief and
easy to read result overview: print less filler characters, drop the starting
times, drop the tracebacks. This can be found in the individual test logs.
Because the tracebacks are no longer in the text report, the suite_test.py can
just print the reports and expect that output instead of asserting individual
contents.
In the text report, print duration in precision of .1 seconds.
Add origin information and a traceback text to the junit XML result to give
more context when browsing the result XML. For 'AssertionError', add the source
line of where the assertion hit.
Drop the explicit Failure exception. We don't need one specific exception to
mark a failure, instead any arbitrary exception is treated as a failure. Use
the exception's class name as fail_type.
Though my original idea was to use raising exceptions as the only way to cause
a test failure, I'm keeping the set_fail() function as an alternative way,
because it allows test specific cleanup and may come in handy later. To have
both ways integrate seamlessly, shift some result setting into 'finally'
clauses and make sure higher levels (suite, trial) count the contained items'
stati.
Minor tweak: write the 'pass' and 'skip' reports in lower case so that the
'FAIL' stands out.
Minor tweak: pass the return code that the program exit should return further
outward, so that the exit(1) call does not cause a SystemExit exception to be
logged.
The aims of this patch are:
- Logs are readable so that it is clear which logging belongs to which test and
suite.
- The logging origins are correct (vs. parents gone missing as previously)
- A single test error does not cause following tests or suites to be skipped.
- An exception "above" Exception, i.e. SystemExit and the like, *does*
immediately abort all tests and suites, and the results for tests that were
not run are reported as "unknown" (rather than skipped on purpose):
- Raising a SystemExit aborts all.
- Hitting ctrl-c aborts all.
- The resulting summary in the log is brief and readable.
Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
|
|
When trying to reserve more resources than available in the resources.conf,
actually print an intelligible error message: catch the nameless error from
solve() and fill in what was requested for solution.
Change-Id: Iba3707f1aaeb40a58c616c33af52a60c9a2e7e1f
|
|
Also add various comments to illustrate what is going on during origin
resolution.
In the regression tests' expectations, some duplicate entries in the origins
are removed, and hence no list of deeper origin ancestry is printed anymore.
Change-Id: I42c3b8635b54c31c27699140e200c1f75a6ada29
|
|
The "Affero" nature makes sense for the Osmocom network components like
BSC, SGSN, etc. as they are typically operated to provide a network
service.
For testing, this doesn't make so much sense as it is difficult to
imagine people creating a business out of offering to run test cases on
an end-to-end Osmocom GSM network. So let's drop the 'Affero' here.
All code is so far developed by sysmocom staff, so as Managing Director
of sysmocom I can effect such a license change unilaterally.
Change-Id: I8959c2d605854ffdc21cb29c0fe0e715685c4c05
|
|
Change-Id: I4b033dc0d3d43825901308cda05c6999946b664e
|
|
Add test of this to sms_test.py
Change-Id: Ib0b420a81c1b211a9f03a2521d244b3a427d5e5b
|
|
Preparation for following commits to add smpp support, as we will have a
class SmppClient with a method accepting an Sms object to send it.
Change-Id: I1f28e14e963abb64df687b69d54975be2aeb0d0d
|
|
Implement the Modem.log_info() function, use that instead of logging all modem
properties.
Tweak mo_mt_sms.py print() statements.
Pass modem object to SMS generation to inlcude the modem name as SMS token.
Change-Id: I2b17fce0b3b05594fd9038b54e5b65f5127bd0a4
|
|
It's the NITB's address, so it should go in the nitb.* scope.
Change-Id: I71a5ef153b7156b0644253f5aa8a0c848f42ab3b
|
|
I would like to use the IP addresses also for OsmoBSC processes, so it is more
than clear now that 'nitb_iface' was the wrong naming choice.
The only distinction we may need in the future is public versus loopback
interface. To add that, we may add a trait to the 'ip_address' resource
like:
ip_address:
- addr: 10.42.42.1
type: public
- addr: 127.0.0.1
type: loopback
This way we can substitute public vs loopback addresses flexibly (e.g. using
scenarios).
Change-Id: I3ad583ae7a33f7a7bb56fe78a125f73c56a0e860
|
|
Change-Id: Icd9022f5732caac65d16fc49f2ed7833199523de
|
|
This file is already created in run_suites().
Creating it in here too makes a new file name()_2.xml created and
populated instead of the original name().xml one.
Change-Id: Ibec14a1ef5fd029aa37ce666d00236bb872d636e
|
|
* Add Junit output file support
* Differentiate between an expected failure test and an error in the
test, as described in JUnit.
* In case of an error/exception during test, record and attach it to the
Test object and continue running the tests, and show it at the end
during the trial report.
Change-Id: Iedf6d912b3cce3333a187a4ac6d5c6b70fe9d5c5
|
|
No actual failure known, but using the abspath will prevent further problems
when running a binary in its own run dir and passing trial paths to it.
Adjust test expectations.
Change-Id: Ia2ba876a42eef9122a92645edd1f28ddb56e6be3
|
|
On the log output, clearly show which state dir is used. Since all runs should
use the same state dir, this is important and/or reassuring to see.
Change-Id: Ie92f1879a35d8bb399ee916b7ef72f9ee7d47409
|
|
Mostly to clarify an empty lock in the end.
Change-Id: If80960b331d30125e97878d52edba9b4a63e87c7
|
|
Remove the stdout printing from the lock_test_help.py, so that there is no
problem with a failing test because one output ending up in stdout before the
other. So only output the current lock from lock_test.sh.
Change-Id: I7fe80a9de88ddee1d66ca5c692948fc9673e4230
|
|
Change-Id: Ideab363753559ba0b49352dc9da03a27d7b0dadf
|
|
Log on level 'log', more clearly show whether it's for reservation or actual
use, show the origin that is asking for them.
Change-Id: I3b78c7bdcaec90943900343c878099160f8d2f64
|
|
Change-Id: I1159395251332f3b1af3b3a322e7191559105faa
|
|
Rename log_sink to log_write_func to make it more clear.
Remember the list of log targets as static member LogTarget.all_targets.
Make each LogTarget instance register with the list of targets implicitly.
No longer create a default log target, rather create one explicitly in
osmo-gsm-tester.py.
Change-Id: I5844485eaed536cb34b44bfe23dc635fe1982dcd
|
|
Tweak test expectations to include the new debug logging.
Go through the paths in alphabetical order to get deterministic logging output,
so the test expectations always match.
Change-Id: I11a905b2467cda691d9ccea30ae436bac96476c9
|
