aboutsummaryrefslogtreecommitdiffstats
path: root/selftest
AgeCommit message (Collapse)AuthorFilesLines
2020-12-11osmo_ctrl.py: add RateCountersNeels Hofmeyr4-0/+227
First user will be the upcoming handover_2G/handover.py test in I0b2671304165a1aaae2b386af46fbd8b098e3bd8. Change-Id: Id799b3bb81eb9c04d13c26ff611e40363920300e
2020-12-11add test.report_fragment()Neels Hofmeyr4-8/+95
Allow enriching the junit output with arbitrary subtasks within a test. The current aim is, for handover tests, to not just show that a test failed, but to show exactly which steps worked and which didn't, e.g.: handover.py/01_bts0_started PASSED handover.py/02.1_ms0_attach PASSED handover.py/02.2_ms1_attach PASSED handover.py/02.3_subscribed_in_msc PASSED handover.py/03_call_established PASSED handover.py/04.1_bts1_started FAILED In this case it is immediately obvious from looking at the jenkins results analyzer that bts1 is the cause of the test failure, and it is visible which parts of the test are flaky, over time. First user Will be the upcoming handover_2G suite, in I0b2671304165a1aaae2b386af46fbd8b098e3bd8. Change-Id: I4ca9100b6f8db24d1f7e0a09b3b7ba88b8ae3b59
2020-12-11do not redirect_stdout in test scriptsNeels Hofmeyr1-16/+16
In a test, I called print() on a multi-line string and saw the log showing each line 0.2 seconds apart. redirect.stdout seems to be pretty inefficient. Instead, put a print() function into the testenv, to directly call log() on the strings passed to print(). The initial idea for redirect_stdout was that we could print() in any deeper functions called from a test script. But we have no such nested print() anywhere, only in test scripts themselves. As a result of this, a multi-line print() in test scripts now no longer puts the log prefix (timestamp, test name...) and suffix (backtrace / source position) to each single line, but prints the multiline block between a single log prefix and suffix -- exactly like the log() function does everywhere else. I actually briefly implemented adding the log prefix to each separate line everywhere, but decided that it is not a good idea: in some places we log config file snippets and other lists, and prepending the log prefix to each line makes pasting such a snippet from (say) a jenkins log super cumbersome. And the log prefix (backtrace) attached on each separate line makes multiline blocks very noisy, unreadable. Change-Id: I0972c66b9165bd7f2b0b387e0335172849199193
2020-12-11typo / newline in selftest log outputNeels Hofmeyr4-6/+4
...because it otherwise causes confusing selftest log changes in upcoming I0972c66b9165bd7f2b0b387e0335172849199193 Change-Id: I1eec31c5baba5ef33d0c8441b1e94a00d79af9bf
2020-12-11fix: line nr in test name in wrong placesNeels Hofmeyr1-6/+6
test.Test() overrides name() in order to provide source line number information. However, overriding name() is the wrong place for that, as name() is also often used for identifying an object - when listing the tests of a suite, the line number should not appear in the test name. For example, the line number sometimes ends up in the test results in jenkins, making 'foo.py' and 'foo.py:23' two distinct report items. Instead, add a separate function Origin.src() that defaults to name(), but specific classes can override src() if they wish to provide more detailed information with the object name. Override src() in Test, not name(). Use src() in backtraces. The suite_test.ok shows that the backtracing in the log remains unchanged, but the place where the test name is printed is corrected: I am 'test_suite' / 'hello_world.py:23' becomes I am 'test_suite' / 'hello_world.py' (Notice that "[LINENR]" in suite_test.ok is a masking of an actual number, done within the selftest suite) Change-Id: I0c4698fa2b3db3de777d8b6dcdcee84e433c62b7
2020-12-11configure fixed ARFCN per btsNeels Hofmeyr4-114/+2
Remove ARFCNs as a concept from resource pool, assign a fixed ARFCN to each BTS and TRX in the resource pools. Using ARFCNs on specific bands as resources was an idea that is hard to implement, because specific BTS dictate selection of bands which influences which ARFCNs can be picked. That means reserving ARFCN resources is only possible after reserving specific BTS resources, but the tester is currently not capable of such two-stage resolution. Writing handover tests, I got the problem that both BTS in a scenario attempt to use the same ARFCN. The by far easiest solution is to assign one fixed ARFCN to each BTS and TRX. If ever needed, a scenario modifier can still configure different ARFCNs. (Due to uncertainty about OC2G operation stability, I prefer to leave OC2G on ARFCN 50, as it happened to end up being configured before this patch.) Change-Id: I0a6c60544226f4261f9106013478d6a27fc39f38
2020-12-11log reserved resourcesNeels Hofmeyr1-0/+53
On non-debug log level, show something like this at the beginning of each suite: 03:45:49.439720 tst handover:sysmo+secondbts-trx-b200: RESERVED RESOURCES for handover: bts sysmoBTS 1002 Ettus B200 ip_address 10.42.42.2 10.42.42.3 10.42.42.4 10.42.42.5 10.42.42.6 10.42.42.7 modem sierra_1st sierra_2nd Change-Id: Ic23556eafee654c93d13c5ef405028da09bd51d7
2020-12-10cfg templates: log level and source basename last, log no category-hexNeels Hofmeyr1-0/+6
Makes for saner log output in stderr files. Change-Id: I2755beffd18d6779367c04e4aaaed119d66eb3be
2020-11-11resource: Support waiting for reserved resources until availablePau Espin Pedrol3-0/+380
Before this patch, almost everything was in place to support concurrent osmo-gsm-tester instances sharing a common state dir. However, during resource reservation, if the reservation couldn't be done due to too many resources being in use, osmo-gsm-tester would fail and skip the test suite. With this patch, OGT will wait until some reserved resources are released and then try requesting the reservation again. Change-Id: I938602ee890712fda82fd3f812d8edb1bcd05e08
2020-06-15Add per-test KPI supportPau Espin Pedrol3-3/+48
tests can now use 'tenv.test().set_kpis(some_dict)' to set any kind of data as KPIs, which will be presented in the junit report. The representation of KPIs in the xml file doesn't follow the junit format, mainly because it has no support for per-test properties. Change-Id: I00e976f65a202e82d440bf33708f06c8ce2643e2
2020-06-15selftest: Add test to verify junit xml reportPau Espin Pedrol8-0/+216
Change-Id: I8cad02abe776cc00b513113dbaf3c948ea7956cd
2020-06-15Implement per-test timeout guardPau Espin Pedrol4-16/+49
Timeout value can be specified by test in suite.conf: config: suite: <suite_name>: <test_name>: timeout: 2 # 2 seconds timeout Change-Id: I522f51f77f8be64ebfdb5d5e07ba92baf82d7706
2020-06-12Refactor test specific config parsing & generationPau Espin Pedrol1-5/+16
The dictionary is generated once during test creation at suite startup. Change-Id: I4de511d64ccf65d2ec6eec45dcf28529b74c5d26
2020-06-12Remove all references to defaults: timeout: from suite.confPau Espin Pedrol4-8/+0
This feature is not really implemented and maybe never was. In any case, it makes sense to have that working per-test so we can specify different values per test in case it's needed. Change-Id: I3c1b95c10e974da87ec9abd25578d8bcc0bc55a3
2020-06-11testenv: Add debug log when overlaying templates dirPau Espin Pedrol1-0/+3
Change-Id: I10fdfc5b2e3bf4c96941a3fae53230131f384de9
2020-06-04testenv: Support test overlaying a directory to look for templatesPau Espin Pedrol6-0/+168
This way tests which require a very specific config file can override specific template files used by object classes. Change-Id: I65d1b1e826d2d430ee83810d998b98d0ccaa07cd
2020-06-04selftest: template: Test existing overlay dir featuresPau Espin Pedrol5-82/+209
Change-Id: Iff1a4dafcbf2193e95b1fb6e5a6712f1bcdd6309
2020-06-04testenv: Get rid of 'resources' global variablePau Espin Pedrol2-13/+13
Change-Id: Id7576d21d0f9de29dd9bc4211eaf9b9ec634c990
2020-05-28Support identifying different tgz files based on run_label attributePau Espin Pedrol7-2/+29
tgz files in trials can be categorized in subdirectories, allowing to select different bianry files at runtime based on the target run node which is going to run them. This way for instance one can have a binary linked against libs for eg. CentOS under run_label "centos/" or an ARM target under "arm", and then use "run_label: arm" on the resource using it. Change-Id: Iaf2e97da3aff693395f44f0e93b184d4846cf6da
2020-05-25config: suites_dir and scenarios_dir are now a list of pathsPau Espin Pedrol17-15/+25
This allows inheriting suites or scenarios from eg. sysmocom/ dir, while still allowing to apply new suites and scenarios on top. Change-Id: Icecdae32d400a6b6da2ebf167c1c795f7a74ae96
2020-05-21schema: Allow objects registering their own schema typesPau Espin Pedrol3-0/+53
Change-Id: I998c8674a55531909bfeac420064c3f238cea126
2020-05-12config: Allow setting trial directory in main.confPau Espin Pedrol3-3/+6
Change-Id: Ia4141001d084f690897dbdff5eae6c69ff2e521c
2020-05-12Cmdline arg -c sets main configuration file (old paths.conf) instead of dir ↵Pau Espin Pedrol6-42/+27
containing it It has been notified that current configuration system is difficult to understand and to use, so it has been envisioned to refactor it a bit. The idea is that the user passes a -c path/to/main.conf file, which in turn contains whatever osmo-gsm-tester main settings supports (basically what old paths.conf used to be, plus some files harcoded to the same -c directory are now configurable through the main configuration file). Change-Id: Ieca65b71b543c44cfcec8e83efd0fe053c432e55
2020-05-11selftest: Introduce scenario_testPau Espin Pedrol10-0/+178
This covers some unit tests for scenario module (Scenario class). Change-Id: I4c80047bb03ae8254c192057007fa7df84478605
2020-05-11Split Scenario class to its own filePau Espin Pedrol1-14/+18
Change-Id: Ia029de7ecda4c8dc3d0b4c412e4c9c0a65cf0185
2020-05-07Deprecate envvar OSMO_GSM_TESTER_CONF and remove other onesPau Espin Pedrol2-2/+2
Environment variable OSMO_GSM_TESTER_CONF is marked as deprecated and an error is logged each time it's used. Same feature is available through "-c" command line parameters, so having the envvar only makes things more complex for no good reason. It cannot yet be completely dropped since some environemnt still make use of it. Give some time to users to adapt their setups. Other environment variables setting some config apths can be dropped since they are not being really used in any setup. Change-Id: I7eb69f870d0dcb5906d45ae067d6bed1aabf5862
2020-05-07Allow suites to dynamically register schemas so tests can receive parametersPau Espin Pedrol13-14/+440
Change-Id: Idbe99a35993d193cd97059feb980e61ff14c67ad
2020-05-07Makefile: Implement clean targetPau Espin Pedrol1-0/+5
Change-Id: Ia21b0f184bfd454b3835774949f581a27860a80a
2020-05-06selftest: Move tests into own subdirectoriesPau Espin Pedrol52-17/+37
Change-Id: Id21cda19df131e24402e1a593036e1e33a914920
2020-05-06selftest: Remove unused misc.py filePau Espin Pedrol1-7/+0
Change-Id: I167bce29d7b22956c82053dcae5634c23d405b3f
2020-05-06selftest/process_test: Fix output changing in new python versionPau Espin Pedrol2-6/+9
On older versions apparently upon child termination due to SIGINT subprocess.poll() returned 1. On new python versions (such as 3.8.2), -2 is returned, according to documentation: A negative value -N indicates that the child was terminated by signal N (Unix only). Let's catch the SIGINT in the child process and exit with a known 42 value to fix different behavior. Change-Id: I7949ff2b435e91e890061e6840b0f411f8b0a817
2020-05-06selftest/trial_test: Fix repr(RuntimeError) changing in new python versionPau Espin Pedrol4-6/+6
If using python 3.8.2, the trailing comma at the end of parameters is not longer there (probably got fixed, since it's not needed). That change breaks expected output. Change-Id: I6da3024f946c0e761099058e812c0eacf3d6071f
2020-05-05tests: Replace 'suite' with 'tenv'Pau Espin Pedrol3-3/+3
Change-Id: I777ec59ffd83b861ddc09e85e52d7a8896264b8c
2020-05-05Move trial.py and suite.py to core/Pau Espin Pedrol2-2/+2
Change-Id: Ic9a4c90c190e9d8e1829fbef83a9243f3c6f655b
2020-05-05Move all obj/ references in suite.py to testenv.pyPau Espin Pedrol3-3/+3
Change-Id: If4ab39be7a97d33e82c5a34e2a10dfec38613a4e
2020-05-04Move test.py and report.py to core/Pau Espin Pedrol1-2/+2
Change-Id: Ibb7fa5ab40bcf1e59705bdd2c2c5a76025b2b544
2020-05-04Move resource.py to core/Pau Espin Pedrol1-2/+1
Change-Id: I0f36a5e61ff0ef14d08165c9c184acdf2d6e6901
2020-05-04Generate schemas dynamically from pieces provided by each object classPau Espin Pedrol3-14/+22
This way we benefit from: * knowing which attributes are used/required by each object class and subclass * Having validation function definitions near the class going to use them Change-Id: I8fd6773c51d19405a585977af4ed72cad2b21db1
2020-04-11Move object classes used by tests into obj/ subdirPau Espin Pedrol1-1/+1
Change-Id: I0b2963cf00acd1f9036dca2e87aa3b87a0f8066d
2020-04-11Create core directory to contain most of code not in object classes used by ↵Pau Espin Pedrol10-13/+13
tests Change-Id: I9aec6c55ccd71894182057d36e0025b69925d314
2019-09-18suite_test: suite requires the fake trial to implement get_run_dir()Pau Espin Pedrol1-5/+24
If class Test calls get_run_dir(), it will generate the run dir from Suite.get_run_dir() + append, and Suite.get_run_dir() in turn will do the same with Trial.get_run_dir(). In suite_test.py, however, we are passing a plain log.Origin object (parent of Trial), which doesn't implement this function. We need to add it since tests will start using get_run_dir() to write their own log files. Generated tmp files are deleted on success and left there on error, in order to be able to debug stuff easily. Change-Id: Ic8326659a9ced0351e1e7e57ef21ce36f777729a
2019-04-04process: Early return during process termination if no proc runningPau Espin Pedrol1-56/+0
This avoids extra unneeded logging about killing with signal when actually no signal is being sent. Change-Id: I5b5dd78fe3301d8eef2ab93da3b37029268ae198
2019-03-05process: Make killing processes non-sequentialHolger Hans Peter Freyther1-0/+56
Change-Id: Icf1ac6774ea11880542012fd6c6ac73302bb74f5
2018-11-21templates: bsc/nitb: Remove explicit no force-combined-siPau Espin Pedrol1-2/+0
It's always been the default anyway, so no need to explicitly state it. Furthermore, recently (osmobsc 167cb828665b91ed7d28007abd4a1862ebbb78c9) force-combined-si was enabled by default for nanoBTS due to issues with some MS not registering when it's not enabled, so let's osmo-bsc decide were should it be enabled. Related: OS#3063 Change-Id: I97b10406a4a0048c66a47aa7d32a113f0c0366d1
2018-10-03templates: Update .cfg files to use logging level set-allPau Espin Pedrol2-2/+2
VTY cmd "logging level all" is deprecated in favour of "logging level set-all", but the required behaviour in this case is "logging level set-all". Change-Id: Ice54fca6be86f05a02ce4ae783e6e42e6958b95d
2018-08-29osmo-bts-trx: Big refactor and cleanup of osmo-trx related codePau Espin Pedrol4-35/+82
* This commit is a preparation for future commits to add support for different osmo-trx devices and backends like osmo-trx-lms. * Drop deprecated osmo-trx-* cmd line params and use VTY cfg to set them. * As number of osmo-trx related osmo-gsm-tester attributes grow, group them togther in an "osmo_trx" dictionary. Change-Id: I77d29413c9e3b600b796627ba366f80c3281b7e1
2018-08-21Introduce scenario modifiersPau Espin Pedrol5-2/+199
Before this patch, scenarios were only used to select resources with specific attributes. This commit introduces "modifiers" in scenarios, which allows setting or modifing config attributes of resources once they have been reserved. This way same test can be run selecting same resources but modifying its configuration, allowing for instance running different number of TRX, different timeslot configuration, etc. Modifiers are described by placing a "modifiers" dictionary in any scenario file, similar to the current "resources" one used to select requird resources. The "modifiers" dictionary is overlaid on top of the "resources" one resulting from combining all the "resources" dictionary of all scenario files. Change-Id: If8c422c67d9a971d9ce2c72594f55cde2db7550d
2018-04-04ms: Create a cumulative distribution function classHolger Hans Peter Freyther2-0/+132
We are using the CDF to decide which percentage of the jobs should be running at a given point. The x-axis is time and the y-axis the percentage of how many jobs should be running. There are three functions to do this. The first one is a constant which would result in everything being started right now, one to start them linearly and the last (formula from Qt/3rdparty) to first accelerate and decelerate slowly. Change-Id: I9e3064f4c3c4c7af5d3491f850090516e541f4d3
2017-12-18cosmetic: adjust resource_test to comp128v1Neels Hofmeyr2-20/+20
Since libosmocore doesn't support XOR at all, it seems weird to use it in the test, even though it's just a selftest without libosmocore involved... Change-Id: I51edec255e7ef277907817b3187c2f492465467f
2017-12-05bts_*: Add incrementing bvci and rac valuesPau Espin Pedrol2-8/+10
Change-Id: Ie65d7d33322c775a1040b5c81f367d882a7c2c5b
generated by cgit v1.2.1 (git 2.18.0) at 2021-03-05 11:18:34 +0000