path: root/selftest
AgeCommit message (Collapse)AuthorFilesLines
2019-09-18suite_test: suite requires the fake trial to implement get_run_dir()Pau Espin Pedrol1-5/+24
If class Test calls get_run_dir(), it will generate the run dir from Suite.get_run_dir() + append, and Suite.get_run_dir() in turn will do the same with Trial.get_run_dir(). In, however, we are passing a plain log.Origin object (parent of Trial), which doesn't implement this function. We need to add it since tests will start using get_run_dir() to write their own log files. Generated tmp files are deleted on success and left there on error, in order to be able to debug stuff easily. Change-Id: Ic8326659a9ced0351e1e7e57ef21ce36f777729a
2019-04-04process: Early return during process termination if no proc runningPau Espin Pedrol1-56/+0
This avoids extra unneeded logging about killing with signal when actually no signal is being sent. Change-Id: I5b5dd78fe3301d8eef2ab93da3b37029268ae198
2019-03-05process: Make killing processes non-sequentialHolger Hans Peter Freyther1-0/+56
Change-Id: Icf1ac6774ea11880542012fd6c6ac73302bb74f5
2018-11-21templates: bsc/nitb: Remove explicit no force-combined-siPau Espin Pedrol1-2/+0
It's always been the default anyway, so no need to explicitly state it. Furthermore, recently (osmobsc 167cb828665b91ed7d28007abd4a1862ebbb78c9) force-combined-si was enabled by default for nanoBTS due to issues with some MS not registering when it's not enabled, so let's osmo-bsc decide were should it be enabled. Related: OS#3063 Change-Id: I97b10406a4a0048c66a47aa7d32a113f0c0366d1
2018-10-03templates: Update .cfg files to use logging level set-allPau Espin Pedrol2-2/+2
VTY cmd "logging level all" is deprecated in favour of "logging level set-all", but the required behaviour in this case is "logging level set-all". Change-Id: Ice54fca6be86f05a02ce4ae783e6e42e6958b95d
2018-08-29osmo-bts-trx: Big refactor and cleanup of osmo-trx related codePau Espin Pedrol4-35/+82
* This commit is a preparation for future commits to add support for different osmo-trx devices and backends like osmo-trx-lms. * Drop deprecated osmo-trx-* cmd line params and use VTY cfg to set them. * As number of osmo-trx related osmo-gsm-tester attributes grow, group them togther in an "osmo_trx" dictionary. Change-Id: I77d29413c9e3b600b796627ba366f80c3281b7e1
2018-08-21Introduce scenario modifiersPau Espin Pedrol5-2/+199
Before this patch, scenarios were only used to select resources with specific attributes. This commit introduces "modifiers" in scenarios, which allows setting or modifing config attributes of resources once they have been reserved. This way same test can be run selecting same resources but modifying its configuration, allowing for instance running different number of TRX, different timeslot configuration, etc. Modifiers are described by placing a "modifiers" dictionary in any scenario file, similar to the current "resources" one used to select requird resources. The "modifiers" dictionary is overlaid on top of the "resources" one resulting from combining all the "resources" dictionary of all scenario files. Change-Id: If8c422c67d9a971d9ce2c72594f55cde2db7550d
2018-04-04ms: Create a cumulative distribution function classHolger Hans Peter Freyther2-0/+132
We are using the CDF to decide which percentage of the jobs should be running at a given point. The x-axis is time and the y-axis the percentage of how many jobs should be running. There are three functions to do this. The first one is a constant which would result in everything being started right now, one to start them linearly and the last (formula from Qt/3rdparty) to first accelerate and decelerate slowly. Change-Id: I9e3064f4c3c4c7af5d3491f850090516e541f4d3
2017-12-18cosmetic: adjust resource_test to comp128v1Neels Hofmeyr2-20/+20
Since libosmocore doesn't support XOR at all, it seems weird to use it in the test, even though it's just a selftest without libosmocore involved... Change-Id: I51edec255e7ef277907817b3187c2f492465467f
2017-12-05bts_*: Add incrementing bvci and rac valuesPau Espin Pedrol2-8/+10
Change-Id: Ie65d7d33322c775a1040b5c81f367d882a7c2c5b
2017-12-05OsmoPcuSysmo: Integrate with Sysmobts and OsmoSgsnPau Espin Pedrol4-5/+64
Change-Id: I01485c5d74e5fe62d0ffea9eb1fad29041426eef
2017-11-17Move Test class to its own modulePau Espin Pedrol1-5/+5
Make the code more reachable for newcomers, as well as more organized for people who work a lot with it. SuiteRun in is already quite big, and having the Test class in there make it unnecessarily more big, and makes it difficult to find stuff. At the same time, having a which does actually not contain the Test class but other stuff, makes it even more confusing. Change-Id: I9c8d67f598466ba52a4827ff77027b9eae85929a
2017-11-17Rename test module to testenv and update referencesPau Espin Pedrol3-3/+3
Preparation to move current Test class into its own module test from suite module. Change-Id: I520bd046cb09042b5567d967f951f050e4e02e85
2017-11-08Use unique incrementing value for BTS CellIdPau Espin Pedrol3-3/+4
Change-Id: If4559b945a3e71f3a36fd7ac760cb094278f6b39
2017-09-16resource_test: Fix trailing whitespacePau Espin Pedrol1-1/+1
Change-Id: I654484b21ca2e950ff3019ee20133e9cb50296bb
2017-09-16resource_test: update to check list features (ciphers)Pau Espin Pedrol3-242/+187
Change-Id: I532f66585ab6f7be1a71853190791feeecbf263c
2017-09-16Use own format to specify encryption algorithmPau Espin Pedrol1-1/+1
... instead of using the one from from osmo vty directly. This way we avoid having multiple word attribute value and we can skip using quotes in the conf files. Change-Id: I5265cc9990dd5e99dba1f6262b3a8c597a3e958d
2017-09-16suite_test: Update tests to check new feature fixesPau Espin Pedrol3-51/+69
Now that combination and matching of lists works properly, update the suite test to check a more complex scenario. Change-Id: Ib0b82dc0c2a7f27cd6b2ffc32eed5f1287692ddc
2017-09-16resource: Handle lists correctly in item_matchesPau Espin Pedrol4-5/+41
We want to handle lists in the same way as we handle them in combine(). Without this commit, reserve()->find() failed to match objects containing dictionaries inside lists correctly (such as trx configs). A few attributes are added to trx_list of some resources in suite_test/resources.conf to show a case in which resource reservation would fail without this patch. It failed because before this patch, dictionaries inside lists are compared to be equal instead of being compared element by element to see if one dictionary is a subset of the other one (for each element in the lists). Change-Id: I8588d5b788b9f74a9cc84b8bdcb049921788bb48
2017-09-16config: Fix combination of listsPau Espin Pedrol2-0/+118
This commit fixes combination of resources containing lists. For lists containing complex types, it has been decided to handle them as sorted list, where position in list matters. In this case, combine is called recursively for each element in dest and src sharing position in the list, and assumes that if one list is shorter than the other, then it has to be combined against empty set for that tye. For instance this is useful when defining trx_list properties, where a BTS can have a different amount of TRX but we may be interested in restricting the first TRX and don't care about extra TRX. For lists containing simple types (eg. integers or strings), we just want to merge both lists and we only need to check if the value is already there, ie. handle them as unsortered sets. This case won't work if we call combine for each element of the list because for a simple case it will just end up checking if a[i] == b[i]. This kind of operation for simple types is needed in later commits where cipher attribute is introduced. Without this patch, having following 2 scenarios and trying them to use together "-s foosuite:cipher-a50+ciphera51" will fail: cipher_a50.conf: bts: - ciphers: - 'a5 0' cipher_a51.conf bts: - ciphers: - 'a5 1' ValueError: cannot combine dicts, conflicting items (values 'a5 0' and 'a5 1') Change-Id: Ib7a38f10eb9de338a77bf1fa3afceb9df1532015
2017-09-16Replicate resources based on times attr before combine timePau Espin Pedrol5-3/+184
As suite.conf and scenarios need to match 1-to-1 in lists, it's important to extend the dictionaries by replicating the objects with a 'times' values higher than 1 in order to match the objects correctly. Since dictionanries are expanded at combine time, there's no need to expand them during reserve() time because they are already expanded. As a result, this commit reworks the kind of schema applied in each place (and takes the change to start validating scenario files, which were neglected previously). Two unit tests are added as a show case. Unfortunately output showing scenario dictionaries needs to be ignored while verifying because it was encountered that different versions of python print dictionary elements in different order. Change-Id: I25eb639c7e3cf3b4c67a205422808bffbdd791e6
2017-09-16nominal_power and max_power_red attrs can now be set per resourcePau Espin Pedrol4-5/+37
Before this commit, only max_power_red was specified and it could only be used as a defaults and could not be set per object. Together with nominal_power, it can be useful to be able to set them to different values for different objects, as for example different osmo-bts-trx objects can require different values. Change-Id: I472742e98052cc39686d38c945be76d7f50eeebd
2017-09-14fix and lock_testNeels Hofmeyr3-1/+41
Do not pass os.O_TRUNC to open(), that immediately empties out the lock file. Fix lock_test to be able to catch this error. So far things were happening too fast to notice it. Caught by running 'make check' on a main unit APU that has different timing behavior. Change-Id: I872a3d2548ac84097ac7acf13cb12c36822e076e
2017-07-20remove timer section from osmo-{bts,bsc}.cfg.tmplHarald Welte2-22/+0
As described in and related gerrit patch the config files currently contain timers with a 'zero' value, leading to issues. The config file templates used in osmo-gsm-tester suffer actually from exactly the same issue: They basically contain values that were written back to the file based on the default initialization in libbsc, which unfortunately was zero in some cases. Let's remove those values, they are the (bad) default anyway. libbsc patches in the pipeline will introduce reasonable defaults and forbid the use of zero for timers. Change-Id: I8cde29a597a17c2659b3b87268be4b12975f4bef
2017-07-03Add microsecond accuracy to log timestampsPau Espin Pedrol1-1/+1
This is useful while debugging and trying to check events across other outputs such as pcap files, process logs, etc. Change-Id: I43bb5c6e9977189251802bc2b078c52eb046bab8
2017-06-18Add support for SMPP testingPau Espin Pedrol3-10/+32
As defined in [1], the different related actors are implemented in this commit: ESME and SMSC. SMSC: In Osmocom, the SMSC is currently implemented inside the NITB or the MSC. A new Smsc abstract class is created to shared code between the NITB and the MSC, and also makes it easier for later when the SMSC is splitted. ESMEs can be dynamically added to its configuration in a similar way to how the BTSs are added. ESME: A new class Esme is created which can be used by tests to control an ESME to interact with the SMSC. The ESME functionalities are implemented using python-smpplib. Required version of this library is at least 43cc6f819ec76b2c0a9d36d1d439308634716227, which contains support for python 3 and some required features to poll the socket. This commit already contains a few tests which checks different features and tests the API. Extending tested features or scenarios can be later done quite easily. The tests are not enabled by default right now, because there are several of them in a suite and the ip_address resources are not freed after every tests which ends up in the suite failing due to missing reserved resources. All the tests run alone work though. When the issue is fixed they can then be added to the default list of tests to be run. [1] Change-Id: I14ca3cb009d6d646a449ca99b0200da12085c0da cosmetic follow-upNeels Hofmeyr2-18/+15
This is kept separate to not clutter up previous patch I5f9b53150f2bb6fa9d63ce27f0806f0ca6a45e90. Change-Id: I0ce50375fdb028da96c2159d577d8ed1967d4fe6
2017-06-13fix and refactor logging: drop 'with', simplifyNeels Hofmeyr8-77/+66
With the recent fix of the junit report related issues, another issue arose: the 'with log.Origin' was changed to disallow __enter__ing an object twice to fix problems, now still code would fail because it tries to do 'with' on the same object twice. The only reason is to ensure that logging is associated with a given object. Instead of complicating even more, implement differently. Refactor logging to simplify use: drop the 'with Origin' style completely, and instead use the python stack to determine which objects are created by which, and which object to associate a log statement with. The new way: we rely on the convention that each class instance has a local 'self' referencing the object instance. If we need to find an origin as a new object's parent, or to associate a log message with, we traverse each stack frame, fetching the first local 'self' object that is a log.Origin class instance. How to use: Simply call log.log() anywhere, and it finds an Origin object to log for, from the stack. Alternatively call self.log() for any Origin() object to skip the lookup. Create classes as child class of log.Origin and make sure to call super().__init__(category, name). This constructor will magically find a parent Origin on the stack. When an exception happens, we first escalate the exception up through call scopes to where ever it is handled by log.log_exn(). This then finds an Origin object in the traceback's stack frames, no need to nest in 'with' scopes. Hence the 'with log.Origin' now "happens implicitly", we can write pure natural python code, no more hassles with scope ordering. Furthermore, any frame can place additional logging information in a frame by calling log.ctx(). This is automatically inserted in the ancestry associated with a log statement / exception. Change-Id: I5f9b53150f2bb6fa9d63ce27f0806f0ca6a45e90
2017-06-09refactor: fix error handling; fix log.Origin; only one trialNeels Hofmeyr6-46/+98
A bit of refactoring to fix logging and error reporting, and simplify the code. This transmogrifies some of the things committed in 0ffb41440661631fa1d520c152be4cf8ebd4c46b "Add JUnit XML reports; refactor test reporting", which did not fully match the code structuring ideas used in osmo-gsm-tester. Also solve some problems present from the start of the code base. Though this is a bit of a code bomb, it would take a lot of time to separate this into smaller bits: these changes are closely related and resulted incrementally from testing error handling and logging details. I hope it's ok. Things changed / problems fixed: Allow only a single trial to be run per cmdline invocation: unbloat trial and suite invocation in There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun. SuiteDefinition had a list of tests, which was modified by the SuiteRun to record test results. Instead, have only the test basenames in the SuiteDefinition and create a new set of Test() instances for each SuiteRun, to ensure that no state leaks between separate suite runs. State leaking across runs can be seen in where an earlier sms test for sysmo succeeds, but its state gets overwritten by the later sms test for trx that fails. The end result is that both tests failed, although the first run was successful. Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once, skipping the second entry. The problem there is that we'd still __exit__ twice or more, popping the Origin off the stack even though it should still remain. We could count __enter__ recurrences, but instead, completely disallow entering a second time. A code path should have one 'with' statement per object, at pivotal points like run_suites or run_tests. Individual utility functions should not do 'with' on a central object. The structure needed is, in pseudo code: try: with trial: try: with suite_run: try: with test: test_actions() The 'with' needs to be inside the 'try', so that the exception can be handled in __exit__ before it reaches the exception logging. To clarify this, like test exceptions caught in, also move suite exception handling from Trial into SuiteRun.run_tests(). There are 'with self' in and SuiteRun.run_tests(), which are well placed, because these are pivotal points in the main code path. Log output: clearly separate logging of distinct suites and test scripts, by adding more large_separator() calls at the start of each test. Place these separator calls in more logical places. Add separator size and spacing args. Log output: print tracebacks only once, for the test script where they happen. Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and suite.test_skipped_ctr, instead add SuiteRun.count_test_results(). For test failure reporting, store the traceback text in a separate member var. In the text report, apply above changes and unclutter to achieve a brief and easy to read result overview: print less filler characters, drop the starting times, drop the tracebacks. This can be found in the individual test logs. Because the tracebacks are no longer in the text report, the can just print the reports and expect that output instead of asserting individual contents. In the text report, print duration in precision of .1 seconds. Add origin information and a traceback text to the junit XML result to give more context when browsing the result XML. For 'AssertionError', add the source line of where the assertion hit. Drop the explicit Failure exception. We don't need one specific exception to mark a failure, instead any arbitrary exception is treated as a failure. Use the exception's class name as fail_type. Though my original idea was to use raising exceptions as the only way to cause a test failure, I'm keeping the set_fail() function as an alternative way, because it allows test specific cleanup and may come in handy later. To have both ways integrate seamlessly, shift some result setting into 'finally' clauses and make sure higher levels (suite, trial) count the contained items' stati. Minor tweak: write the 'pass' and 'skip' reports in lower case so that the 'FAIL' stands out. Minor tweak: pass the return code that the program exit should return further outward, so that the exit(1) call does not cause a SystemExit exception to be logged. The aims of this patch are: - Logs are readable so that it is clear which logging belongs to which test and suite. - The logging origins are correct (vs. parents gone missing as previously) - A single test error does not cause following tests or suites to be skipped. - An exception "above" Exception, i.e. SystemExit and the like, *does* immediately abort all tests and suites, and the results for tests that were not run are reported as "unknown" (rather than skipped on purpose): - Raising a SystemExit aborts all. - Hitting ctrl-c aborts all. - The resulting summary in the log is brief and readable. Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
2017-06-07error log: clarify for reserving more resources than availableNeels Hofmeyr1-1/+1
When trying to reserve more resources than available in the resources.conf, actually print an intelligible error message: catch the nameless error from solve() and fill in what was requested for solution. Change-Id: Iba3707f1aaeb40a58c616c33af52a60c9a2e7e1f
2017-06-07fix problem in origin ancestry: don't add self twiceNeels Hofmeyr2-13/+10
Also add various comments to illustrate what is going on during origin resolution. In the regression tests' expectations, some duplicate entries in the origins are removed, and hence no list of deeper origin ancestry is printed anymore. Change-Id: I42c3b8635b54c31c27699140e200c1f75a6ada29
2017-06-03Re-License under GPLv3-or-later instead of AGPLv3-or-laterHarald Welte1-3/+3
The "Affero" nature makes sense for the Osmocom network components like BSC, SGSN, etc. as they are typically operated to provide a network service. For testing, this doesn't make so much sense as it is difficult to imagine people creating a business out of offering to run test cases on an end-to-end Osmocom GSM network. So let's drop the 'Affero' here. All code is so far developed by sysmocom staff, so as Managing Director of sysmocom I can effect such a license change unilaterally. Change-Id: I8959c2d605854ffdc21cb29c0fe0e715685c4c05
2017-05-31sms_test: add output checksNeels Hofmeyr2-0/+6
Change-Id: I4b033dc0d3d43825901308cda05c6999946b664e
2017-05-31fix: Sms.__eq__ bug due to typoNeels Hofmeyr1-0/+2
Add test of this to Change-Id: Ib0b420a81c1b211a9f03a2521d244b3a427d5e5b
2017-05-31Move Sms class to a separate modulePau Espin Pedrol1-12/+12
Preparation for following commits to add smpp support, as we will have a class SmppClient with a method accepting an Sms object to send it. Change-Id: I1f28e14e963abb64df687b69d54975be2aeb0d0d
2017-05-30cosmetic: review's loggingNeels Hofmeyr1-1/+1
Implement the Modem.log_info() function, use that instead of logging all modem properties. Tweak print() statements. Pass modem object to SMS generation to inlcude the modem name as SMS token. Change-Id: I2b17fce0b3b05594fd9038b54e5b65f5127bd0a4
2017-05-29config: nitb template: move ip_address to nitb.ip_addressNeels Hofmeyr2-6/+6
It's the NITB's address, so it should go in the nitb.* scope. Change-Id: I71a5ef153b7156b0644253f5aa8a0c848f42ab3b
2017-05-29rename resource nitb_iface to ip_addressNeels Hofmeyr10-34/+34
I would like to use the IP addresses also for OsmoBSC processes, so it is more than clear now that 'nitb_iface' was the wrong naming choice. The only distinction we may need in the future is public versus loopback interface. To add that, we may add a trait to the 'ip_address' resource like: ip_address: - addr: type: public - addr: type: loopback This way we can substitute public vs loopback addresses flexibly (e.g. using scenarios). Change-Id: I3ad583ae7a33f7a7bb56fe78a125f73c56a0e860
2017-05-29nitb config: set: logging color 1, category 1Neels Hofmeyr1-2/+2
Change-Id: Icd9022f5732caac65d16fc49f2ed7833199523de Remove accidental double creation of filePau Espin Pedrol1-1/+1
This file is already created in run_suites(). Creating it in here too makes a new file name()_2.xml created and populated instead of the original name().xml one. Change-Id: Ibec14a1ef5fd029aa37ce666d00236bb872d636e
2017-05-19Add JUnit XML reports; refactor test reportingPau Espin Pedrol5-9/+56
* Add Junit output file support * Differentiate between an expected failure test and an error in the test, as described in JUnit. * In case of an error/exception during test, record and attach it to the Test object and continue running the tests, and show it at the end during the trial report. Change-Id: Iedf6d912b3cce3333a187a4ac6d5c6b70fe9d5c5
2017-05-15trial: always use abspathNeels Hofmeyr2-2/+3
No actual failure known, but using the abspath will prevent further problems when running a binary in its own run dir and passing trial paths to it. Adjust test expectations. Change-Id: Ia2ba876a42eef9122a92645edd1f28ddb56e6be3
2017-05-15log state dirNeels Hofmeyr1-1/+1
On the log output, clearly show which state dir is used. Since all runs should use the same state dir, this is important and/or reassuring to see. Change-Id: Ie92f1879a35d8bb399ee916b7ef72f9ee7d47409
2017-05-11cosmetic: lock test: add quotesNeels Hofmeyr2-8/+8
Mostly to clarify an empty lock in the end. Change-Id: If80960b331d30125e97878d52edba9b4a63e87c7
2017-05-11lock test: make more robust against timingNeels Hofmeyr2-8/+0
Remove the stdout printing from the, so that there is no problem with a failing test because one output ending up in stdout before the other. So only output the current lock from Change-Id: I7fe80a9de88ddee1d66ca5c692948fc9673e4230
2017-05-08selftest: suite_test: ignore line numbersNeels Hofmeyr2-12/+13
Change-Id: Ideab363753559ba0b49352dc9da03a27d7b0dadf
2017-05-08improve logging for resource allocationNeels Hofmeyr2-14/+17
Log on level 'log', more clearly show whether it's for reservation or actual use, show the origin that is asking for them. Change-Id: I3b78c7bdcaec90943900343c878099160f8d2f64
2017-05-08log: make 32 the default origin_width.Neels Hofmeyr2-3/+3
Change-Id: I1159395251332f3b1af3b3a322e7191559105faa tweak LogTarget list, require explicit LogTargetNeels Hofmeyr1-1/+1
Rename log_sink to log_write_func to make it more clear. Remember the list of log targets as static member LogTarget.all_targets. Make each LogTarget instance register with the list of targets implicitly. No longer create a default log target, rather create one explicitly in Change-Id: I5844485eaed536cb34b44bfe23dc635fe1982dcd
2017-05-04config: resolve real paths from symlinks, add paths debug loggingNeels Hofmeyr4-2/+18
Tweak test expectations to include the new debug logging. Go through the paths in alphabetical order to get deterministic logging output, so the test expectations always match. Change-Id: I11a905b2467cda691d9ccea30ae436bac96476c9