path: root/selftest
AgeCommit message (Collapse)AuthorFilesLines
2017-07-20remove timer section from osmo-{bts,bsc}.cfg.tmplHarald Welte2-22/+0
As described in https://osmocom.org/issues/2380 and related gerrit patch https://gerrit.osmocom.org/#/c/3306 the config files currently contain timers with a 'zero' value, leading to issues. The config file templates used in osmo-gsm-tester suffer actually from exactly the same issue: They basically contain values that were written back to the file based on the default initialization in libbsc, which unfortunately was zero in some cases. Let's remove those values, they are the (bad) default anyway. libbsc patches in the pipeline will introduce reasonable defaults and forbid the use of zero for timers. Change-Id: I8cde29a597a17c2659b3b87268be4b12975f4bef
2017-07-03Add microsecond accuracy to log timestampsPau Espin Pedrol1-1/+1
This is useful while debugging and trying to check events across other outputs such as pcap files, process logs, etc. Change-Id: I43bb5c6e9977189251802bc2b078c52eb046bab8
2017-06-18Add support for SMPP testingPau Espin Pedrol3-10/+32
As defined in [1], the different related actors are implemented in this commit: ESME and SMSC. SMSC: In Osmocom, the SMSC is currently implemented inside the NITB or the MSC. A new Smsc abstract class is created to shared code between the NITB and the MSC, and also makes it easier for later when the SMSC is splitted. ESMEs can be dynamically added to its configuration in a similar way to how the BTSs are added. ESME: A new class Esme is created which can be used by tests to control an ESME to interact with the SMSC. The ESME functionalities are implemented using python-smpplib. Required version of this library is at least 43cc6f819ec76b2c0a9d36d1d439308634716227, which contains support for python 3 and some required features to poll the socket. This commit already contains a few tests which checks different features and tests the API. Extending tested features or scenarios can be later done quite easily. The tests are not enabled by default right now, because there are several of them in a suite and the ip_address resources are not freed after every tests which ends up in the suite failing due to missing reserved resources. All the tests run alone work though. When the issue is fixed they can then be added to the default list of tests to be run. [1] http://opensmpp.org/specs/SMPP_v3_4_Issue1_2.pdf Change-Id: I14ca3cb009d6d646a449ca99b0200da12085c0da
2017-06-13log_test.py: cosmetic follow-upNeels Hofmeyr2-18/+15
This is kept separate to not clutter up previous patch I5f9b53150f2bb6fa9d63ce27f0806f0ca6a45e90. Change-Id: I0ce50375fdb028da96c2159d577d8ed1967d4fe6
2017-06-13fix and refactor logging: drop 'with', simplifyNeels Hofmeyr8-77/+66
With the recent fix of the junit report related issues, another issue arose: the 'with log.Origin' was changed to disallow __enter__ing an object twice to fix problems, now still code would fail because it tries to do 'with' on the same object twice. The only reason is to ensure that logging is associated with a given object. Instead of complicating even more, implement differently. Refactor logging to simplify use: drop the 'with Origin' style completely, and instead use the python stack to determine which objects are created by which, and which object to associate a log statement with. The new way: we rely on the convention that each class instance has a local 'self' referencing the object instance. If we need to find an origin as a new object's parent, or to associate a log message with, we traverse each stack frame, fetching the first local 'self' object that is a log.Origin class instance. How to use: Simply call log.log() anywhere, and it finds an Origin object to log for, from the stack. Alternatively call self.log() for any Origin() object to skip the lookup. Create classes as child class of log.Origin and make sure to call super().__init__(category, name). This constructor will magically find a parent Origin on the stack. When an exception happens, we first escalate the exception up through call scopes to where ever it is handled by log.log_exn(). This then finds an Origin object in the traceback's stack frames, no need to nest in 'with' scopes. Hence the 'with log.Origin' now "happens implicitly", we can write pure natural python code, no more hassles with scope ordering. Furthermore, any frame can place additional logging information in a frame by calling log.ctx(). This is automatically inserted in the ancestry associated with a log statement / exception. Change-Id: I5f9b53150f2bb6fa9d63ce27f0806f0ca6a45e90
2017-06-09refactor: fix error handling; fix log.Origin; only one trialNeels Hofmeyr6-46/+98
A bit of refactoring to fix logging and error reporting, and simplify the code. This transmogrifies some of the things committed in 0ffb41440661631fa1d520c152be4cf8ebd4c46b "Add JUnit XML reports; refactor test reporting", which did not fully match the code structuring ideas used in osmo-gsm-tester. Also solve some problems present from the start of the code base. Though this is a bit of a code bomb, it would take a lot of time to separate this into smaller bits: these changes are closely related and resulted incrementally from testing error handling and logging details. I hope it's ok. Things changed / problems fixed: Allow only a single trial to be run per cmdline invocation: unbloat trial and suite invocation in osmo-gsm-tester.py. There is a SuiteDefinition, intended to be immutable, and a mutable SuiteRun. SuiteDefinition had a list of tests, which was modified by the SuiteRun to record test results. Instead, have only the test basenames in the SuiteDefinition and create a new set of Test() instances for each SuiteRun, to ensure that no state leaks between separate suite runs. State leaking across runs can be seen in http://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run/453/ where an earlier sms test for sysmo succeeds, but its state gets overwritten by the later sms test for trx that fails. The end result is that both tests failed, although the first run was successful. Fix a problem with Origin: log.Origin allowed to be __enter__ed more than once, skipping the second entry. The problem there is that we'd still __exit__ twice or more, popping the Origin off the stack even though it should still remain. We could count __enter__ recurrences, but instead, completely disallow entering a second time. A code path should have one 'with' statement per object, at pivotal points like run_suites or run_tests. Individual utility functions should not do 'with' on a central object. The structure needed is, in pseudo code: try: with trial: try: with suite_run: try: with test: test_actions() The 'with' needs to be inside the 'try', so that the exception can be handled in __exit__ before it reaches the exception logging. To clarify this, like test exceptions caught in Test.run(), also move suite exception handling from Trial into SuiteRun.run_tests(). There are 'with self' in Test.run() and SuiteRun.run_tests(), which are well placed, because these are pivotal points in the main code path. Log output: clearly separate logging of distinct suites and test scripts, by adding more large_separator() calls at the start of each test. Place these separator calls in more logical places. Add separator size and spacing args. Log output: print tracebacks only once, for the test script where they happen. Have less state that duplicates other state: drop SuiteRun.test_failed_ctr and suite.test_skipped_ctr, instead add SuiteRun.count_test_results(). For test failure reporting, store the traceback text in a separate member var. In the text report, apply above changes and unclutter to achieve a brief and easy to read result overview: print less filler characters, drop the starting times, drop the tracebacks. This can be found in the individual test logs. Because the tracebacks are no longer in the text report, the suite_test.py can just print the reports and expect that output instead of asserting individual contents. In the text report, print duration in precision of .1 seconds. Add origin information and a traceback text to the junit XML result to give more context when browsing the result XML. For 'AssertionError', add the source line of where the assertion hit. Drop the explicit Failure exception. We don't need one specific exception to mark a failure, instead any arbitrary exception is treated as a failure. Use the exception's class name as fail_type. Though my original idea was to use raising exceptions as the only way to cause a test failure, I'm keeping the set_fail() function as an alternative way, because it allows test specific cleanup and may come in handy later. To have both ways integrate seamlessly, shift some result setting into 'finally' clauses and make sure higher levels (suite, trial) count the contained items' stati. Minor tweak: write the 'pass' and 'skip' reports in lower case so that the 'FAIL' stands out. Minor tweak: pass the return code that the program exit should return further outward, so that the exit(1) call does not cause a SystemExit exception to be logged. The aims of this patch are: - Logs are readable so that it is clear which logging belongs to which test and suite. - The logging origins are correct (vs. parents gone missing as previously) - A single test error does not cause following tests or suites to be skipped. - An exception "above" Exception, i.e. SystemExit and the like, *does* immediately abort all tests and suites, and the results for tests that were not run are reported as "unknown" (rather than skipped on purpose): - Raising a SystemExit aborts all. - Hitting ctrl-c aborts all. - The resulting summary in the log is brief and readable. Change-Id: Ibf0846d457cab26f54c25e6906a8bb304724e2d8
2017-06-07error log: clarify for reserving more resources than availableNeels Hofmeyr1-1/+1
When trying to reserve more resources than available in the resources.conf, actually print an intelligible error message: catch the nameless error from solve() and fill in what was requested for solution. Change-Id: Iba3707f1aaeb40a58c616c33af52a60c9a2e7e1f
2017-06-07fix problem in origin ancestry: don't add self twiceNeels Hofmeyr2-13/+10
Also add various comments to illustrate what is going on during origin resolution. In the regression tests' expectations, some duplicate entries in the origins are removed, and hence no list of deeper origin ancestry is printed anymore. Change-Id: I42c3b8635b54c31c27699140e200c1f75a6ada29
2017-06-03Re-License under GPLv3-or-later instead of AGPLv3-or-laterHarald Welte1-3/+3
The "Affero" nature makes sense for the Osmocom network components like BSC, SGSN, etc. as they are typically operated to provide a network service. For testing, this doesn't make so much sense as it is difficult to imagine people creating a business out of offering to run test cases on an end-to-end Osmocom GSM network. So let's drop the 'Affero' here. All code is so far developed by sysmocom staff, so as Managing Director of sysmocom I can effect such a license change unilaterally. Change-Id: I8959c2d605854ffdc21cb29c0fe0e715685c4c05
2017-05-31sms_test: add output checksNeels Hofmeyr2-0/+6
Change-Id: I4b033dc0d3d43825901308cda05c6999946b664e
2017-05-31fix: Sms.__eq__ bug due to typoNeels Hofmeyr1-0/+2
Add test of this to sms_test.py Change-Id: Ib0b420a81c1b211a9f03a2521d244b3a427d5e5b
2017-05-31Move Sms class to a separate modulePau Espin Pedrol1-12/+12
Preparation for following commits to add smpp support, as we will have a class SmppClient with a method accepting an Sms object to send it. Change-Id: I1f28e14e963abb64df687b69d54975be2aeb0d0d
2017-05-30cosmetic: review mo_mt_sms.py's loggingNeels Hofmeyr1-1/+1
Implement the Modem.log_info() function, use that instead of logging all modem properties. Tweak mo_mt_sms.py print() statements. Pass modem object to SMS generation to inlcude the modem name as SMS token. Change-Id: I2b17fce0b3b05594fd9038b54e5b65f5127bd0a4
2017-05-29config: nitb template: move ip_address to nitb.ip_addressNeels Hofmeyr2-6/+6
It's the NITB's address, so it should go in the nitb.* scope. Change-Id: I71a5ef153b7156b0644253f5aa8a0c848f42ab3b
2017-05-29rename resource nitb_iface to ip_addressNeels Hofmeyr10-34/+34
I would like to use the IP addresses also for OsmoBSC processes, so it is more than clear now that 'nitb_iface' was the wrong naming choice. The only distinction we may need in the future is public versus loopback interface. To add that, we may add a trait to the 'ip_address' resource like: ip_address: - addr: type: public - addr: type: loopback This way we can substitute public vs loopback addresses flexibly (e.g. using scenarios). Change-Id: I3ad583ae7a33f7a7bb56fe78a125f73c56a0e860
2017-05-29nitb config: set: logging color 1, category 1Neels Hofmeyr1-2/+2
Change-Id: Icd9022f5732caac65d16fc49f2ed7833199523de
2017-05-24trial.py: Remove accidental double creation of filePau Espin Pedrol1-1/+1
This file is already created in run_suites(). Creating it in here too makes a new file name()_2.xml created and populated instead of the original name().xml one. Change-Id: Ibec14a1ef5fd029aa37ce666d00236bb872d636e
2017-05-19Add JUnit XML reports; refactor test reportingPau Espin Pedrol5-9/+56
* Add Junit output file support * Differentiate between an expected failure test and an error in the test, as described in JUnit. * In case of an error/exception during test, record and attach it to the Test object and continue running the tests, and show it at the end during the trial report. Change-Id: Iedf6d912b3cce3333a187a4ac6d5c6b70fe9d5c5
2017-05-15trial: always use abspathNeels Hofmeyr2-2/+3
No actual failure known, but using the abspath will prevent further problems when running a binary in its own run dir and passing trial paths to it. Adjust test expectations. Change-Id: Ia2ba876a42eef9122a92645edd1f28ddb56e6be3
2017-05-15log state dirNeels Hofmeyr1-1/+1
On the log output, clearly show which state dir is used. Since all runs should use the same state dir, this is important and/or reassuring to see. Change-Id: Ie92f1879a35d8bb399ee916b7ef72f9ee7d47409
2017-05-11cosmetic: lock test: add quotesNeels Hofmeyr2-8/+8
Mostly to clarify an empty lock in the end. Change-Id: If80960b331d30125e97878d52edba9b4a63e87c7
2017-05-11lock test: make more robust against timingNeels Hofmeyr2-8/+0
Remove the stdout printing from the lock_test_help.py, so that there is no problem with a failing test because one output ending up in stdout before the other. So only output the current lock from lock_test.sh. Change-Id: I7fe80a9de88ddee1d66ca5c692948fc9673e4230
2017-05-08selftest: suite_test: ignore line numbersNeels Hofmeyr2-12/+13
Change-Id: Ideab363753559ba0b49352dc9da03a27d7b0dadf
2017-05-08improve logging for resource allocationNeels Hofmeyr2-14/+17
Log on level 'log', more clearly show whether it's for reservation or actual use, show the origin that is asking for them. Change-Id: I3b78c7bdcaec90943900343c878099160f8d2f64
2017-05-08log: make 32 the default origin_width.Neels Hofmeyr2-3/+3
Change-Id: I1159395251332f3b1af3b3a322e7191559105faa
2017-05-08log.py: tweak LogTarget list, require explicit LogTargetNeels Hofmeyr1-1/+1
Rename log_sink to log_write_func to make it more clear. Remember the list of log targets as static member LogTarget.all_targets. Make each LogTarget instance register with the list of targets implicitly. No longer create a default log target, rather create one explicitly in osmo-gsm-tester.py. Change-Id: I5844485eaed536cb34b44bfe23dc635fe1982dcd
2017-05-04config: resolve real paths from symlinks, add paths debug loggingNeels Hofmeyr4-2/+18
Tweak test expectations to include the new debug logging. Go through the paths in alphabetical order to get deterministic logging output, so the test expectations always match. Change-Id: I11a905b2467cda691d9ccea30ae436bac96476c9
2017-05-04put the example suite in /example, not /selftest/real_suiteNeels Hofmeyr11-178/+0
Also drop the env file and tweak the README.txt Change-Id: Ieea274dfd6756498b760c18a5852398cfa396b50
2017-05-04Add remote user for RemoteProcressPau Espin Pedrol1-1/+1
Use it to set root user for SysmoBTS, otherwise if osmo-gsm-tester is run by another user it will fail to connect Change-Id: I67d4126fc75cb9c2d249c713cd6f14db1f1e21da
2017-05-04fix template_test after 8dbad8701f (tweak config)Neels Hofmeyr1-2/+2
2017-05-04lock_test: make more robustNeels Hofmeyr3-16/+41
Before, the test would rely on timeouts to work out. Instead, use marker files to indicate whether to wait longer. Firstly to signal a blocking process should end, secondly that it has indeed ended. Also use a mktemp tempdir instead of /tmp
2017-05-04nitb,sysmobts: tweak configNeels Hofmeyr1-3/+3
Set saner MCC+MNC, use more power.
2017-05-03resources: update modems, sierra_3 replaced by a GobiNeels Hofmeyr1-5/+5
Change-Id: I95832920a83ede29f314dfce23e1089729dcf3c7
2017-05-02cosmetic: rename to sms_was_received()Neels Hofmeyr1-1/+1
2017-05-02mo_mt_sms.py: remove short timeout; cosmetic: log propertiesNeels Hofmeyr1-1/+3
2017-05-02default-suites.conf: enable only sysmo for nowNeels Hofmeyr1-2/+0
2017-05-02resources.conf: configure sierra modemsNeels Hofmeyr1-79/+19
2017-04-27fix 'make check'Neels Hofmeyr7-11/+15
Apply various fixes that arose from test case code rot. These tests will now be used to verify patches submitted to gerrit, so they need to be up to par. Change-Id: I5277be0c434226d9d02e038f0bc72fd2557350c1 Related: OS#2215
2017-04-14rename default.conf to defaults.confNeels Hofmeyr1-0/+0
Change-Id: I0928ed306b0ca9b173dac092c77a9c6986714803
2017-04-13fix octphy, fix conf, improve loggingYour Name5-8/+9
Clearly separate the kinds of BTS hardware the GSM tester knows ('type') from the NITB's bts/type config item ('osmobsc_bts_type' -- not 'osmonitb_...' to stay in tune with future developments: it is the libbsc that needs this). For BTS hardware kinds, use the full name of the binary for osmo driven models: osmo-bts-sysmo, osmo-bts-trx, osmo-bts-octphy. Change-Id: I1aa9b48e74013a93f9db1a34730f17717fb3b36c
2017-04-12octphy (untested); fix regression test expectationsNeels Hofmeyr11-42/+227
Change-Id: Ie9986e0fe49171fb616ce92c3d8652002318f94f
2017-04-10trying to get sysmobts to work and various detailsNeels Hofmeyr5-12/+15
I know that these commit messages aren't very good, but the code is not stable yet, so I'm not bothering with details. Change-Id: I2d5e5f4a5407725d71093cbd71ef97b271eb8197
2017-04-09implement modem and nitb bits for a real testNeels Hofmeyr2-5/+24
Change-Id: I1ca3bd874aed1265d83a46340e9b3e469075c7ee
2017-04-08review jenkins scriptsNeels Hofmeyr1-0/+1
Change-Id: I420910bd2d30e96605ecf3acb779ce726c75d912
2017-04-08core implementationNeels Hofmeyr70-0/+2206
code bomb implementing the bulk of the osmo-gsm-tester Change-Id: I53610becbf643ed51b90cfd9debc6992fe211ec9