summaryrefslogtreecommitdiffstats
path: root/doc/manuals/chapters/install.adoc
blob: 112d4f921f6f87bfb1d960e533b5d5dfcfff16f8 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
== Installation on Main Unit

The main unit is a general purpose computer that orchestrates the tests. It
runs the core network components, controls the modems and so on. This can be
anything from a dedicated production rack unit to your laptop at home.

This manual will assume that tests are run from a jenkins build slave, by a user
named 'jenkins' that belong to group 'osmo-gsm-tester'. The user configuration
for manual test runs and/or a different user name is identical, simply replace
the user name or group.

=== Dependencies

On a Debian/Ubuntu based system, these commands install the packages needed to
run the osmo-gsm-tester.py code, i.e. install these on your main unit:

----
apt-get install \
  dbus \
  tcpdump \
  sqlite3 \
  python3 \
  python3-yaml \
  python3-mako \
  python3-gi \
  ofono \
  patchelf \
  sudo \
  libcap2-bin \
  python3-pip
pip3 install pydbus
pip3 install git+git://github.com/podshumok/python-smpplib.git
----

IMPORTANT: ofono may need to be installed from source to contain the most
recent fixes needed to operate your modems. This depends on the modem hardware
used and the tests run. Please see <<hardware_modems>>.

To run osmo-bts-trx with a USRP attached, you may need to install a UHD driver.
Please refer to http://osmocom.org/projects/osmotrx/wiki/OsmoTRX#UHD for
details; the following is an example for the B200 family USRP devices:

----
apt-get install libuhd-dev uhd-host
/usr/lib/uhd/utils/uhd_images_downloader.py
----

[[jenkins_deps]]
==== Jenkins Build Dependencies

Each of the jenkins builds requires individual dependencies. This is generally
the same as for building the software outside of osmo-gsm-tester and will not
be detailed here. For the Osmocom projects, refer to
http://osmocom.org/projects/cellular-infrastructure/wiki/Build_from_Source . Be
aware of specific requirements for BTS hardware: for example, the
osmo-bts-sysmo build needs the sysmoBTS SDK installed on the build slave, which
should match the installed sysmoBTS firmware.


[[configure_build_slave]]
=== Jenkins Build Slave

==== Create 'jenkins' User on Main Unit

On the main unit, create a jenkins user:

----
useradd -m jenkins
----

==== Install Java on Main Unit

To be able to launch the Jenkins build slave, a Java RE must be available on
the main unit. For example:

----
apt-get install default-jdk
----

==== Allow SSH Access from Jenkins Master

Create an SSH keypair to be used for login on the osmo-gsm-tester. This may be
entered on the jenkins web UI; alternatively, use the jenkins server's shell:

Login on the main jenkins server shell and create an SSH keypair, for example:

----
# su jenkins
$ mkdir -p /usr/local/jenkins/keys
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/jenkins/.ssh/id_rsa): /usr/local/jenkins/keys/osmo-gsm-tester-rnd
Enter passphrase (empty for no passphrase): <enter a passphrase>
Enter same passphrase again: <enter a passphrase>
Your identification has been saved in /usr/local/jenkins/keys/osmo-gsm-tester-rnd
Your public key has been saved in /usr/local/jenkins/keys/osmo-gsm-tester-rnd.pub.
The key fingerprint is:
...
----

Copy the public key to the main unit, e.g. copy-paste:

----
cat /usr/local/jenkins/keys/osmo-gsm-tester-rnd.pub
# copy this public key
----

On the main unit:

----
mkdir ~jenkins/.ssh
cat > ~jenkins/.ssh/authorized_keys
# paste above public key and hit Ctrl-D
chown -R jenkins: ~jenkins/.ssh
----

Make sure that the user running the jenkins master accepts the main unit's host
identification. There must be an actual RSA host key available in the
known_hosts file for the jenkins master to be able to log in. Simply calling
ssh and accepting the host key as usual is not enough. Jenkins may continue to
say "Host key verification failed".

To place an RSA host key in the jenkins' known_hosts file, you may do:

On the Jenkins master:

----
main_unit_ip=10.9.8.7
ssh-keyscan -H $main_unit_ip >> ~jenkins/.ssh/known_hosts
chown jenkins: ~jenkins/.ssh/known_hosts
----

Verify that the jenkins user on the Jenkins master has SSH access to the main
unit:

----
su jenkins
main_unit_ip=10.9.8.7
ssh -i /usr/local/jenkins/keys/osmo-gsm-tester-rnd jenkins@$main_unit_ip
exit
----

[[install_add_build_slave]]
==== Add Build Slave

In the jenkins web UI, add a new build slave for the osmo-gsm-tester:

* 'Manage Jenkins'
** 'Manage Nodes'
*** 'New Node'
**** Enter a node name, e.g. "osmo-gsm-tester-1" +
     (the "-1" is just some identification in case you'd like to add another
     setup later).
**** 'Permanent Agent'

Configure the node as:

* '# of executors': 1
* 'Remote root directory': "/home/jenkins"
* 'Labels': "osmo-gsm-tester" +
  (This is a general label common to all osmo-gsm-tester build slaves you may set up in the future.)
* 'Usage': 'Only build jobs with label expressions matching this node'
* 'Launch method': 'Launch slave agents via SSH'
** 'Host': your main unit's IP address
** 'Credentials': choose 'Add' / 'Jenkins'
*** 'Domain': 'Global credentials (unrestricted)'
*** 'Kind': 'SSH Username with private key'
*** 'Scope': 'Global'
*** 'Username': "jenkins" +
    (as created on the main unit above)
*** 'Private Key': 'From a file on Jenkins master'
**** 'File': "/usr/local/jenkins/keys/osmo-gsm-tester-rnd"
*** 'Passphrase': enter same passphrase as above
*** 'ID': "osmo-gsm-tester-1"
*** 'Name': "jenkins for SSH to osmo-gsm-tester-1"

The build slave should be able to start now.


==== Add Build Jobs

There are various jenkins-build-* scripts in osmo-gsm-tester/contrib/, which
can be called as jenkins build jobs to build and bundle binaries as artifacts,
to be run on the osmo-gsm-tester main unit and/or BTS hardware.

Be aware of the dependencies, as hinted at in <<jenkins_deps>>.

While the various binaries could technically be built on the osmo-gsm-tester
main unit, it is recommended to use a separate build slave, to take load off
of the main unit.

On your jenkins master, set up build jobs to call these scripts -- typically
one build job per script. Look in contrib/ and create one build job for each of
the BTS types you would like to test, as well as one for the 'build-osmo-nitb'.

These are generic steps to configure a jenkins build
job for each of these build scripts, by example of the
jenkins-build-osmo-nitb.sh script; all that differs to the other scripts is the
"osmo-nitb" part:

* 'Project name': "osmo-gsm-tester_build-osmo-nitb" +
  (Replace 'osmo-nitb' according to which build script this is for)
* 'Discard old builds' +
  Configure this to taste, for example:
** 'Max # of build to keep': "20"
* 'Restrict where this project can be run': Choose a build slave label that
  matches the main unit's architecture and distribution, typically a Debian
  system, e.g.: "linux_amd64_debian8"
* 'Source Code Management':
** 'Git'
*** 'Repository URL': "git://git.osmocom.org/osmo-gsm-tester"
*** 'Branch Specifier': "*/master"
*** 'Additional Behaviors'
**** 'Check out to a sub-directory': "osmo-gsm-tester"
* 'Build Triggers' +
  The decision on when to build is complex. Here are some examples:
** Once per day: +
   'Build periodically': "H H * * *"
** For the Osmocom project, the purpose is to verify our software changes.
   Hence we would like to test every time our code has changed:
*** We could add various git repositories to watch, and enable 'Poll SCM'.
*** On jenkins.osmocom.org, we have various jobs that build the master branches
    of their respective git repositories when a new change was merged. Here, we
    can thus trigger e.g. an osmo-nitb build for osmo-gsm-tester everytime the
    master build has run: +
    'Build after other projects are built': "OpenBSC"
*** Note that most of the Osmocom projects also need to be re-tested when their
    dependencies like libosmo* have changed. Triggering on all those changes
    typically causes more jenkins runs than necessary: for example, it rebuilds
    once per each dependency that has rebuilt due to one libosmocore change.
    There is so far no trivial way known to avoid this. It is indeed safest to
    rebuild more often.
* 'Build'
** 'Execute Shell'
+
----
#!/bin/sh
set -e -x
./osmo-gsm-tester/contrib/jenkins-build-osmo-nitb.sh
----
+
(Replace 'osmo-nitb' according to which build script this is for)

* 'Post-build Actions'
** 'Archive the artifacts': "*.tgz, *.md5" +
   (This step is important to be able to use the built binaries in the run job
   below.)


TIP: When you've created one build job, it is convenient to create further
build jobs by copying the first and, e.g., simply replacing all "osmo-nitb"
with "osmo-bts-trx".

==== Add Run Job

This is the build job that actually runs the tests on the GSM hardware:

* It sources the artifacts from the build jobs.
* It runs on the osmo-gsm-tester main unit's build slave.

Here is the configuration for the run job:

* 'Project name': "osmo-gsm-tester_run"
* 'Discard old builds' +
  Configure this to taste, for example:
** 'Max # of build to keep': "20"
* 'Restrict where this project can be run': "osmo-gsm-tester" +
  (to match the 'Label' configured in <<install_add_build_slave>>).
* 'Source Code Management':
** 'Git'
*** 'Repository URL': "git://git.osmocom.org/osmo-gsm-tester"
*** 'Branch Specifier': "*/master"
*** 'Additional Behaviors'
**** 'Check out to a sub-directory': "osmo-gsm-tester"
**** 'Clean before checkout'
* 'Build Triggers' +
  The decision on when to build is complex. For this run job, it is suggested
  to rebuild:
** after each of above build jobs that produced new artifacts: +
   'Build after other projects are built': "osmo-gsm-tester_build-osmo-nitb,
   osmo-gsm-tester_build-osmo-bts-sysmo, osmo-gsm-tester_build-osmo-bts-trx" +
   (Add each build job name you configured above)
** as well as once per day: +
   'Build periodically': "H H * * *"
** and, in addition, whenever the osmo-gsm-tester scripts have been modified: +
   'Poll SCM': "H/5 * * * *" +
   (i.e. look every five minutes whether the upstream git has changed)
* 'Build'
** Copy artifacts from each build job you have set up:
*** 'Copy artifacts from another project'
**** 'Project name': "osmo-gsm-tester_build-osmo-nitb"
**** 'Which build': 'Latest successful build'
**** enable 'Stable build only'
**** 'Artifacts to copy': "*.tgz, *.md5"
*** Add a separate similar 'Copy artifacts...' section for each build job you
    have set up.
** 'Execute Shell'
+
----
#!/bin/sh
set -e -x

# debug: provoke a failure
#export OSMO_GSM_TESTER_OPTS="-s debug -t fail"

PATH="$PWD/osmo-gsm-tester/src:$PATH" \
  ./osmo-gsm-tester/contrib/jenkins-run.sh
----
+
Details:

*** The 'jenkins-run.sh' script assumes to find the 'osmo-gsm-tester.py' in the
    '$PATH'. To use the most recent osmo-gsm-tester code here, we direct
    '$PATH' to the actual workspace checkout. This could also run from a sytem
    wide install, in which case you could omit the explicit PATH to
    "$PWD/osmo-gsm-tester/src".
*** This assumes that there are configuration files for osmo-gsm-tester placed
    on the system (see <<config_paths>>).
*** If you'd like to check the behavior of test failures, you can uncomment the
    line below "# debug" to produce a build failure on every run. Note that
    this test typically produces a quite empty run result, since it launches no
    NITB nor BTS.
* 'Post-build Actions'
** 'Archive the artifacts'
*** 'Files to archive': "*-run.tgz, *-bin.tgz" +
    This stores the complete test report with config files, logs, stdout/stderr
    output, pcaps as well as the binaries used for the test run in artifacts.
    This allows analysis of older builds, instead of only the most recent build
    (which cleans up the jenkins workspace every time). The 'trial-N-run.tgz'
    and 'trial-N-bin.tgz' archives are produced by the 'jenkins-run.sh' script,
    both for successful and failing runs.

=== Install osmo-gsm-tester on Main Unit

This assumes you have already created the jenkins user (see <<configure_build_slave>>).

==== User Permissions

On the main unit, create a group for all users that should be allowed to use
the osmo-gsm-tester, and add users (here 'jenkins') to this group.

----
groupadd osmo-gsm-tester
gpasswd -a jenkins osmo-gsm-tester
----

NOTE: you may also need to add users to the 'usrp' group, see
<<user_config_uhd>>.

A user added to a group needs to re-login for the group permissions to take
effect.

This group needs the following permissions:

===== Paths

Assuming that you are using the example config, prepare a system wide state
location in '/var/tmp':

----
mkdir -p /var/tmp/osmo-gsm-tester/state
chown -R :osmo-gsm-tester /var/tmp/osmo-gsm-tester
chmod -R g+rwxs /var/tmp/osmo-gsm-tester
setfacl -d -m group:osmo-gsm-tester:rwx /var/tmp/osmo-gsm-tester/state
----

IMPORTANT: the state directory needs to be shared between all users potentially
running the osmo-gsm-tester to resolve resource allocations. Above 'setfacl'
command sets the access control to keep all created files group writable.

With the jenkins build as described here, the trials will live in the build
slave's workspace. Other modes of operation (a daemon scheduling concurrent
runs, *TODO*) may use a system wide directory to manage trials to run:

----
mkdir -p /var/tmp/osmo-gsm-tester/trials
chown -R :osmo-gsm-tester /var/tmp/osmo-gsm-tester
chmod -R g+rwxs /var/tmp/osmo-gsm-tester
----

===== Allow DBus Access to ofono

Put a DBus configuration file in place that allows the 'osmo-gsm-tester' group
to access the org.ofono DBus path:

----
cat > /etc/dbus-1/system.d/osmo-gsm-tester.conf <<END
<!-- Additional rules for the osmo-gsm-tester to access org.ofono from user
     land -->

<!DOCTYPE busconfig PUBLIC "-//freedesktop//DTD D-BUS Bus Configuration 1.0//EN"
 "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd">
<busconfig>

  <policy group="osmo-gsm-tester">
    <allow send_destination="org.ofono"/>
  </policy>

</busconfig>
END
----

(No restart of dbus nor ofono necessary.)

[[install_capture_packets]]
===== Capture Packets

In order to allow collecting pcap traces of the network communication for later
reference, allow the osmo-gsm-tester group to capture packets using the 'tcpdump'
program:

----
chgrp osmo-gsm-tester /usr/sbin/tcpdump
chmod 750 /usr/sbin/tcpdump
setcap cap_net_raw,cap_net_admin=eip /usr/sbin/tcpdump
----

Put 'tcpdump' in the '$PATH' -- assuming that 'tcpdump' is available for root:

----
ln -s `which tcpdump` /usr/local/bin/tcpdump
----

TIP: Why a symlink in '/usr/local/bin'? On Debian, 'tcpdump' lives in
'/usr/sbin', which is not part of the '$PATH' for non-root users. To avoid
hardcoding non-portable paths in the osmo-gsm-tester source, 'tcpdump' must be
available in the '$PATH'. There are various trivial ways to modify '$PATH' for
login shells, but the jenkins build slave typically runs in a *non-login*
shell; modifying non-login shell enviroments is not trivially possible without
also interfering with files installed from debian packages. Probably the
easiest way to allow all users and all shells to find the 'tcpdump' binary is
to actually place a symbolic link in a directory that is already part of the
non-login shell's '$PATH'. Above example places such in '/usr/local/bin'.

Verify that a non-login shell can find 'tcpdump':

----
su jenkins -c 'which tcpdump'
# should print: "/usr/local/bin/tcpdump"
----

WARNING: When logged in via SSH on your main unit, running 'tcpdump' to capture
packets may result in a feedback loop: SSH activity to send tcpdump's output to
your terminal is in turn is picked up in the tcpdump trace, and so forth. When
testing 'tcpdump' access, make sure to have proper filter expressions in place.

TODO: allow skipping pcaps by configuration if access to tcpdump is not wanted

==== Allow Core Files

In case a binary run for the test crashes, a core file of the crash should be
written. This requires a limit rule. Create a file with the required rule:

----
sudo -s
echo "@osmo-gsm-tester - core unlimited" > /etc/security/limits.d/osmo-gsm-tester_allow-core.conf
----

Re-login the user to make these changes take effect.

Set the *kernel.core_pattern* sysctl to *core* (usually the default). For each
binary run by osmo-gsm-tester, a core file will then appear in the same dir that
contains stdout and stderr for that process (because this dir is set as CWD).

----
sysctl -w kernel.core_pattern=core
----

==== Allow Realtime Priority

Certain binaries should be run with real-time priority, like 'osmo-bts-trx'.
Add this permission on the main unit:

----
sudo -s
echo "@osmo-gsm-tester - rtprio 99" > /etc/security/limits.d/osmo-gsm-tester_allow-rtprio.conf
----

Re-login the user to make these changes take effect.

[[user_config_uhd]]
==== UHD

Grant permission to use the UHD driver to run USRP devices for osmo-bts-trx, by
adding the jenkins user to the 'usrp' group:

----
gpasswd -a jenkins usrp
----

==== Log Rotation

To avoid clogging up /var/log, it makes sense to choose a sane maximum log size:

----
echo maxsize 10M > /etc/logrotate.d/maxsize
----

==== Install Scripts

IMPORTANT: When using the jenkins build slave as configured above, *there is no
need to install the osmo-gsm-tester sources on the main unit*. The jenkins job
will do so implicitly by checking out the latest osmo-gsm-tester sources in the
workspace for every run. If you're using only the jenkins build slave, you may
skip this section.

If you prefer to use a fixed installation of the osmo-gsm-tester sources
instead of the jenkins workspace, you can:

. From the run job configured above, remove the line that says
+
----
PATH="$PWD/osmo-gsm-tester/src:$PATH" \
----
+
so that this uses a system wide installation instead.

. Install the sources e.g. in '/usr/local/src' as indicated below.

On the main unit, to install the latest in '/usr/local/src':

----
apt-get install git
mkdir -p /usr/local/src
cd /usr/local/src
git clone git://git.osmocom.org/osmo-gsm-tester
----

To allow all users to run 'osmo-gsm-tester.py', from login as well as non-login
shells, the easiest solution is to place a symlink in '/usr/local/bin':

----
ln -s /usr/local/src/osmo-gsm-tester/src/osmo-gsm-tester.py /usr/local/bin/
----

(See also the tip in <<install_capture_packets>> for a more detailed
explanation.)

The example configuration provided in the source is suitable for running as-is,
*if* your hardware setup matches (you could technically use that directly by a
symlink e.g. from '/usr/local/etc/osmo-gsm-tester' to the 'example' dir). If in
doubt, rather copy the example, point 'paths.conf' at the 'suites' dir, and
adjust your own configuration as needed. For example:

----
cd /etc
cp -R /usr/local/src/osmo-gsm-tester/example osmo-gsm-tester
sed -i 's#\.\./suites#/usr/local/src/osmo-gsm-tester/suites#' osmo-gsm-tester/paths.conf
----

NOTE: The configuration will be looked up in various places, see
<<config_paths>>.


== Hardware Choice and Configuration

=== SysmoBTS

To use the SysmoBTS in the osmo-gsm-tester, the following systemd services must
be disabled:

----
systemctl mask osmo-nitb osmo-bts-sysmo osmo-pcu sysmobts-mgr
----

This stops the stock setup keeping the BTS in operation and hence allows the
osmo-gsm-tester to install and launch its own versions of the SysmoBTS
software.

==== IP Address

To ensure that the SysmoBTS is always reachable at a fixed known IP address,
configure the eth0 to use a static IP address:

Adjust '/etc/network/interfaces' and replace the line

----
iface eth0 inet dhcp
----

with

----
iface eth0 inet static
  address 10.42.42.114
  netmask 255.255.255.0
  gateway 10.42.42.1
----

You may set the name server in '/etc/resolve.conf' (most likely to the IP of
the gateway), but this is not really needed by the osmo-gsm-tester.

==== Allow Core Files

In case a binary run for the test crashes, a core file of the crash should be
written. This requires a limits rule. Append a line to /etc/limits like:

----
ssh root@10.42.42.114
echo "* C16384" >> /etc/limits
----

==== Reboot

Reboot the BTS and make sure that the IP address for eth0 is now indeed
10.42.42.114, and that no osmo* programs are running.

----
ip a
ps w | grep osmo
----

==== SSH Access

Make sure that the jenkins user on the main unit is able to login on the
sysmoBTS, possibly erasing outdated host keys after a new rootfs was loaded:

On the main unit, for example do:

----
su - jenkins
ssh root@10.42.42.114
----

Fix any problems until you get a login on the sysmoBTS.


[[hardware_modems]]
=== Modems

TODO: describe modem choices and how to run ofono

[[hardware_trx]]
=== osmo-bts-trx

TODO: describe B200 family