Backend icinga is not running after Director deploy config

orsa · November 20, 2019, 9:37pm

Hi,
today upgrade icinga master from 2.11.1 to 2.11.2
now then i run deploy director config icinga service stopd and cant restarting
icinga2.log

[2019-11-21 00:32:53 +0300] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.2.2.125]:51812), user: director, agent: ).
[2019-11-21 00:32:53 +0300] information/HttpServerConnection: Request: POST /v1/config/stages/director (from [10.2.2.125]:51812), user: director, agent: ).
[2019-11-21 00:32:53 +0300] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/363db427-13f3-494d-b72e-69cbcf8fcb34/zones.d/director-global/001-director-basics.conf
[2019-11-21 00:32:53 +0300] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/363db427-13f3-494d-b72e-69cbcf8fcb34/zones.d/director-global/commands.conf
[2019-11-21 00:32:53 +0300] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/363db427-13f3-494d-b72e-69cbcf8fcb34/zones.d/director-global/host_templates.conf
[2019-11-21 00:32:53 +0300] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/363db427-13f3-494d-b72e-69cbcf8fcb34/zones.d/director-global/hostgroups.conf
[2019-11-21 00:32:53 +0300] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/363db427-13f3-494d-b72e-69cbcf8fcb34/zones.d/director-global/service_apply.conf
[2019-11-21 00:32:53 +0300] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/363db427-13f3-494d-b72e-69cbcf8fcb34/zones.d/director-global/service_templates.conf
[2019-11-21 00:32:53 +0300] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/363db427-13f3-494d-b72e-69cbcf8fcb34/zones.d/director-global/timeperiod_templates.conf
[2019-11-21 00:32:53 +0300] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/363db427-13f3-494d-b72e-69cbcf8fcb34/zones.d/director-global/timeperiods.conf
[2019-11-21 00:32:53 +0300] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/363db427-13f3-494d-b72e-69cbcf8fcb34/zones.d/director-global/user_templates.conf
[2019-11-21 00:32:53 +0300] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/363db427-13f3-494d-b72e-69cbcf8fcb34/zones.d/dp-monitor01.difco.local/agent_endpoints.conf
[2019-11-21 00:32:53 +0300] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/363db427-13f3-494d-b72e-69cbcf8fcb34/zones.d/dp-monitor01.difco.local/agent_zones.conf
[2019-11-21 00:32:53 +0300] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/363db427-13f3-494d-b72e-69cbcf8fcb34/zones.d/dp-monitor01.difco.local/hosts.conf
[2019-11-21 00:32:53 +0300] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/363db427-13f3-494d-b72e-69cbcf8fcb34/zones.d/dp-monitor01.difco.local/notification_apply.conf
[2019-11-21 00:32:53 +0300] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/363db427-13f3-494d-b72e-69cbcf8fcb34/zones.d/dp-monitor01.difco.local/notification_templates.conf
[2019-11-21 00:32:53 +0300] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/363db427-13f3-494d-b72e-69cbcf8fcb34/zones.d/dp-monitor01.difco.local/services.conf
[2019-11-21 00:32:53 +0300] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/363db427-13f3-494d-b72e-69cbcf8fcb34/zones.d/dp-monitor01.difco.local/users.conf
[2019-11-21 00:32:53 +0300] information/HttpServerConnection: HTTP client disconnected (from [10.2.2.125]:51812)
[2019-11-21 00:32:54 +0300] information/ApiListener: New client connection from [10.2.2.125]:51814 (no client certificate)
[2019-11-21 00:32:54 +0300] information/HttpServerConnection: Request: GET /v1/ (from [10.2.2.125]:51814), user: director, agent: ).
[2019-11-21 00:32:54 +0300] information/HttpServerConnection: Request: GET /v1/objects/apilisteners/api?all_joins=1 (from [10.2.2.125]:51814), user: director, agent: ).
[2019-11-21 00:32:54 +0300] information/HttpServerConnection: HTTP client disconnected (from [10.2.2.125]:51814)
[2019-11-21 00:32:56 +0300] information/Application: Received request to shut down.
[2019-11-21 00:32:56 +0300] information/Application: Shutting down...
[2019-11-21 00:32:57 +0300] information/ApiListener: New client connection from [10.2.2.120]:36938 (no client certificate)
[2019-11-21 00:32:57 +0300] information/ApiListener: No data received on new API connection from [10.2.2.120]:36938. Ensure that the remote endpoints are properly configured in a cluster setup.
[2019-11-21 00:33:00 +0300] information/CheckerComponent: 'checker' stopped.
[2019-11-21 00:33:00 +0300] information/NotificationComponent: 'notification' stopped.
[2019-11-21 00:33:00 +0300] information/InfluxdbWriter: 'influxdb' paused.
[2019-11-21 00:33:00 +0300] information/DbConnection: Pausing IDO connection: ido-mysql
[2019-11-21 00:33:00 +0300] information/IdoMysqlConnection: Disconnected from 'ido-mysql' database 'icinga2'.
[2019-11-21 00:33:00 +0300] information/IdoMysqlConnection: 'ido-mysql' paused.
[2019-11-21 00:33:00 +0300] information/DbConnection: 'ido-mysql' stopped.
[2019-11-21 00:33:00 +0300] information/ApiListener: 'api' stopped.

orsa · November 21, 2019, 2:26pm

fogot startup log
but here type 2.9 version

[2018-07-31 11:18:57 +0300] information/cli: Icinga application loader (version: r2.9.1-1)
[2018-07-31 11:18:57 +0300] information/cli: Loading configuration file(s).
[2018-07-31 11:18:57 +0300] information/ConfigItem: Committing config item(s).
[2018-07-31 11:18:57 +0300] information/ApiListener: My API identity: dp-monitor01.difco.local
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 19 Services.
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 1 InfluxdbWriter.
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 2 HostGroups.
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 1 FileLogger.
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 1 NotificationComponent.
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 2 NotificationCommands.
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 186 Notifications.
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 1 IcingaApplication.
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 167 Hosts.
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 1 ApiListener.
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 2 Comments.
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 1 CheckerComponent.
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 15 Zones.
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 13 Endpoints.
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 4 ApiUsers.
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 213 CheckCommands.
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 1 TimePeriod.
[2018-07-31 11:18:57 +0300] information/ConfigItem: Instantiated 1 User.
[2018-07-31 11:18:57 +0300] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2018-07-31 11:18:57 +0300] information/cli: Finished validating the configuration file(s).

dnsmichi · November 21, 2019, 2:29pm

What exactly does this mean, what does systemctl status icinga2 say? Any more insights in the icinga2.log or a crash log even?

orsa · November 21, 2019, 3:21pm

Hi Michael,
systemctl status icinga2
● icinga2.service - Icinga host/service/network monitoring system
Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/icinga2.service.d
└─limits.conf
Active: failed (Result: exit-code) since Чт 2019-11-21 18:18:19 MSK; 2s ago
Process: 1615 ExecStart=/usr/sbin/icinga2 daemon --close-stdio -e ${ICINGA2_ERROR_LOG} (code=exited, status=132)
Process: 1586 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/default/icinga2 (code=exited, status=0/SUCCESS)
Main PID: 1615 (code=exited, status=132)

ноя 21 17:34:12 dp-monitor01 icinga2[1615]: [2019-11-21 17:34:12 +0300] information/ConfigItem: Instantiated 240 CheckCommands.
ноя 21 17:34:12 dp-monitor01 icinga2[1615]: [2019-11-21 17:34:12 +0300] information/ConfigItem: Instantiated 1 TimePeriod.
ноя 21 17:34:12 dp-monitor01 icinga2[1615]: [2019-11-21 17:34:12 +0300] information/ConfigItem: Instantiated 1 User.
ноя 21 17:34:12 dp-monitor01 icinga2[1615]: [2019-11-21 17:34:12 +0300] information/ConfigItem: Instantiated 200 Services.
ноя 21 17:34:13 dp-monitor01 icinga2[1615]: [2019-11-21 17:34:13 +0300] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
ноя 21 17:34:13 dp-monitor01 icinga2[1615]: [2019-11-21 17:34:13 +0300] information/cli: Closing console log.
ноя 21 17:34:13 dp-monitor01 systemd[1]: Started Icinga host/service/network monitoring system.
ноя 21 18:18:19 dp-monitor01 systemd[1]: icinga2.service: Main process exited, code=exited, status=132/n/a
ноя 21 18:18:19 dp-monitor01 systemd[1]: icinga2.service: Unit entered failed state.
ноя 21 18:18:19 dp-monitor01 systemd[1]: icinga2.service: Failed with result 'exit-code'.

folder /var/log/icinga2/crash/ - empty

/var/log/icinga2/icinga2.log
[2019-11-21 18:18:11 +0300] information/HttpServerConnection: HTTP client disconnected (from [10.2.2.125]:33852)
[2019-11-21 18:18:13 +0300] information/Application: Received request to shut down.
[2019-11-21 18:18:14 +0300] information/Application: Shutting down…
[2019-11-21 18:18:15 +0300] information/ApiListener: New client connection from [10.2.2.125]:33856 (no client certificate)
[2019-11-21 18:18:15 +0300] information/HttpServerConnection: Request: GET /v1/ (from [10.2.2.125]:33856), user: director, agent: ).
[2019-11-21 18:18:15 +0300] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.2.2.125]:33856), user: director, agent: ).
[2019-11-21 18:18:15 +0300] information/HttpServerConnection: Request: GET /v1/config/stages/director/572c5498-f4c2-4b8a-8abb-29848e6054b3 (from [10.2.2.125]:33856), user: director, agent: ).
[2019-11-21 18:18:15 +0300] information/HttpServerConnection: Request: GET /v1/config/files/director/572c5498-f4c2-4b8a-8abb-29848e6054b3/status (from [10.2.2.125]:33856), user: director, agent: ).
[2019-11-21 18:18:15 +0300] information/HttpServerConnection: Request: GET /v1/config/files/director/572c5498-f4c2-4b8a-8abb-29848e6054b3/startup.log (from [10.2.2.125]:33856), user: director, agent: ).
[2019-11-21 18:18:15 +0300] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.2.2.125]:33856), user: director, agent: ).
[2019-11-21 18:18:15 +0300] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.2.2.125]:33856), user: director, agent: ).
[2019-11-21 18:18:15 +0300] information/HttpServerConnection: HTTP client disconnected (from [10.2.2.125]:33856)
[2019-11-21 18:18:18 +0300] information/CheckerComponent: ‘checker’ stopped.
[2019-11-21 18:18:18 +0300] information/NotificationComponent: ‘notification’ stopped.
[2019-11-21 18:18:18 +0300] information/InfluxdbWriter: ‘influxdb’ paused.
[2019-11-21 18:18:18 +0300] information/DbConnection: Pausing IDO connection: ido-mysql
[2019-11-21 18:18:18 +0300] information/IdoMysqlConnection: Disconnected from ‘ido-mysql’ database ‘icinga2’.
[2019-11-21 18:18:18 +0300] information/IdoMysqlConnection: ‘ido-mysql’ paused.
[2019-11-21 18:18:18 +0300] information/DbConnection: ‘ido-mysql’ stopped.
[2019-11-21 18:18:18 +0300] information/ApiListener: ‘api’ stopped.

after sever reboot all work fine

dnsmichi · November 21, 2019, 4:24pm

Odd. Can you please share the output of icinga2 --version in full?

Also, when you manually restart the service (don’t reboot, that’s not necessary), did the Director config deployment succeed in its listing?

orsa · November 21, 2019, 5:03pm

$ icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.11.2-1)

Copyright (c) 2012-2019 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Ubuntu
  Platform version: 16.04.6 LTS (Xenial Xerus)
  Kernel: Linux
  Kernel version: 4.4.0-169-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 5.4.0
  Build host: runner-LTrJQZ9N-project-298-concurrent-0

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

after restart service cant up (
its news for me

$ **sudo systemctl status icinga2.service** 
● icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/icinga2.service.d
           └─limits.conf
   Active: active (running) since Чт 2019-11-21 19:56:07 MSK; 4min 26s ago
  Process: 1579 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/default/icinga2 (code=exited, status=0/SUCCESS)
 Main PID: 1617 (icinga2)
    Tasks: 95
   Memory: 68.1M
      CPU: 22.656s
   CGroup: /system.slice/icinga2.service
           ├─1617 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log
           ├─1769 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log
           ├─1775 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log
        
$ **sudo systemctl restart icinga2.service**                                                                                                    
$ **sudo systemctl status icinga2.service**  
● icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/icinga2.service.d
           └─limits.conf
   Active: failed (Result: exit-code) since Чт 2019-11-21 20:00:51 MSK; 2s ago
  Process: 4696 ExecStart=/usr/sbin/icinga2 daemon --close-stdio -e ${ICINGA2_ERROR_LOG} (code=exited, status=132)
  Process: 4687 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/default/icinga2 (code=exited, status=0/SUCCESS)
 Main PID: 4696 (code=exited, status=132)

ноя 21 20:00:51 dp-monitor01 icinga2[4696]: [2019-11-21 20:00:51 +0300] information/ConfigItem: Instantiated 240 CheckCommands.
ноя 21 20:00:51 dp-monitor01 icinga2[4696]: [2019-11-21 20:00:51 +0300] information/ConfigItem: Instantiated 1 TimePeriod.
ноя 21 20:00:51 dp-monitor01 icinga2[4696]: [2019-11-21 20:00:51 +0300] information/ConfigItem: Instantiated 1 User.
ноя 21 20:00:51 dp-monitor01 icinga2[4696]: [2019-11-21 20:00:51 +0300] information/ConfigItem: Instantiated 200 Services.
ноя 21 20:00:51 dp-monitor01 icinga2[4696]: [2019-11-21 20:00:51 +0300] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
ноя 21 20:00:51 dp-monitor01 icinga2[4696]: [2019-11-21 20:00:51 +0300] information/cli: Closing console log.
ноя 21 20:00:51 dp-monitor01 systemd[1]: Started Icinga host/service/network monitoring system.
ноя 21 20:00:51 dp-monitor01 systemd[1]: icinga2.service: Main process exited, code=exited, status=132/n/a
ноя 21 20:00:51 dp-monitor01 systemd[1]: icinga2.service: Unit entered failed state.
ноя 21 20:00:51 dp-monitor01 systemd[1]: icinga2.service: Failed with result 'exit-code'.

orsa · November 26, 2019, 4:46pm

hi, any idea?

if I understand correctly, the problem is that not all icinga components have been updated correctly

how can the situation be corrected?

dnsmichi · November 27, 2019, 8:07am

I don’t quite understand the exit code of 132 here. This would mean a SIGILL signal with a followed core and crash dump.

Try enabling core dumps and see whether one gets generated.

orsa · November 27, 2019, 6:51pm

i do core dump,
sudo gdb /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 /var/lib/cores/core.icinga2.6782

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x00007f375c4ceedc in _xend () at pthread_rwlock_unlock.c:38
38      pthread_rwlock_unlock.c: No such file or directory.
[Current thread is 1 (Thread 0x7f37570cc700 (LWP 6834))]
(gdb) bt
#0  0x00007f375c4ceedc in _xend () at pthread_rwlock_unlock.c:38
#1  __GI___pthread_rwlock_unlock (rwlock=0x7f36e0003160) at pthread_rwlock_unlock.c:38
#2  0x00007f37567e558f in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#3  0x00007f375bf0bfaf in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#4  0x00007f375bf0d180 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#5  0x00007f375bf0e0f8 in RSA_sign () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#6  0x00007f375bf13f64 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#7  0x00007f375bf3ae2d in EVP_SignFinal () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#8  0x00007f375c277bdc in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
#9  0x00007f375c27a147 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
#10 0x00000000005c8792 in ?? ()
#11 0x0000000000737aaa in ?? ()
#12 0x0000000000761d1b in boost::asio::detail::executor_op<boost::asio::detail::binder2<boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::handshake_op, boost::asio::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::executor>, void> >, boost::system::error_code, unsigned long>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) ()
#13 0x0000000000a7d8c8 in boost::asio::detail::executor_op<boost::asio::executor::function, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) ()
#14 0x00000000009ab7c9 in ?? ()
#15 0x0000000000a814f9 in ?? ()
#16 0x00000000007389f6 in boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::handshake_op, boost::asio::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::executor>, void> > >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) ()
#17 0x0000000000ab8846 in ?? ()
#18 0x0000000000608f9a in icinga::IoEngine::RunEventLoop() ()
#19 0x00007f375b912c80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#20 0x00007f375c4c96ba in start_thread (arg=0x7f37570cc700) at pthread_create.c:333
#21 0x00007f375d7d441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb)

dnsmichi · November 28, 2019, 8:24am

A guess - is the hardware involving Intel’s Haswell architecture? Then these are relevant:
https://sourceware.org/bugzilla/show_bug.cgi?id=17561

It seems that the _xend CPU instruction is not properly implemented which leads to crashes.

It may also be a combination with glibc provided in Ubuntu 16.04.

orsa · November 28, 2019, 9:42pm

You right, its Haswell server

what would you recommend in this situation, install 18.04 ?

dnsmichi · November 29, 2019, 7:52am

If that’s possible, it would allow to mitigate further or even the problem away.

orsa · November 29, 2019, 3:17pm

)
thank you, i try it