mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8

The task was shut down prematurely (before the stack was manually stopped)
It seems there was an error during the task execution
Task details
Desired state
shutdown
Node ID
xabg2jtix07dgcn2y30ri5x8m
Service ID
qoio2zh6mw32tqom1leobewsq
Task ID
p9nkio9pq6mn93jnv7bcrqpn8
Container ID
25fa9c1998bcdabe9a4f5140542d022874bef2a552a4d05f2e437ad2a5450115
Created at
2020-12-29T16:07:34.290491+00:00
Updated at
2020-12-29T16:08:28.187318+00:00
Started at
2020-12-29T16:07:55.113634+00:00
Memory limit (MB)
3944.539
Logs
Snapshot of the most recent log entries
mishmash_error.2.p9nkio9pq6mn@manager1    |  'uname': {'machine': 'x86_64',
mishmash_error.2.p9nkio9pq6mn@manager1    |            'node': 'mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8',
mishmash_error.2.p9nkio9pq6mn@manager1    |            'processor': 'x86_64',
mishmash_error.2.p9nkio9pq6mn@manager1    |            'release': '4.15.0-121-generic',
mishmash_error.2.p9nkio9pq6mn@manager1    |            'system': 'Linux',
mishmash_error.2.p9nkio9pq6mn@manager1    |            'version': '#123-Ubuntu SMP Mon Oct 5 16:16:40 UTC 2020'}}
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:07:55 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r cpuacct.usage /: b'/:\n3737464750\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:07:55 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.monitor.packet[1] DEBUG Starting LiveCapture on interface eth0 with display filter: tcp.port==1883 or tcp.port==80 or tcp.port==81 or tcp.port==5683 or udp.port==5683
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:00 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r cpuacct.usage /: b'/:\n5119276209\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:00 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r cpu.cfs_period_us /: b'/:\n100000\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:00 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r cpu.cfs_quota_us /: b'/:\n-1\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:00 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r memory.usage_in_bytes /: b'/:\n203153408\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:00 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r memory.limit_in_bytes /: b'/:\n9223372036854771712\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:05 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r cpuacct.usage /: b'/:\n5179682617\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:05 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r cpu.cfs_period_us /: b'/:\n100000\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:05 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r cpu.cfs_quota_us /: b'/:\n-1\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:05 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r memory.usage_in_bytes /: b'/:\n203132928\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:05 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r memory.limit_in_bytes /: b'/:\n9223372036854771712\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:05 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.monitor.base[1] DEBUG ZADD (2 items): wotemu:system:mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:10 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r cpuacct.usage /: b'/:\n5249112239\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:10 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r cpu.cfs_period_us /: b'/:\n100000\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:10 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r cpu.cfs_quota_us /: b'/:\n-1\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:10 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r memory.usage_in_bytes /: b'/:\n204222464\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:10 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r memory.limit_in_bytes /: b'/:\n9223372036854771712\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:15 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r cpuacct.usage /: b'/:\n5310014700\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:15 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r cpu.cfs_period_us /: b'/:\n100000\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:15 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r cpu.cfs_quota_us /: b'/:\n-1\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:15 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r memory.usage_in_bytes /: b'/:\n204107776\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:15 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r memory.limit_in_bytes /: b'/:\n9223372036854771712\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:15 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.monitor.base[1] DEBUG ZADD (2 items): wotemu:system:mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:20 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r cpuacct.usage /: b'/:\n5368116048\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:20 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r cpu.cfs_period_us /: b'/:\n100000\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:20 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r cpu.cfs_quota_us /: b'/:\n-1\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:20 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r memory.usage_in_bytes /: b'/:\n203001856\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:20 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.utils[1] DEBUG /usr/bin/cgget -v -r memory.limit_in_bytes /: b'/:\n9223372036854771712\n\n'
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:22 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.cli.app[1] ERROR WoT app error: Exception('Error raised at 2020-12-29 16:08:22.091424 UTC')
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:22 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.cli.app[1] INFO Cancelling WoT app task: <Task finished name='Task-4' coro=<app() done, defined at /usr/local/lib/python3.8/dist-packages/wotemu/apps/error.py:9> exception=Exception('Error raised at 2020-12-29 16:08:22.091424 UTC')>
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:22 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.cli.app[1] WARNING Error during WoT app cancelation
mishmash_error.2.p9nkio9pq6mn@manager1    | Traceback (most recent call last):
mishmash_error.2.p9nkio9pq6mn@manager1    |   File "/usr/local/lib/python3.8/dist-packages/wotemu/cli/app.py", line 163, in _stop
mishmash_error.2.p9nkio9pq6mn@manager1    |     await asyncio.wait_for(app_task, _TIMEOUT)
mishmash_error.2.p9nkio9pq6mn@manager1    |   File "/usr/lib/python3.8/asyncio/tasks.py", line 483, in wait_for
mishmash_error.2.p9nkio9pq6mn@manager1    |     return fut.result()
mishmash_error.2.p9nkio9pq6mn@manager1    |   File "/usr/local/lib/python3.8/dist-packages/wotemu/apps/error.py", line 19, in app
mishmash_error.2.p9nkio9pq6mn@manager1    |     raise Exception(f"Error raised at {now} UTC")
mishmash_error.2.p9nkio9pq6mn@manager1    | Exception: Error raised at 2020-12-29 16:08:22.091424 UTC
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:22 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.cli.app[1] DEBUG Shutting down Servient: <wotpy.wot.servient.Servient object at 0x7fa7ddc2f550>
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:22 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.monitor.base[1] DEBUG Stopping node monitor
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:22 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.monitor.system[1] DEBUG Cancelled system usage task
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:22 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.monitor.packet[1] DEBUG Cancelled Task for: <SpawnProcess name='PysharkCapture-eth0' pid=87 parent=1 started daemon>
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:22 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.monitor.packet[1] DEBUG Setting stop event for: <SpawnProcess name='PysharkCapture-eth0' pid=87 parent=1 started daemon>
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:22 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.monitor.packet[1] DEBUG Waiting 5.0 secs for exit: <SpawnProcess name='PysharkCapture-eth0' pid=87 parent=1 started daemon>
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:27 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.monitor.packet[1] WARNING <SpawnProcess name='PysharkCapture-eth0' pid=87 parent=1 started daemon> did not terminate with stop event
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:27 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.monitor.packet[1] INFO Terminating process: <SpawnProcess name='PysharkCapture-eth0' pid=87 parent=1 started daemon>
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:27 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.monitor.packet[1] DEBUG Waiting 5.0 secs for exit: <SpawnProcess name='PysharkCapture-eth0' pid=87 parent=1 started daemon>
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:27 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.monitor.base[1] DEBUG Closed Redis connection
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:27 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.cli.app[1] DEBUG Closing Redis
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:27 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.cli.app[1] DEBUG Stopping loop
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:27 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.cli.app[1] DEBUG Closing loop
mishmash_error.2.p9nkio9pq6mn@manager1    | 2020-12-29 16:08:27 mishmash_error.2.p9nkio9pq6mn93jnv7bcrqpn8 wotemu.cli.app[1] WARNING Exiting with code: 1