Gateway service stops working

chester40391

I am using the gateway as a service on my always on Windows computer. It works perfectly when it is working but stops functioning every few days yet the service is still running. It starts working again once I restart the service.

Anything to look for to troubleshoot this?
Is there a command that will get me to the live service output? (The output you would normally see in the command line window if you aren’t running as a service) This may give me some clue to what is going on, it would at least show me if the service is responding to requests at all when it stops working.

Thanks.

bk

I have the same thing, but not in days… could be a week or two. I have “assumed” its just a windows thing. Unless you reboot window machines every so often, the memory leakage from any other program affect anything / all things / performance on the Win machine, and the only cure to to reboot every so often.
If I cant reboot at the moment, I stop and start the service, I have shortcuts on my desktop for it. My always on Win machine is also a my workstation and server at this moment, after my server died.
I’ll go with PI solution in the near future most likely.

Monocle

@chester40391

You can access the log data via the file system or you can use the following Windows PowerShell command to “follow” the log in a live output window:

Logging
If running the Monocle Gateway as a Windows service the application will write its output data to a log file: monocle-gateway.log.

To monitor the log file while the service is running you can open a Power Shell window and use the following command to “tail” the log file. Get-Content -Path .\monocle-gateway.log -Wait

(See: https://monoclecam.com/monocle-gateway/install/windows#logging)

You can also just run the “monocle-gateway.exe” from the command line and not use the Windows service while you are diagnosing. It might be an interesting test to see if it only fails while running as a service versus running standalone from command line.

Thanks, Robert

tfjield

The same thing happens to me, but I’m running in a container on a QNAP NAS. It happens every week or two. When this happens the Gateway is still running, but it isn’t using any resources. The Amazon devices try to connect but then say that the camera isn’t responding.

The camera mostly viewed is a Unifi camera running in standalone mode. I use @proxy, and just added @hangup to see if that would make a difference.

I should have copied the console output, but from memory it showed the Alexa device connecting, then disconnecting after a little while. I don’t remember see any errors.

tfjield

@Monocle, I finally had the problem again.

Here is the end of my console output from today:

------------------------------------------------- 
INITIALIZE RTSP STREAM: Front Facing East 
------------------------------------------------- 
 - NAME : Front Facing East 
 - LABEL : PRIMARY 
 - URL : rtsp://192.168.1.47:554/s1 
 - UUID : STREAM:<streamhash> 
 - SESS : <sesshash> 
 - MODIF : Mon Jul 06 2020 22:46:51 GMT+0000 (UTC) 
 - TAGS : @hangup,@proxy 
------------------------------------------------- 
 
2020-08-19T20:06:34.573Z [INFO] [RTSP PROXY] REGISTERING STREAM [Front Facing East/PRIMARY]; (STREAM:<streamhash>) 
2020-08-19T20:06:34.574Z [DEBUG] <RTSP-PROXY> [REQUEST] --> [REGISTER] rtsp://192.168.1.47:554/s1 
2020-08-19T20:06:34.574Z [TRACE] <RTSP-PROXY> [REQUEST] --> [HEADERS] { 
 "cseq": "1", 
 "transport": "reuse_connection=0;preferred_delivery_protocol=udp;proxy_url_suffix=STREAM:<streamhash>;username=<username>;password=<password>" 
} 
2020-08-19T20:06:34.851Z [INFO] [192.168.1.165:33414 <code>] RTSP CLIENT SOCKET CONNECTED 
2020-08-19T20:06:34.986Z [INFO] [192.168.1.165:33414 <code>] RTSP CLIENT ATTACHED TO STREAM: Front Facing East (STREAM:<streamhash>) 
2020-08-19T20:06:34.988Z [INFO] [192.168.1.165:33414 <code>] RTSP ENDPOINT SOCKET CONNECTED {127.0.0.1:8554} 
2020-08-19T20:06:34.988Z [DEBUG] [192.168.1.165:33414 <code>] [CLIENT REQUEST] --> [DESCRIBE] rtsp://<rtsphash>.mproxy.io:443/STREAM:<streamhash>?session=<sesshash> 
2020-08-19T20:06:34.988Z [TRACE] [192.168.1.165:33414 <code>] [CLIENT REQUEST] --> [HEADERS] { 
 "accept": "application/sdp", 
 "user-agent": "Fire OS/6.0 stagefright/1.2 (Linux;Android 7.1.2)", 
 "cseq": "1" 
} 
2020-08-19T20:06:34.988Z [DEBUG] [192.168.1.165:33414 <code>] [ENDPOINT REQUEST] --> [DESCRIBE] rtsp://<rtsphash>.mproxy.io:443/STREAM:<streamhash> 
2020-08-19T20:06:34.988Z [TRACE] [192.168.1.165:33414 <code>] [ENDPOINT REQUEST] --> [HEADERS] { 
 "accept": "application/sdp", 
 "user-agent": "Fire OS/6.0 stagefright/1.2 (Linux;Android 7.1.2)", 
 "cseq": "1" 
} 
2020-08-19T20:07:04.986Z [WARN] [192.168.1.165:33414 <code>] RTSP CLIENT SOCKET TIMEOUT 
2020-08-19T20:07:04.988Z [WARN] [192.168.1.165:33414 <code>] RTSP ENDPOINT SOCKET TIMEOUT [192.168.1.165:33414 <code>] 
2020-08-19T20:07:04.989Z [INFO] [192.168.1.165:33414 <code>] RTSP ENDPOINT SOCKET CLOSED [192.168.1.165:33414 <code>] 
2020-08-19T20:07:04.989Z [INFO] [192.168.1.165:33414 <code>] RTSP CLIENT SOCKET CLOSED 
2020-08-19T20:07:04.989Z [INFO] [192.168.1.165:33414 <code>] RTSP CLIENT DETACHED FROM STREAM: Front Facing East (STREAM:<streamhash>) 
2020-08-19T20:07:04.989Z [INFO] [RTSP PROXY] DE-REGISTERING STREAM [Front Facing East/PRIMARY]; NO LONGER IN USE 
2020-08-19T20:07:04.990Z [DEBUG] <RTSP-PROXY> [REQUEST] --> [DEREGISTER] rtsp://192.168.1.47:554/s1 
2020-08-19T20:07:04.990Z [TRACE] <RTSP-PROXY> [REQUEST] --> [HEADERS] { 
 "cseq": "1", "transport": "reuse_connection=0;preferred_delivery_protocol=udp;proxy_url_suffix=STREAM:<streamhash>" 
}

I didn’t know what would compromise security, so I replaced the hashes, etc.

Does this help explain what’s happening? The Echo Show would just say that the camera isn’t responding. I restarted the container and it’s working perfectly again.

Monocle

@tfjield

I don’t see anything in the log that indicates a failure or error. Is it possible that you have a public IP address change from your ISP? If you public IP address gets changed, Monocle Gateway won’t know about it and thus won’t update our servers with your latest address … until you restart it … and then the gateway will automatically re-register using your new address.

Thanks, Robert

tfjield

@Monocle
I don’t think so. My external IP is the same now as it has been for years.

I’ll post the next time it happens again. Maybe the timing will yield some clue.

chester40391

@Monocle said in Gateway service stops working:

can also just run the “monocle-gateway.exe” from the command line and not use the Windows service while you are diagnosing. It might be an interesting test to see if it only fails while running as a service versus running standalone from command line

It fails regardless if I use the service or the .exe. I noticed a comment below mine regarding public IP changes causing an issue. I will have to check for this. It is possible my IP changes frequently. If that is the case is there anything I can do aside from programatically restarting the service daily?

Monocle

@chester40391 said in Gateway service stops working:

It is possible my IP changes frequently. If that is the case is there anything I can do aside from programatically restarting the service daily?

Probably not … however, if you find this to be the case let us know and we can add a periodic check in the gateway to try and detect IP changes and re-register any updated address with the servers.

Thanks, Robert

chester40391

I confirmed my IP is changing frequently but I also experienced the gateway stopped working without an IP change. Still running but Alexa integration stops working until I restart the gateway app/service. I will try to make time to grab logs.

“we can add a periodic check in the gateway to try and detect IP changes and re-register any updated address with the servers.” I think this would be a great implementation. The reason I found Monocle is because I am in the country with very limited internet options and low caps which means A. I cant use heavy cloud based services even if I wanted to. B. As a result of the scrappy technology behind my ISP solution I have frequent IP changes. This might be too edge case for the effort but another possible solution might be to allow the ability to statically set a DNS name instead of detecting the IP. I could use a dynamic DNS provider and a client on my network to keep DNS updated of IP changes.

tfjield

Had the same problem again. The camera is working fine (continually recording on Blue Iris) and it works again with Alexa after restarting Monocle.

Here’s the console output that I could capture.

2020-10-06T17:33:18.224Z [WARN] [192.168.1.165:34024 <Key3>] RTSP ENDPOINT SOCKET TIMEOUT [192.168.1.165:34024 <Key3>]
2020-10-06T17:33:18.224Z [INFO] [192.168.1.165:34024 <Key3>] RTSP ENDPOINT SOCKET CLOSED [192.168.1.165:34024 <Key3>]
2020-10-06T17:33:18.228Z [INFO] [192.168.1.165:34024 <Key3>] RTSP CLIENT SOCKET CLOSED
2020-10-06T17:33:18.228Z [INFO] [192.168.1.165:34024 <Key3>] RTSP CLIENT DETACHED FROM STREAM: Front Facing East (STREAM:9941432f-959c-4efa-
2020-10-06T17:33:18.228Z [INFO] [RTSP PROXY] DE-REGISTERING STREAM [Front Facing East/PRIMARY]; NO LONGER IN USE
2020-10-06T17:33:18.229Z [DEBUG] <RTSP-PROXY> [REQUEST] --> [DEREGISTER] rtsp://192.168.1.47:554/s1
2020-10-06T17:33:18.229Z [TRACE] <RTSP-PROXY> [REQUEST] --> [HEADERS] {
"cseq": "1",
"transport": "reuse_connection=0;preferred_delivery_protocol=udp;proxy_url_suffix=STREAM:<stream>"
}
-------------------------------------------------
INITIALIZE RTSP STREAM: Front Facing East
-------------------------------------------------
- NAME : Front Facing East
- LABEL : PRIMARY
- URL : rtsp://192.168.1.47:554/s1
- UUID : STREAM:<stream>
- SESS : <sess>
- MODIF : Mon Jul 06 2020 22:46:51 GMT+0000 (UTC)
- TAGS : @hangup,@proxy
-------------------------------------------------
2020-10-06T17:33:20.238Z [INFO] [RTSP PROXY] REGISTERING STREAM [Front Facing East/PRIMARY]; (STREAM:<stream>)
2020-10-06T17:33:20.238Z [DEBUG] <RTSP-PROXY> [REQUEST] --> [REGISTER] rtsp://192.168.1.47:554/s1
2020-10-06T17:33:20.239Z [TRACE] <RTSP-PROXY> [REQUEST] --> [HEADERS] {
"cseq": "1",
"transport": "reuse_connection=0;preferred_delivery_protocol=udp;proxy_url_suffix=STREAM:<stream>;username=ubnt;pass
}
2020-10-06T17:33:20.536Z [INFO] [192.168.1.165:34026 <key>] RTSP CLIENT SOCKET CONNECTED
2020-10-06T17:33:20.665Z [INFO] [192.168.1.165:34026 <key>] RTSP CLIENT ATTACHED TO STREAM: Front Facing East (STREAM:9941432f-959c-4efa-99
2020-10-06T17:33:20.665Z [INFO] [192.168.1.165:34026 <key>] RTSP ENDPOINT SOCKET CONNECTED {127.0.0.1:8554}
2020-10-06T17:33:20.665Z [DEBUG] [192.168.1.165:34026 <key>] [CLIENT REQUEST] --> [DESCRIBE] rtsp://<rtsp>.mp
2020-10-06T17:33:20.665Z [TRACE] [192.168.1.165:34026 <key>] [CLIENT REQUEST] --> [HEADERS] {
"accept": "application/sdp",
"user-agent": "Fire OS/6.0 stagefright/1.2 (Linux;Android 7.1.2)",
"cseq": "1"
}
2020-10-06T17:33:20.666Z [DEBUG] [192.168.1.165:34026 <key>] [ENDPOINT REQUEST] --> [DESCRIBE] rtsp://<rtsp>.
2020-10-06T17:33:20.666Z [TRACE] [192.168.1.165:34026 <key>] [ENDPOINT REQUEST] --> [HEADERS] {
"accept": "application/sdp",
"user-agent": "Fire OS/6.0 stagefright/1.2 (Linux;Android 7.1.2)",
"cseq": "1"
}
2020-10-06T17:33:50.667Z [WARN] [192.168.1.165:34026 <key>] RTSP CLIENT SOCKET TIMEOUT
2020-10-06T17:33:50.668Z [WARN] [192.168.1.165:34026 <key>] RTSP ENDPOINT SOCKET TIMEOUT [192.168.1.165:34026 <key>]
2020-10-06T17:33:50.668Z [INFO] [192.168.1.165:34026 <key>] RTSP ENDPOINT SOCKET CLOSED [192.168.1.165:34026 <key>]
2020-10-06T17:33:50.670Z [INFO] [192.168.1.165:34026 <key>] RTSP CLIENT SOCKET CLOSED
2020-10-06T17:33:50.670Z [INFO] [192.168.1.165:34026 <key>] RTSP CLIENT DETACHED FROM STREAM: Front Facing East (STREAM:9941432f-959c-4efa-
2020-10-06T17:33:50.670Z [INFO] [RTSP PROXY] DE-REGISTERING STREAM [Front Facing East/PRIMARY]; NO LONGER IN USE
2020-10-06T17:33:50.671Z [DEBUG] <RTSP-PROXY> [REQUEST] --> [DEREGISTER] rtsp://192.168.1.47:554/s1
2020-10-06T17:33:50.671Z [TRACE] <RTSP-PROXY> [REQUEST] --> [HEADERS] {
"cseq": "1",
"transport": "reuse_connection=0;preferred_delivery_protocol=udp;proxy_url_suffix=STREAM:<stream>"
}
-------------------------------------------------
INITIALIZE RTSP STREAM: Front Facing East
-------------------------------------------------
- NAME : Front Facing East
- LABEL : PRIMARY
- URL : rtsp://192.168.1.47:554/s1
- UUID : STREAM:<stream>
- SESS : <sess2>
- MODIF : Mon Jul 06 2020 22:46:51 GMT+0000 (UTC)
- TAGS : @hangup,@proxy
-------------------------------------------------
2020-10-06T17:33:52.835Z [INFO] [RTSP PROXY] REGISTERING STREAM [Front Facing East/PRIMARY]; (STREAM:<stream>)
2020-10-06T17:33:52.836Z [DEBUG] <RTSP-PROXY> [REQUEST] --> [REGISTER] rtsp://192.168.1.47:554/s1
2020-10-06T17:33:52.836Z [TRACE] <RTSP-PROXY> [REQUEST] --> [HEADERS] {
"cseq": "1",
"transport": "reuse_connection=0;preferred_delivery_protocol=udp;proxy_url_suffix=STREAM:<stream>;username=ubnt;pass
}
2020-10-06T17:33:53.165Z [INFO] [192.168.1.165:34028 <key2>] RTSP CLIENT SOCKET CONNECTED
2020-10-06T17:33:53.293Z [INFO] [192.168.1.165:34028 <key2>] RTSP CLIENT ATTACHED TO STREAM: Front Facing East (STREAM:9941432f-959c-4efa-99
2020-10-06T17:33:53.293Z [INFO] [192.168.1.165:34028 <key2>] RTSP ENDPOINT SOCKET CONNECTED {127.0.0.1:8554}
2020-10-06T17:33:53.293Z [DEBUG] [192.168.1.165:34028 <key2>] [CLIENT REQUEST] --> [DESCRIBE] rtsp://<rtsp>.mp
2020-10-06T17:33:53.293Z [TRACE] [192.168.1.165:34028 <key2>] [CLIENT REQUEST] --> [HEADERS] {
"accept": "application/sdp",
"user-agent": "Fire OS/6.0 stagefright/1.2 (Linux;Android 7.1.2)",
"cseq": "1"
}
2020-10-06T17:33:53.294Z [DEBUG] [192.168.1.165:34028 <key2>] [ENDPOINT REQUEST] --> [DESCRIBE] rtsp://<rtsp>.
2020-10-06T17:33:53.294Z [TRACE] [192.168.1.165:34028 <key2>] [ENDPOINT REQUEST] --> [HEADERS] {
"accept": "application/sdp",
"user-agent": "Fire OS/6.0 stagefright/1.2 (Linux;Android 7.1.2)",
"cseq": "1"
}
2020-10-06T17:34:23.293Z [WARN] [192.168.1.165:34028 <key2>] RTSP CLIENT SOCKET TIMEOUT
2020-10-06T17:34:23.294Z [WARN] [192.168.1.165:34028 <key2>] RTSP ENDPOINT SOCKET TIMEOUT [192.168.1.165:34028 <key2>]
2020-10-06T17:34:23.294Z [INFO] [192.168.1.165:34028 <key2>] RTSP ENDPOINT SOCKET CLOSED [192.168.1.165:34028 <key2>]
2020-10-06T17:34:23.299Z [INFO] [192.168.1.165:34028 <key2>] RTSP CLIENT SOCKET CLOSED
2020-10-06T17:34:23.299Z [INFO] [192.168.1.165:34028 <key2>] RTSP CLIENT DETACHED FROM STREAM: Front Facing East (STREAM:9941432f-959c-4efa-
2020-10-06T17:34:23.299Z [INFO] [RTSP PROXY] DE-REGISTERING STREAM [Front Facing East/PRIMARY]; NO LONGER IN USE
2020-10-06T17:34:23.300Z [DEBUG] <RTSP-PROXY> [REQUEST] --> [DEREGISTER] rtsp://192.168.1.47:554/s1
2020-10-06T17:34:23.300Z [TRACE] <RTSP-PROXY> [REQUEST] --> [HEADERS] {
"cseq": "1",
"transport": "reuse_connection=0;preferred_delivery_protocol=udp;proxy_url_suffix=STREAM:<stream>"
}
^C[RTSP Server Stopping]
[RTSP Server Stopped]
[Monocle Stopping]
[Monocle Stopped]
[ERROR] RtspProxyController::deregister() { Error: read ECONNRESET
at _errnoException (util.js:1031:13)
at TCP.onread (net.js:619:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
[ERROR] RtspProxyController::register() { Error: read ECONNRESET
at _errnoException (util.js:1031:13)
at TCP.onread (net.js:619:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
[ERROR] RtspProxyController::deregister() { Error: read ECONNRESET
at _errnoException (util.js:1031:13)
at TCP.onread (net.js:619:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
[ERROR] RtspProxyController::register() { Error: read ECONNRESET
at _errnoException (util.js:1031:13)
at TCP.onread (net.js:619:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
[ERROR] RtspProxyController::deregister() { Error: read ECONNRESET
at _errnoException (util.js:1031:13)
at TCP.onread (net.js:619:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
[ERROR] RtspProxyController::register() { Error: read ECONNRESET
at _errnoException (util.js:1031:13)
at TCP.onread (net.js:619:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
[ERROR] RtspProxyController::deregister() { Error: read ECONNRESET
at _errnoException (util.js:1031:13)
at TCP.onread (net.js:619:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
[ERROR] RtspProxyController::register() { Error: read ECONNRESET
at _errnoException (util.js:1031:13)
at TCP.onread (net.js:619:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
[ERROR] RtspProxyController::deregister() { Error: read ECONNRESET
at _errnoException (util.js:1031:13)
at TCP.onread (net.js:619:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
[ERROR] RtspProxyController::register() { Error: read ECONNRESET
at _errnoException (util.js:1031:13)
at TCP.onread (net.js:619:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
[ERROR] RtspProxyController::deregister() { Error: read ECONNRESET
at _errnoException (util.js:1031:13)
at TCP.onread (net.js:619:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
[RTSP Proxy Stopped] CODE=0; SIGNAL=SIGINT
[Monocle Disconnected]

rick

I’m having same issue with Gateway not working after a few days. Here is the log hopefully someone can assist. Fyi I’m only using one instance of gateway.

MONOCLE RTSP SERVICE - INITIALIZED

FQDN = 20fa027d-d919-4768-b031-5403dba0a00d.mproxy.io
HOST = 192.168.1.72
PORT = 443

WARNING
GATEWAY INSTANCE PROMOTED!

Another gateway instance has detached from your user account!
This could cause problems, please make sure that only one instance of Monocle Gateway is running on your network and associated with your account.

THIS GATEWAY INSTANCE [192.168.1.72] IS NOW THE ACTIVE GATEWAY FOR THIS ACCOUNT.

2020-12-04T08:57:35.849Z [ERROR] [Monocle Error] read ECONNRESET
[ ‘[Monocle Error]’, ‘read ECONNRESET’ ]
[Monocle Disconnected]
[Monocle Reconnecting] … in 8 seconds (attempt #1)
[Monocle Connecting]
[Monocle Connected]
[RTSP Server Registered]

MONOCLE RTSP SERVICE - INITIALIZED

FQDN = 20fa027d-d919-4768-b031-5403dba0a00d.mproxy.io
HOST = 192.168.1.72
PORT = 443

2020-12-04T09:20:09.977Z [ERROR] [Monocle Error] Closing connection due to PING response timeout; last PING response received was at: [2020-12-4 01:14:29]
[ ‘[Monocle Error]’,
‘Closing connection due to PING response timeout; last PING response received was at: [2020-12-4 01:14:29]’ ]
[Monocle Disconnected]
[Monocle Reconnecting] … in 5 seconds (attempt #1)
[Monocle Connecting]
2020-12-04T09:29:56.255Z [ERROR] [Monocle Error] read ECONNRESET
[ ‘[Monocle Error]’, ‘read ECONNRESET’ ]
[Monocle Disconnected]
[Monocle Reconnecting] … in 13 seconds (attempt #2)
[Monocle Connecting]
2020-12-04T09:30:09.264Z [ERROR] [Monocle Error] getaddrinfo ENOTFOUND api.monoclecam.com api.monoclecam.com:443
[ ‘[Monocle Error]’,
‘getaddrinfo ENOTFOUND api.monoclecam.com api.monoclecam.com:443’ ]
[Monocle Disconnected]
[Monocle Reconnecting] … in 18 seconds (attempt #3)
[Monocle Connecting]
2020-12-04T09:30:27.266Z [ERROR] [Monocle Error] getaddrinfo ENOTFOUND api.monoclecam.com api.monoclecam.com:443
[ ‘[Monocle Error]’,
‘getaddrinfo ENOTFOUND api.monoclecam.com api.monoclecam.com:443’ ]
[Monocle Disconnected]
[Monocle Reconnecting] … in 23 seconds (attempt #4)
[Monocle Connecting]
2020-12-04T09:31:02.316Z [ERROR] [Monocle Error] getaddrinfo ENOTFOUND api.monoclecam.com api.monoclecam.com:443
[ ‘[Monocle Error]’,
‘getaddrinfo ENOTFOUND api.monoclecam.com api.monoclecam.com:443’ ]
[Monocle Disconnected]
[Monocle Reconnecting] … in 29 seconds (attempt #5)

Gateway service stops working

MONOCLE RTSP SERVICE - INITIALIZED

FQDN = 20fa027d-d919-4768-b031-5403dba0a00d.mproxy.io HOST = 192.168.1.72 PORT = 443

MONOCLE RTSP SERVICE - INITIALIZED

FQDN = 20fa027d-d919-4768-b031-5403dba0a00d.mproxy.io HOST = 192.168.1.72 PORT = 443

FQDN = 20fa027d-d919-4768-b031-5403dba0a00d.mproxy.io
HOST = 192.168.1.72
PORT = 443

FQDN = 20fa027d-d919-4768-b031-5403dba0a00d.mproxy.io
HOST = 192.168.1.72
PORT = 443