With the boom of video doorbells with the likes of Ring, Skybell and Next doorbell cams, I came to the realization that I did not want to be cloud dependent for this type service for long term reliability, privacy and cost.
I finally found last year a wifi video doorbell which is cost effective and support RTSP and now ONVIF streaming:
The RCA HSDB2A which is made by Hikvision and has many clones (EZViz, Nelly, Laview). It has an unusual vertical aspect ratio designed to watch packages delivered on the floor....
It also runs on 5GHz wifi which is a huge advantage. I have tried running IPCams on 2.4GHz before and it is a complete disaster for your WIFI bandwidth. Using a spectrum analyzer, you will see what I mean. It completely saturates the wifi channels because of the very high IO requirements and is a horrible design. 2.4GHz gets range but is too limited in bandwidth to support any kind of video stream reliably... unless you have a dedicated SSID and channel available for it.
The video is recorded locally on my NVR. I was able to process the stream from it on Home Assistant to get it to do facial recognition and trigger automations on openLuup like any other IPCams. This requires quite a bit of CPU power to do...
I also get snapshots through push notifications through pushover on motion like all of my other IPcams. Movement detection is switched on and off by openLuup... based on house mode.
Sharing a few options for object recognition which then can be used as triggers for home automation.asmirnou/watsor asmirnou/watsor
My two favorites so far:
Object detection for video surveillance. Contribute to asmirnou/watsor development by creating an account on GitHub.skvark/opencv-python skvark/opencv-python
Automated CI toolchain to produce precompiled opencv-python, opencv-python-headless, opencv-contrib-python and opencv-contrib-python-headless packages. - skvark/opencv-python
I have optimized my facial recognition scheme and discovered a few things:
My wifi doorbell, the RCA HSDB2, was overloaded by having to provide too many concurrent rtsp streams which was causing the streams themselves to be unreliable:
stream to QNAP NVR
stream to home assistant (regular)
stream to home assistant facial recognition.
I decided to use the proxy function of the QNAP NVR to now only pull 2 streams from the doorbell and have the NVR be the source for home assistant. This stabilized the system quite a bit.
The second optimization was to find out that by default home assistant processes images every 10s. It made me think that the processing was slow but it turns out that it was just not being triggered frequently enough. I turned it up to 2s and now I have a working automation to trigger an openLuup scene, triggering opening a doorlock with conditionals on house mode and geofence. Now I am looking to offload this processing from the cpu to an intel NCS2 stick so I might test some other components than Dlib to make things run even faster.
Sharing what I have learned and some modifications to components with their benefits.Establishing and maintaining a camera stream (over rtsp or http protocol) Have the ability to extract a single frame from the open stream in order to process it Pre process using the same steps as a video frame and store in memory a predetermined number of pictures as the known people to later compare with. In reality what is being compared are arrays of numbers generated by a model. Run a face detection and localization on the frame using one model Using the resulting location of 4., Extract from the picture, the face and encode it into a array of number Run a classification or comparison between the pre-set faces and the face on the video and spit out the "inference" or "prediction" to determine if they are close enough to be the same person.
On home assistant/python3, facial recognition involves the following steps:
Even though a few components have been created on home-assistant for many years to do this, I ran into challenges which forced me to improve/optimize the process.Home Assistant's camera does not establish and keep open a stream in the background. It can open one on demand through its UI but doesn't keep it open. This forces the facial camera component to have to re-establish a new stream to get a single frame every time it needs to process an image causing up to 2s of delays, unacceptable for my application. I therefore rewrote the ffmpeg camera component to use opencv and maintain a stream within a python thread and since I have a GPU, I decided to decode the video using my GPU to relieve the CPU. This also required playing with some subtleties to avoid uselessly decoding frames we won't process while still needing to remove them from the thread buffer. The frame extraction was pretty challenging using ffmpeg which is why I opted to use opencv instead, as it executes the frame synchonization and alignment from the byte stream for us. The pre-set pictures was not a problem and a part of every face component. I started with the dlib component which had two models for ease of use. It makes use of the dlib library and the "facial_recognition" wrapper which has a python3 API but the CNN model requires a GPU and while it works well for me, turned out not to be the best as explained in this article and also quite resource intensive:https://www.learnopencv.com/face-detection-opencv-dlib-and-deep-learning-c-python/
So I opted to move to the opencv DNN algorithm instead. Home Assistant has an openCV component but it is a bit generic and I couldn't figure out how to make it work. In any case, it did not have the steps 5 and 6 I wanted. For the face encoding step, I struggled quite a bit as it is quite directly connected to what option I would chose for step 6. From my investigation, I came to this: https://www.pyimagesearch.com/2018/09/24/opencv-face-recognition/
"*Use dlib’s embedding model (but not it’s k-NN for face recognition)
In my experience using both OpenCV’s face recognition model along with dlib’s face recognition model, I’ve found that dlib’s face embeddings are more discriminative, especially for smaller datasets.
Furthermore, I’ve found that dlib’s model is less dependent on:
Preprocessing such as face alignment
Using a more powerful machine learning model on top of extracted face embeddings
If you take a look at my original face recognition tutorial, you’ll notice that we utilized a simple k-NN algorithm for face recognition (with a small modification to throw out nearest neighbor votes whose distance was above a threshold).
The k-NN model worked extremely well, but as we know, more powerful machine learning models exist.
To improve accuracy further, you may want to use dlib’s embedding model, and then instead of applying k-NN, follow Step #2 from today’s post and train a more powerful classifier on the face embeddings.*"
The trouble from my research is that I can see some people have tried but I have not seen posted anywhere a solution to translating the location array output from the opencv dnn model into a dlib rect object format for dlib to encode. Well, I did just that...For now I am sticking with the simple euclidian distance calculation and a distance threshold to determine the face match as it has been quite accurate for me but the option of going for a much more complex classification algorithm is open... when I get to it.
So in summary, the outcome is modifications to:rafale77/home-assistant rafale77/home-assistant
A. the ffmpeg camera component to switch to opencv and enable background maintenance of a stream with one rewritten file:
:house_with_garden: Open source home automation that puts local control and privacy first - rafale77/home-assistantB. Changes to the dlib face recognition component to support the opencv face detection model:
The world's simplest facial recognition api for Python and the command line - rafale77/face_recognitionD. And additions of the new model to the face_recognition library involving adding a couple of files:
Trained models for the face_recognition python library - rafale77/face_recognition_modelsinit.py
Trained models for the face_recognition python library - rafale77/face_recognition_models
Overall these changes significantly improved speed and decreased cpu and gpu utilization rate over any of the original dlib components.
At the moment the CUDA use for this inference is broken on openCV using the latest CUDA so I have not even switched on the GPU for facial detection yet (it worked fine using the dlib cnn model) but a fix may already have been posted so I will recompile openCV shortly...
Edit: Sure enough openCV is fixed. I am running the face detection on the GPU now.
At the moment i'm using the Surveillance software in Synology but I'm limited to 6 cameras (2 included and I took a 4pack a while ago)
But I have 8 cameras, so right now, 2 of them are not in the NVR!
I checked back in time motioneye but this software is very slow and all my cameras feed was lagging...
any other solution? 😉
Something fun to do if you have a camera located on your driveway:Home Assistant OpenALPR Local OpenALPR Local
This home assistant component enables recognition of a license plate which in turn could open the garage door...
Instructions on how to integrate licences plates with OpenALPR local into Home Assistant.
Sharing an excellent skill I use to locally stream from my IPCams to echo shows:Monocle
Your video stream does not need to go to the cloud. This skill just forwards the local stream address to the echo device when the camera name is called. It does require them to host the address and camera information (credentials) on their server though. I personally block all my IP cameras from accessing the internet from the router.
therealdb last edited by
@Elcid Yep, I was thinking about this. The only problem is that the video is taken exclusively from the app it's responding, so we'll see. I'll probably just run another eth cable outside and put a new cam anyway. Right now I'm too busy with my central air cooling and the pool that the doorbell has less priority than before (and that's because we had 2 months of complete lock down, so the only visitor was couriers/postmen). The doorbel per se is not bad => https://www.youtube.com/watch?v=8LG-Of--JU8
prophead last edited by
The device type is not sticking, it keeps getting reset to 0
prophead last edited by
@prophead answering my own question here.
I'm very pleased I figured out how to use Pushover for notifications with images.
because luup.inet.wget cannot do post I used curl to grab the images(s) and another curl to post the image(s) to pushover. Each pushover notification can only have one image so when I need multiple camera angles like for the doorbell I just send two notifications. Works great! Man, OpenLuup is a great piece of software. Just great.
akbooer last edited by akbooer
Really glad things are working well for you.
As mentioned earlier, the module you’d need to use to do this directly in Lua would be LuaSocket.
However, if you now have something that works, why change? Just for future reference, then!
@Elcid Hi, have the sensor, I will place outside in UV and waterproof case. As below:
I have read the ebay ad very well and I have not yet tested it. But I leave the light sensor to OFF, hold time to 5 secs and only questions are:
- Is the radar sensor on the front or the back? Thus does it sense "through" the letters or the back of the device?
- The "cat" problem... I will also put it around 20cm above floor level outside and put around 100% initially which is equivalent to 10m. "how" and what have you done? Like this:
For this to work I would need to know exactly where the sensor is
Elcid last edited by Elcid
@sender I have not opened that model but i do have one. it has a 360 degree field that seems to come out the top, which is next to the led.
Before connecting do some testing on the outputs as i beleive it is labeled incorrectly. i.e. L/N are reversed, check my image.
This one is in a waterproof box by back gate and faces car port.
As to cats put metal at bottom and front as shown in your diagram. To add strong winds moving bushes can also set it off.
Your setting are. on on off off off off
cw-kid last edited by
I also have the Ezviz DB1 video doorbell.
I wrote about the wiring and installation on my blog. I used an Everspring door contact sensor with binary inputs to also get a trigger into Vera.
Overall pleased with it, but the doorbell can sometimes drop off the Ezviz app / their cloud and say offline sometimes, even though I can still ping and access it locally on my LAN and the local Rtsp stream also still works.
This happens occasionally after a power cut or when my Internet has gone off. I have to reset the doorbell camera and run through its setup again to recover.
rafale77 last edited by
I had this problem early on, a couple of times but haven't seen it for at least a year. I posted the process to recover it on the IPcam forum which can be tricky if you changed the default credentials. I also have not upgraded firmware since the installation. It's been rock stable for me.
I'm watching the Foscam VD1 for my integration. It has all my requirements, but there are still some problems with the software unfortunately. The webhook for a trigger does not work as expected.
BTW I found out that the Google also a new API for their cameras. Has anyone tried to integrate the Nest Doorbell (or another Nest cam) in openluup?