Opencv GStreamer (windows) video streaming tutorial + full source code for RTSP HLS streaming

5 Dec, 2021

Opencv C++ simple tutorial to use GStreamer to send video to Server that converts RTSP to HLS video stream. The source code and all needed configurations are included.

Opencv is a powerful computer vision library. You can use it in production and use it for image and video processing and modern machine learning. In some applications, You may want to stream your processed video results from your C++ OpenCV app outside and not just use a simple OpenCV graphical interface. The video streaming of your results is what you are looking for. Do you want to stream processed video from your IoT device? Yes, This is mainly for Linux. Do you want to stream processed video to the Web player, broadcast the video or just use VLC to play video processed by OpenCV? You may be interested in reading the next lines.

Opencv video stream to VLC or WEB

There are basically two main options with OpenCV. The first one is to write a streaming application using FFMPEG. This is a little bit more advanced approach. I will publish an article about this topic in the future. The second option is to compile and link OpenCV with GStreamer. Then write an OpenCV application that uses the GStreamer pipeline to stream your video out.

Prerequisites/requirements for Opencv GStreamer application

It is really good to have some experience with building OpenCV with CMake. My environment is as follows:

Windows 10
CMake 3.22.0
Visual Studio 2022
OpenCV-4.5.4
rtsp-simple-server (NGINX in future texts)https://github.com/aler9/rtsp-simple-server#hls-general-usage
GStreamer MSVC 64-bit (VS 2019, Release CRT) both runtime and development installer from following links. (lead to the official website of GStreamer)

18.5 runtime installer

1.18.5 development installer

Steps to compile Opencv with Gstreamer

Since this is quite advanced, I will not go into so many details. You can still find them on my blog https://funvision.blogspot.com in posts related to GStreamer.

Install GStreamer runtime and development MSVC 64-bit installer. Links above.
Add GStreamer to system variable Path as on the picture below, The OpenCV will come later once lib is built.

Path system variables for Gstreamer windows 10 — Path system variables

Now, restart your machine to get this, Path effected for linking your project.
Download, extract OpenCV source OpenCV-4.5.4
Get Visual Studio 2022 community (or 2019), Start CMake 3.22 tested https://cmake.org/download/
Set source code as a folder, where you extract OpenCV
Set “Where to build the binaries”, where your custom-built OpenCV lib will be assembled.

Cmake opencv configuration — CMake configuration

Hit Configure button, and select VS 2022, 64-bit compiler.
This part is a little bit advanced. You have to hit configure several times and select options, where Cmake hit some problem with your project. I usually uncheck any Java, Python staff, where I am missing prerequisites or I simply do not care.
Now, Check options WITH_FFMPEG, WITH_GSTREAMER and for GStreamer add GSTREAMER_app_LIBRARY and GSTREAMER_base_Library paths, where your GStreamer installation is located. Hit CMake Configure and all other GStreamer variables will be resolved automatically. Just follow the picture below.

Build opencv with Gstreamer configuration of Cmake — CMake OpenCV Gstreamer configuration

Hit configure and resolve possible problems again.
If OK, Hit generate
If OK, Open project in Visual Studio 2022
Switch to solution configuration Release, x64.

First, build ALL_BUILD in CMakeTargets
Then build INSTALL in CMakeTarget

OpenCV lib with FFMPEG and GStreamer is ready.

Start H264 streaming from Opencv APP using GStreamer, rtsp-simple-server

The task now is to demonstrate Opencv App together with GStreamer. Let's take a notebook camera, a process by a deep neural network, and send result stream by RTSP (real-time streaming protocol) to RTSP-simple-server. The server now receives the one-to-one stream from the OpenCV app and creates an HSL stream for a wide audience inside my network. This stream can be received in a web player or for example in VLC. To publish a video stream on the Web in production, I will recommend NGINX, but for our demonstration, RTSP-simple-server is just fine.

Let's code an Opencv App with a GStreamer videowriter

There are two important parts of the code. The first one is to manage the current size of the images you are writing to VideoWriter. The second part is the correct setting of the VideoWriter pipeline. I found it necessary to put space after the last character of GStreamer pipeline configuration on Windows machine”…../mystream “←Space after mystream. To be able to successfully open this VideoWriter someone needs to listen on rtsp://localhost:8554/mystream . To achieve this download rtsp-simple-server and use the default configuration(Anyway check if the server is listening on port 8554). https://github.com/aler9/rtsp-simple-server/releases.

#include <iostream>
#include <opencv2/imgcodecs.hpp>
#include <opencv2/video.hpp>
#include <opencv2/videoio.hpp>
#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>
//Comment these two out
/*#include "detector.h"
#include "poseEstimation.h" */
using namespace cv;
int main() 
{
//VideoCapture cap(0, cv::CAP_MSMF);  //Notebook camera input
VideoCapture cap("C:/www/town.avi", cv::CAP_MSMF); // video file input
VideoWriter writer; 
// Write this string to one line to be sure!!
writer.open("appsrc ! videoconvert ! videoscale ! video/x-raw,width=640,height=480
            ! x264enc speed-preset=veryfast tune=zerolatency bitrate=800 !
             rtspclientsink location=rtsp://localhost:8554/mystream ",
              0, 20, Size(640, 480), true);
// Comment this line out
Detector dec = Detector();
Mat img;
for (;;)
   {
      if (!cap.isOpened())
      {
     std::cout << "Video Capture Fail" << std::endl;
      break;
   }
     cap.read(img); 
     // Comment detector lines out
     /* dec.ProcessFrame(img); 
     dec.detect();
     Mat proc = dec.getResult();*/
    
     cv::resize(img, img, Size(640, 480));
     cv::imshow("raw", img);
     writer << img;
     cv::waitKey(25);
    }
}

Configure project and run the code in Visual Studio 2022

This is a standard Opencv configuration. Once OpenCV is built with GStreamer, Just ensure that the system variable path correctly points to GStreamer once again.