Warehouse Monitoring algorithms are developed by MRDVS for use in the warehousing and logistics industry. The Warehouse Monitoring algorithm runs inside the camera without the need for an external industrial computer.

The software only requires the user to calibrate the external parameters of the camera and draw multiple storage locations, so that a single camera can be used to supervise 6-8 storage locations. Multiple cameras can be deployed in the same warehouse scenario for detection. The algorithm can automatically obtain the calibration results through the external parameter calibration.

This document mainly introduces MRDVS Warehouse Monitoring deployment and use.

2. Components of the warehouse monitoring system

The system consists of the M/S/T series depth cameras and a set of warehouse monitoring software.

The system includes mid360 radar and warehouse monitoring software, and a set of industrial computers.

3. Camera Hardware Deployment Guide

3.1 Type of camera used

Camera Type

M / S Series Cameras

文件

Camera User Manual⬇️M Series ToF Camera S series obstacle avoidance camera

https://zcnt5g3o2989.feishu.cn/space/api/box/stream/download/asynccode/?code=NDE5NGYzNmJhMjdlZmQ1MTllMzU3OGMxNTUwYWI3ZmVfbzNOTkt0ZVpteG14VUFMbDg5V3hCNHVibWU4SEREVzNfVG9rZW46SHhkZWIwcXlFb084UE54VGc2RWM3d1NNbk1nXzE3NDE1OTE1MDk6MTc0MTU5NTEwOV9WNA

3.2 Radar

Type of radar

mid360

3.3 Camera mounting method

The camera needs to be secured by a bracket, and the camera can be mounted using the mounting method and camera bracket shown below.The following figure shows the mounting schematic.

Figure 4.1 Installation Schematic

3.4 Camera Connection

The camera is 24V powered and connected via a Category 6 / 7 network cable. After the camera has been installed and deployed, the Warehouse Monitoring PC software is opened via a computer to calibrate the camera's external parameters and set the detection range. Algorithms run inside the camera in real-time.

Multiple cameras are deployed and need to be connected via Gigabit switches. The cameras are mounted to a fixed position via a mounting bracket according to the mounting height planned in the CAD to ensure that the cameras are firmly fixed. After the cameras are installed, connect them to a 24V regulated DC power supply and use the corresponding network cable to connect to the switch or master control.

Once the preparation is complete the warehouse within the camera's field of view is set up with goods and the software is deployed.

4. Camera Software Installation Guide

4.1 Software Installation Process

The first time you use the software, you can refer to the following process to complete the operation.

4.2 Warehouse Monitoring Host Computer Operating Environment Requirements

It is recommended that the operational Warehouse Monitoring host computer devices meet the requirements.

Operating system	Windows 10 and above
CPU	Recommended Intel® Core™ i5 7th generation and above
RAM	8G and above
GPU	Intel(R) UHD Graphics/Core or above

4.3 Installation of the camera's host computer software

The camera host software can be accessed and downloaded through the MRDVS Knowledge Base. Click to download→Software download link

Double-click to install the Lanxin-MRDVS host computer software, which needs to be installed outside the c-disk.

After selecting the language to use, click [OK], select an installation directory other than the C drive, and click [Next].

Check [Create Desktop Shortcut] and click [Next].

Click the [Install] button to start installing the software.

After installation, you can run the camera's host computer software.

4.4 Installation of warehouse inspection PC software

Copy the obtained Warehouse Monitoring software to the camera's host computer \Lanxin-MRDVS\Tools.

When replacing, select [Replace files in target].

Double-click and tap to run the software.

In the left device list column, the camera status is [Unknown] indicating that the camera does not have a Warehouse Monitoring algorithm。

4.5 Warehouse Monitoring Software Upgrade

Warehouse Monitoring software upgrades by simply replacing the “.exe” file.

4.6 Algorithm upgrade

Camera algorithm upgrade requires opening the camera's host software and double-clicking [LxCameraViewer].

Click the [Open] button to view camera imaging.

Click Basic Tools → Function Setting → [Firmware Upgrade] to brush the Warehouse Monitoring algorithm. (For the algorithm, please find the technical support staff to get it)

After clicking the [Execute] button, select “Warehouse Monitoring Algorithm” and click the [Open] button to start the algorithm swiping.

During the algorithm flushing process, the camera will reboot and the camera ID will disappear from the device list on the left side, and when it is displayed again, it means the flushing is successful.

At this point, the Warehouse Monitoring software is opened again and the device list shows green.

5. Warehouse Monitoring Software Interface

5.1 Warehouse Monitoring Program Starts

Double-click the [ LxApplicationViewer.exe ] program to start the Warehouse Monitoring software. Click the online camera in the left device list to open the camera image.

The current user selects Administrator, no need to enter the password, directly click to confirm to enter Administrator Mode (User Mode can't set Warehouse Monitoring parameters). Administrator mode will have additional warehouse monitoring and camera configuration options.

5.2 Basic tools

The basic tools mainly include user switching, ideal ground, external reference ground, display cache, algorithm display, export and save map, history playback, and the status of warehouse monitoring.

Basic tools	Functional meaning
User switching	Switch permissions for user permission: only view the test results. Permission for administrator: camera configuration, warehouse monitoring parameter configuration, etc.
Ideal Ground	The floor is referenced to the horizontal line and can be closed.
External reference calibration ground	The horizontal plane of the camera's external reference calibration height, which can be turned off.
Display Cache	No modifications are required.
Display Algorithm	The algorithm is currently being used.
Export to save image	Export stored images.
History Replay	View image logs for RGB and TOF.
Warehouse Monitoring results display	Displays the test results and detects the presence or absence of goods.

5.3 Global configuration

Switches the camera operating mode.

5.4 Warehouse Monitoring

Warehouse Monitoring Global Configuration	Functional meaning
Algorithm version	Current algorithm version information.
AI Filtering	Supports semantics to segment people without detection.
Detection frequency	Algorithm detection frequency
Angle of alignment	Beyond this degree, the detection result will output 2 (2 means an object is in the warehouse, but the angle deviates). Unit: °
Alignment validity	When the alignment check is turned on, it detects the current object angle deviation and whether it is outside the drawn detection area.
Time domain filtering	Filtering parameters, no modifications are required.
Minimum number of points	An object is considered to be present when its point cloud is larger than the minimum number of points.
Saved and distributed	Save the set parameters and send them down to the camera.

Warehouse Monitoring Area Configuration	Functional meaning
Increase in storage space	Add Warehouse Monitoring area.
Deletion of storage positions	Delete the Warehouse Monitoring area.
Current frame	Box the current Warehouse Monitoring area.
Minimum X	Minimum value in the X direction in the Warehouse Monitoring box.
Maximum X	Maximum value in the X direction in the Warehouse Monitoring box.
Minimum Y	Minimum value in the Y direction in the Warehouse Monitoring box.
Maximum Y	Maximum value in the Y direction in the Warehouse Monitoring box.
Minimum Z	Minimum value in the Z direction in the Warehouse Monitoring box.
Maximum Z	Maximum value in the Z direction in the Warehouse Monitoring box.
ID	The name of the Warehouse Monitoring box. When multiple Warehouse Monitoring areas are drawn, they can be differentiated by modifying the ID name.
Drag the mouse to adjust the area	In the adjustment area, press and hold the left button to drag the Warehouse Monitoring area to pan in the x and y direction. Press and hold the right button to move the Warehouse Monitoring area in the z-direction.

Warehouse Monitoring Advanced Functions - Warehouse Frame Copies	Functional meaning
Copy the current frame	The currently selected Warehouse Monitoring area can be copied.
Copy panning needs to be known	Copy Panning Content Tip
Orientations	Copy Warehouse Monitoring area direction. There are three directions, X, Y, and Z.
Interval	Warehouse Monitoring area and copy area spacing.
Quantities	Number of copy warehouse monitoring areas.
Execute	After clicking the Execute button, the copy duplicates multiple identically spaced Warehouse Monitoring areas.

Warehouse Monitoring Advanced Features - QR Code Calibration	Functional meaning
QR Code Requirements for Calibration	QR Code calibration content prompt.
X	Storage widths
Y	Storage lengths
Z_MIN	Minimum depth of storage.
Z_MAX	Maximum depth of storage.
Calibration	Click the Calibration button to calibrate the storage for QR code verification.

5.5 Camera Configuration

Camera settings	Functional meaning
Camera name	Modify the name of the current camera. When there is more than one, it can be judged by the camera name.
Simulation mode	For internal use, the user does not need to open it.
Pictures storage mode	Turn on algorithmic image storage to store image data inside the camera.
Fetch Mode	Fetch mode is divided into MDS and SDK, using MDS will not occupy the camera permissions, you can open the camera host computer normally, using SDK will occupy the camera permissions, when running the algorithm, you can not open the host computer.
x	Camera calibrated x position
y	Camera calibrated y position
z	Camera calibrated height above ground z
roll	Angle of camera calibration
pitch	Angle of camera calibration
yaw	Angle of camera calibration
Ideal Ground	Default
Saved and distributed	After the external parameter calibration is completed, save it by clicking the [Save and Distribute] button, otherwise, the parameter will not take effect.
Reading configuration from the camera	Read current configuration parameters from the camera
External parameter calibration	Calibrates the camera height from the ground, camera roll, pitch, and yaw angles.

5.6 Languages

Switch between English and Chinese.

5.7 Status line

Status line	Functional meaning
RGB	Check to view RGB images
Point cloud	Check to view point cloud image
Stopping algorithm	After clicking Stop Algorithm, the algorithm will not start until the next time the camera is powered up.
Upgrade Firmware	Algorithmic firmware upgrades available
Reboot the camera	Reboot the camera.

6. Warehouse Monitoring Deployment Processes

After opening the Warehouse Monitoring software, the default is RGB map, click to switch to point cloud map, the image will be black, you need to move the mouse to the image display area, through the mouse wheel up and down, hold down the left mouse button to move, before the point cloud data will be displayed.

6.1 Camera Network Configuration

Take the Win10 system as an example, double-click the Control Panel

Go to the Control Panel and click on "View Network Status and Tasks".

Go to the Network and Sharing Center interface and click "Change adapter settings".

Enter the network connection interface and double-click the mouse to select the Ethernet the camera is connected to.

Check the Ethernet status (Gigabit ports are recommended) and click the Properties button.

Enter the Ethernet Properties interface, find "Internet Protocol Version 4 (TCP/IPv4)", and double-click to open.

Enter the Internet Protocol version 4 (TCP/IPv4) interface and select "Use the following IP address" to write the IP address, subnet mask, and default gateway.

IP Address Setting: 192.168.100.xxx

Subnet Mask: 255.255.255.0

Default Gateway: 192.168.100.1, the following figure set up the IP, set up after the completion of the click "OK" button.

After clicking the OK button, you will be automatically returned to the previous screen, click "OK" again to complete the network configuration.

6.2 Firewall shutdown confirmation

Once the camera is securely mounted, the first step in deployment requires that the computer's firewall be turned off.

Win10 system, as an example, double-click the desktop of the [Control Panel], enter and click [System and Security].

Click the [Windows Defender Firewall] button to enter the Firewall page.

Click the [Enable or Disable Windows Defender Firewall] button to enter the Firewall On/Off page.

Select [Turn off Windows Defender Firewall] for both private and public networks and click the [Confirm] button to turn off the firewall.

6.3 Warehouse Monitoring - External Reference Calibration

After the camera IP configuration is completed and the firewall is turned off, the second step of the deployment is to open the software for algorithm external parameter calibration, which is required to calibrate the reference ground specific and camera installation angle deviation.

Click Camera Configuration to switch to the Parameter Calibration interface, before performing external parameter calibration, make sure x, y, z, roll, pitch, and yaw are 0. If it is non-zero, you need to return all the parameters to 0 and click the [Save and Distribute] button, and then re-perform the external parameter calibration after the parameter is saved successfully.

After confirming that the parameters are all 0, click on the [external parameter calibration] button. After successful calibration, the software will pop up a prompt box. You can see the calibrated camera parameters in the right parameter column.

For Warehouse Monitoring, you need to set “pitch” to the calibrated “positive pitch angle needs to be minus 180°, negative pitch angle needs to be plus 180°”, and ensure that the blue arrow Z direction is facing up, and the ground x and y coordinate axes are level with the ground. For example, the pitch of this external reference calibration is 1.695 - 180° = 178.305°. Fill this number into the pitch and flip the coordinate system.

Click the [Save and Distribute] button, the camera calibration parameters are saved successfully. The next step is to set the Warehouse Monitoring parameters (Note: Be sure to set the pitch angle, otherwise the height will be deviated).

6.4 Warehouse Monitoring - Global Settings

Click Warehouse Monitoring to switch to the Warehouse Monitoring parameter setting page. In the global configuration, you need to pay attention to the alignment angle, alignment validity, and minimum number of points.

If there is an angle requirement for the object placed in the warehouse monitoring, you can turn on the alignment validation and set the alignment angle. If the angle of the object exceeds the set angle value, the corresponding result will be output. (No object: 0, object: 1, object angle over alignment angle: 2, object out of warehouse monitoring range: 3)

If you don't want to recognize small goods in the warehouse, or you are concerned about misrecognition, you can filter them by setting the “Minimum Number of Points”. This quantity is the number of point clouds.

After the global configuration settings are completed, you have to click the [Save and Distribute] button to save the parameters to the camera.

6.5 Warehouse Monitoring - Warehouse Configuration

To set the detection range and adjust the detection length and width (xy), it is recommended to switch to the upper and lower views first, as follows are the shortcut keys for each view.

With the pallet in view, click the [Add a storage space] button, and below the storage configuration, the number of storage spaces and the ID name of the storage space will be added. We can modify the ID, currently it is “ Storage Test 1”.

After clicking on “Add storage”, the point cloud will be switched to the lower view, click the mouse on the point cloud image, and press the [X] key on the keyboard to draw the area of the storage, when you press the [X], the “Add Storage” button will be gray, which means that you can draw the storage in the image display area. (Keyboard x key position to start the storage space drawing and the number of sections of the storage space drawing function, click to start drawing, and then click to end the drawing)

At this time, you can zoom in on the image by scrolling the wheel sliding, holding down the left mouse button, framing the whole pallet with the drawn rectangular box, and then releasing the left mouse button, the current Warehouse Monitoring area will be automatically drawn. Press the keyboard [X] button to cancel the detection area drawing. At this time, you can adjust the maximum and minimum X, Y, and Z to confirm the range of region size (after drawing the warehouse, if the region is framed with no problem, be sure to press [X] to cancel the warehouse drawing. If not canceled, it may lead to the failure of the painted warehouse).

Switch to the lower view through the keyboard shortcut, as shown in the following figure, the red arrow indicates the x-direction of the detection range, and the mouse wheel can be scrolled to modify the corresponding value to complete the adjustment of the detection range length and width. The current x-direction is slightly larger than the width of the pallet.

The green arrow indicates the y-direction of the detection range, and the mouse wheel can be scrolled to modify the corresponding value to complete the adjustment of the detection range length and width. The current y-direction is slightly larger than the length of the pallet.

The blue arrow points to the z-direction of the detection range, the mouse wheel scrolling can modify the value, and you can also directly enter the value, the current detection range of the minimum z-value is recommended to be set slightly higher than the ground distance, to prevent misdetection of the ground.

After configuring the detection area, click the [Save and Distribute] button to save the parameters inside the camera, and you can start detecting whether there are any goods in the current Warehouse Monitoring area.

6.6 Warehouse Monitoring - Advanced Settings

6.6.1 Advanced Functions - Copy Cargo Location

Click [Copy Panning Requirements], panning related information will be prompted. Copying the storage space is convenient for users to design the storage space according to the CAD drawing in the actual deployment. Drawing a storage space area, you can make multiple copies, without the need to manually draw in the re-drawing.

The current settings are X-direction, Pitch is 10, and Quantity is 2. Clicking the Execute button prompts, whether or not to confirm the execution of the copy panning.

After clicking yes, two storage spaces of the same size will be generated in the x-positive direction with a spacing of 10, and you can see in the Storage Spaces list box that the generated storage space area is based on the original Warehouse Monitoring ID in the suffix added (0), (1).

Set to y-direction, spacing is -40, and quantity is 3. Clicking the Execute button will generate 3 storage spaces of the same size in the y-negative direction. The storage space list box ID adds storage spaces (2), (3) and (4) to the original (0) and (1).

After copying the storage space, you need to click the [Save and Distribute] button to save the parameter, otherwise the parameter is not effective

6.6.2 Advanced function - QR calibration (two-dimensional code calibration)

QR (two-dimensional code) calibration, which can simulate the actual storage space deployment scenarios, continues the automatic drawing of the storage space area without manual drawing, allowing users to easily deploy. It is recommended that the QR code board is larger than 1m x 1m. Click [QR Code calibration requirements], the calibration process will be prompted.

Place the QR code calibration plate on the target area and switch to RGB image mode to ensure that the QR code is visible in the RGB field of view.

Once the QR code is in the center of the field of view, switch to point cloud image mode and find the [QR Code] calibration field. Pre-enter the size of the storage space X (width of storage space), Y (length of storage space), Z_min (lowest depth of storage space), and Z_max (highest depth of storage space).

Click the [Calibration] button again, the storage space can be drawn automatically based on the X, Y, Z set by QR, and can be viewed through the storage space configuration area box.QR_BOX is the area box set by the QR code.

After the QR Code calibration is completed, you need to click the [Save and Distribute] button, otherwise the parameters will not take effect.

6.7 Warehouse Monitoring Results Viewing

After setting the calibration, Warehouse Monitoring parameters, and storage space monitoring area, click “Basic Tools” and put an object into the monitoring area to view the result. If there is no object, it will show no object in the basic function column.

After placing the pallet, the storage space status display bar will show that the current storage space is in stock, and at the bottom of the software, it will show the Result, with 1 representing the availability of stock.

After turning on the alignment validation, at the bottom of the software, the result will be displayed, 2 means that there is a pallet but there is an angular offset.

After turning on the alignment validation, at the bottom of the software, the result will be displayed, 3 means that there is a pallet but it is beyond the Warehouse Monitoring space. The red box indicates that the storage space monitoring area is 0°, and the green box indicates the actual object point cloud.

6.8 Warehouse Monitoring Log View

When there is a change in the storage space status, click on the history playback, click on the left side of the time you need to view, you can view the time of the effective field of view of the object within the scope of change. The following figure shows the goods moved away.

6.9 Storage Space Monitoring Applications

A brief description of the actual application, drawing two storage space monitoring areas to detect the presence or absence of goods and pallets. One detects the presence or absence of pallets and the other detects the presence or absence of goods.

Through the obtained result, the status of the goods is judged to detect whether the angle of the goods is too large and whether the goods are beyond the detection area. As shown in the following figure, the alignment angle is set to 15°. When the object exceeds 15, the result outputs a result of 2, which means that the angle is too large.

In the case of an alignment angle of 15°, the object is placed horizontally, beyond the detection area, and the output result is 3, which means that it is brought out of the detection area.

7. Communications protocol

7.1 TCP communication

Warehouse Monitoring results can be obtained through TCP communication. Through the network debugging assistant, remote camera IP: 192.168.100.82, port: 14951, send RESULT_UPDATE to see the storage space monitoring results returned.

7.2 UDP communication

Warehouse Monitoring results can be obtained through UDP communication. Through the network debugging assistant, remote camera IP: 192.168.100.82, port: 14950, send RESULT_UPDATE to see the storage space monitoring result information returned.

7.3 HTTP communication

Warehouse Monitoring results can be obtained via HTTP communication on port 14952.

The upper-level system (WMS) sends a result to get a signal to the vision, which returns a string in JSON format.

WMS → Vision	ㅤ
Result	Meaning
RESULT_UPDATE	Trigger visual result to return signal
Vision → WMS	ㅤ
Result	Meaning
IP	Data Source Camera IP
SS	Storage space status, 0 is empty, 1 is occupied
SS_DATA_TIME	Storage Space Monitoring Data Extraction Time
SS_RES_TIME	Storage Space Monitoring Result Output Time
nick_name	Storage space camera name, configurable within the camera
ret_code	Return code, 0 for success