Image Recognition

Overview

Image Recognition is a core capability offered by Clobotics Retail Cloud Service. To proceed with image recognition, follow these steps:

1. Get Scene Code: Use the scene_code to guide our system through the appropriate artificial intelligence image recognition pipeline.

2. Create Task: Generate a task for a single picture or a stitching picture.

3. Get Image Recognition (IR) Result: Manually retrieve results through the Get IR Result API or set a callback_url when creating the task. The callback data schema please refer IR Result. For more details about callback notification, please refer to the Callback Notification section.

Notes:

• To obtain the scene_code, you can retrieve all scenes through the Get Scene List API or manually obtain it from our Customer Success Manager (CSM).

Get Scene List

Description

Retrieve all pre-defined scenes, or specify a name with fuzzy query support to search for a specified scene.

Request Config

Config	Definition
URL	/ir/scene/list?name=<scene_name>
Method	GET

Params

Name	Type	Description	Required
name	string	scene name keywords	No

Response

Schema of data

Name	Type	Description
results	[]object	Scene list
+ code	string	Scene code
+ name	string	Scene name
+ description	string	Scene description

Response Body Sample

{
  "code": 0,
  "message": "",
  "data": {
    "results": [
      {
        "name": "shelf scene",
        "code": "ST-1-9",
        "description": "this is the description of shelf scene"
      }
    ]
  }
}

Create Single Picture Task

Description

To create a single picture image recognition task, specify request_id as task_id with a recommended random uuid value.

Request Config

Config	Definition
URL	/ir/recognize
Method	POST
Content-Type	application/json

Request Body Schema

Name	Type	Description	Required
request_id	string	Request Id，you should guarantee uniqueness(please use uuid），no more than 36 chars in length(lowercase)。 If need to retry, ensure the request id is unchanged	Yes
image_url	string	Image url after uploading by upload api	Yes
scene_code	string	Scene code	Yes
username	string	Visitor account	No
processed_image_url	string	The url of the processed image. Deprecated, can be replaced by setting the ext_info field	No
ext_info	string	Extra information，1000 length maximum. For related information, please refer to Task extra information	No
callback_url	string	Automatic result notification callback url. For more details about callback notification, please refer to the Callback Notification and Callback Notification#IR Result sections.	No
plan_id	string	Visit plan id，Refer to Create Visit Plan	No

Request Body Sample

{
    "request_id":"c0c80b91-694c-4f6e-8232-1a039a2731a0",
    "image_url":"https://f-dev.clobotics.cn/4/0040ca4d85f5e5ef7354102f486ef843.jpg",
    "scene_code":"104",
    "processed_image_url": "https://f-dev.clobotics.cn/4/0040ca4d85f5e5ef7354102f486ef877.jpg",
    "ext_info": "{\"key\":\"value\"}",
    "callback_url": "https://www.clobotics.cn/callback",
    "plan_id": "P2506030702016564724"
}

Response

Schema of data

Name	Type	Description
task_id	string	Task id

Response Body Sample

{
  "code": 0,
  "message": "",
  "data": {
    "task_id": "c0c80b91-694c-4f6e-8232-1a039a2731a0"
  }
}

Create Stitching Picture Task

Description

To create a stitching picture image recognition task, specify request_id as task_id with a recommended random uuid value. And also need to provide sub_task_ids, stitching_info and stitching_info_version.

Notes:

• sub_task_ids are the request_id / task_id of all sub-tasks when calling Create Single Picture Task.
• Ensure the correct sequence of these sub-tasks when Camera SDK returns images because the order must align with the corresponding order in stitching_info.
• stitching_info is output by Camera SDK. Refer to our SDK demo project to use the correct stitching info and avoid making any changes.
• If the sequence is not maintained in the correct order, our cloud service cannot stitch the multiple images.

Request Config

Config	Definition
URL	/ir/stitching
Method	POST
Content-Type	application/json

Request Body Schema

Name	Type	Description	Required
request_id	string	Request Id，you should guarantee uniqueness(please use uuid），no more than 36 chars in length(lowercase)。 If need to retry, ensure the request id is unchanged	Yes
sub_task_ids	[]string	Sub-task ids array，the number of subtasks cannot be less than 2，sub-tasks' order must match with stitching_info. a sub-task id can only belong to one stitching picture task, Creating different stitching picture tasks using one sub-task id will fail with an error	Yes
stitching_info	string	stitching information，returned by APP SDK，see APP SDK Documentation for details. Note: Required when stitching_type is empty or 5。	No
stitching_info_version	int32	stitching info version 0: Classic Stitching 1: Video Stitching 3: Ultimate Video Stitching 4: Yolo Video Stitching returned by APP SDK, see APP SDK Documentation for details. Note: Required when stitching_type is empty or 5。	No
username	string	Visitor account	No
processed_image_url	string	The url of the processed image. Deprecated, can be replaced by setting the ext_info field	No
ext_info	string	Extra information，1000 length maximum. For related information, please refer to Task extra information	No
callback_url	string	Automatic result notification callback url. For more details about callback notification, please refer to the Callback Notification and Callback Notification#IR Result sections.	No
plan_id	string	Visit plan id，Refer to Create Visit Plan	No
stitching_type	int32	Stitching type，default 5. 3: Simple Stitching(All images are directly stitched together at the height of the first image) 4: Multiple single image deduplication without stitching 5: Normal Stitching .	No

Request Body Sample

{
    "request_id":"b5fed073-d66a-451f-93c0-ea412d7543e8",
    "sub_task_ids":["2b2182a0-9cab-41fd-94c6-4692880a690b","2b2182a0-9cab-41fd-94c6-4692880a690b"],
    "stitching_info":"{\"stitchingInfo\":{\"bestPov\":0,\"pair\":[{\"image1Index\":0,\"image2Index\":1,\"pixelCorrespondence\":[],\"homography\":[1.0064635,-0.095225796,70.384,0.03407375,0.98476404,1.7139097,5.052459E-5,-4.1571122E-5,1]},{\"image1Index\":1,\"image2Index\":2,\"pixelCorrespondence\":[],\"homography\":[0.9902558,-0.02223142,69.341,-0.032897126,1.0594548,-21.226934,-0.00014316723,0.0002628908,1]},{\"image1Index\":2,\"image2Index\":3,\"pixelCorrespondence\":[],\"homography\":[0.9452166,-0.06867031,80.68806,-0.0006102832,0.95038676,5.128851,-0.000043942106,-0.000053319545,1]},{\"image1Index\":3,\"image2Index\":4,\"pixelCorrespondence\":[],\"homography\":[0.99139684,-0.05744228,80.414955,-0.020089298,1.006871,-2.0042527,-0.00008886192,0.00003245073,1]},{\"image1Index\":4,\"image2Index\":5,\"pixelCorrespondence\":[],\"homography\":[0.9550804,-0.064189814,81.82138,-0.02587876,0.97099406,6.294556,-0.00011758251,-0.000049771486,1]},{\"image1Index\":5,\"image2Index\":6,\"pixelCorrespondence\":[],\"homography\":[0.9919336,-0.08287014,81.64814,0.013544433,1.0054679,-2.8793876,-0.000057586047,0.000024841353,1]},{\"image1Index\":6,\"image2Index\":7,\"pixelCorrespondence\":[],\"homography\":[1.0070611,-0.065085396,67.927826,0.009846879,0.9966087,-0.24726827,0.000089298584,-0.0000139677,1]}],\"resize\":0.16666666666666666,\"retryHomo\":0}}",
    "stitching_info_version":0,
    "processed_image_url": "https://f-dev.clobotics.cn/4/0040ca4d85f5e5ef7354102f486ef877.jpg",
    "ext_info": "{\"key\":\"value\"}",
    "callback_url": "https://www.clobotics.cn/callback",
    "plan_id": "P2506030702016564724",
    "stitching_type": 1
}

Response

Schema of data

Name	Type	Description
task_id	string	Stitching picture task id

Response Body Sample

{
  "code": 0,
  "message": "",
  "data": {
    "task_id": "b5fed073-d66a-451f-93c0-ea412d7543e8"
  }
}

Get IR Result

Description

This API is for manually retrieving the image recognition result for any task submitted via Create Single Picture Task or Create Stitching Picture Task APIs. The response schema is the same for both.

Notes:

• For automatic result notifications, you can set callback_url when calling Create Single Picture Task or Create Stitching Picture Task.

Request Config

Config	Definition
URL	/ir/result/<task_id>?row_sort=desc
Method	GET

Path

Name	Type	Description	Required
task_id	string	Task Id	Yes

Params

Name	Type	Description	Required
row_sort	string	Sorting rule for rows: desc=rows are sorted from top to bottom, numbered n ~ 0; asc=rows are sorted from top to bottom, numbered 1 ~ n. Default is desc.	No
detail	bool	Whether detailed information is needed. If true, the response will include UPC, etc. Defaults to false.	No

Response

Schema of data

Name	Type	Description
status	string	Task result，PENDING，FAIL，SUCCESS
task_id	string	Task id
row_sort	string	Sorting rule for rows: desc=rows are sorted from top to bottom, numbered n ~ 0; asc=rows are sorted from top to bottom, numbered 1 ~ n.
update_time	int64	Update time（millisecond timestamp）
results	[]object	Results data, length of either single or stitching picture task is 1。
+ image_url	string	IR task imageUrl
+ processed_image_url	string	The url of the processed image.
+ sku_list	[]object	Sku list
+ + sku_id	string	Sku id
+ + sku_name	string	Sku name
+ + sku_count	int64	Sku count
+ + category	string	Sku category
+ + manufacturer	string	Sku manufacturer
+ + flavor	string	Sku flavor
+ + brand	string	Sku brand
+ + sub_brand	string	Sku sub brand
+ + pack_type	string	Sku pack type
+ + pack_size	string	Sku pack size
+ + sub_category	string	Sku sub category
+ + unit	string	Sku unit
+ + area	float64	Total sku area
+ + total_width	float64	Sku total width
+ + upc	string	UPC
+ locations	[]object	Sku location
+ + box_id	int64	Box id
+ + box_type	int32	Box type，0=unknown 1=model 2=embedding 10=price tag 11=POSM 23=promotion .
+ + sku_id	string	Sku id
+ + linked_box_id	int64	Linked box id. Currently only the price tag box is supported, but it will be deprecated in the future
+ + linked_box_ids	[]int64	Linked box ids, such as price tag, POSM, promotion etc.
+ + price	string	Price
+ + position	object	Sku position
+ + + index	int32	Position index
+ + + row	int32	Row index，shown in the row and column diagram
+ + + col	int32	Column index，shown in the row and column diagram
+ + + stacking	int32	Stacking information，0=no stacking，if exist，value will in 1~n
+ + bounding_box	object	Bounding box
+ + + x_min	float64	Top-left x coordinate, shown in the box diagram
+ + + y_min	float64	Top-left y coordinate, shown in the box diagram
+ + + x_max	float64	Bottom-right x coordinate, shown in the box diagram
+ + + y_max	float64	Bottom-right y coordinate, shown in the box diagram
+ qualities	[]object	Image quality，see for details Task-level qualities
+ + id	string	Quality id
+ + name	string	Quality name
+ + value	string	Quality value
+ metrics	[]object	Metric item, see for details Task-level metrics
+ + id	string	Metric id
+ + name	string	Metric name
+ + value	string	Metric value
ext_info	string	Extra info, same as ext_info field in create ir task request

• bounding_box Definition

Assuming that a 350X640 photo is taken, the real coordinates corresponding to (1,1) are (350,640), and the real coordinate position of x_min, x_max, y_min, and y_max needs to be calculated according to the corresponding proportion, and the box diagram is as follows:

• position Definition

The diagram of the rows and columns is as follows:

Notes:

• The number of columns per row in the recognition results is not guaranteed to be exactly the same
• This is a schematic diagram when row_sort is desc. When row_sort is asc, it should be 0~6 from top to bottom.

Response Body Sample

{
  "code": 0,
  "message": "",
  "data": {
    "status": "SUCCESS",
    "task_id": "b5fed073-d66a-451f-93c0-ea412d7543e8",
    "update_time": 1668478888000,
    "results": [
      {
        "image_url": "https://f-dev.clobotics.cn/4/0040ca4d85f5e5ef7354102f486ef843.jpg",
        "processed_image_url": "https://f-dev.clobotics.cn/4/0040ca4d85f5e5ef7354102f486ef877.jpg",
        "sku_list": [
          {
            "sku_id": "1047441",
            "sku_name": "Coke Original Sparkling Can 180 ml",
            "sku_count": 7,
            "brand": "Coke",
            "category": "Sparkling",
            "manufacturer": "Coca-Cola",
            "flavor": "",
            "sub_brand": "Original",
            "pack_type": "Can",
            "pack_size": "180",
            "sub_category": "",
            "unit": "ML",
            "area": 12.28,
            "total_width": 10.14,
            "upc": "3700050773"
          }
        ],
        "locations": [
          {
            "sku_id": "1044059",
            "bounding_box": {
              "x_min": 0.46711602807044983,
              "y_min": 0.7578532695770264,
              "x_max": 0.481109082698822,
              "y_max": 0.771846354007721
            },
            "position": {
              "index": 0,
              "row": 1,
              "col": 2,
              "stacking": 0
            },
            "box_id": 1,
            "box_type": 0,
            "linked_box_id": 0,
            "linked_box_ids": [1],
            "price": "2.52"
          },
          {
            "sku_id": "1044059",
            "bounding_box": {
              "x_min": 0.5359848141670227,
              "y_min": 0.7501513361930847,
              "x_max": 0.5499778389930725,
              "y_max": 0.7641443610191345
            },
            "position": {
              "index": 0,
              "row": 1,
              "col": 3,
              "stacking": 0
            },
            "box_id": 2,
            "box_type": 0,
            "linked_box_id": 0,
            "linked_box_ids": [1],
            "price": "1.53"
          }
        ],
        "qualities": [
          {
            "id": "23",
            "name": "test"
          }
        ],
        "metrics": [
          {
            "id": "3",
            "name": "Total facing",
            "value": "38"
          },
          {
            "id": "2",
            "name": "Total empty space",
            "value": "23"
          },
          {
            "id": "5",
            "name": "Total row",
            "value": "5"
          },
          {
            "id": "9",
            "name": "Total door",
            "value": "1"
          }
        ]
      }
    ]
  }
}

General Information Note

Task extra information

Additional information about the task can be set through the ext_info field, in the format of a serialized json string;. ext_info contains reserved fields, which can be used for functions such as SKU mapping in image recognition results.

Notes：

• Reserved fields should be avoided when defining fields in ext_info.

Reserved fields

参数名	类型	描述
store_name	string	Store name
store_code	string	Store code
store_address	string	Store address
store_channel	string	Store channel
store_retailer	string	Store retailer
store_region	string	Store region
store_bottler	string	Store bottler
store_sales_office	string	Store sales office
store_route	string	Store route

Reserved fields Sample

{
  "store_name": "store name",
  "store_code": "H1216",
  "store_address": "store address",
  "store_channel": "channel",
  "store_retailer": "retailer",
  "store_region": "region",
  "store_bottler": "bottler",
  "store_sales_office": "sales office",
  "store_route": "route"
}