Elasticsearch:language ingest processor - 7.6

在之前的我的一篇文章“Elasticsearch:language ingest processor”。在那篇文章中,我们使用了一个社区的插件来完成这个工作。 在Elastic Stack 7.6的发布中,我们很欣喜地告诉大家:在Elastic Stack中,我们集成了一个语言识别的processor。

 

动手实践

首先,我们需要安装最新的Elastic Stack 7.6的发行版。我们在Kibana中打入如下的命令:

PUT _ingest/pipeline/my-pipeline-id
{
  "description": "describe pipeline",
  "processors": [
    {
      "inference": {
        "model_id": "lang_ident_model_1",
        "inference_config": {
          "classification": {
            "num_top_classes": 1
          }
        },
        "field_mappings": {}
      }
    }
  ]
}

在上面我们创建了一个叫做my-pipeline-id的pipeline。那么当我们把文档导入到Elasticsearch中时,我们可以引用这个pipeline:
 

POST test/_doc/1?pipeline=my-pipeline-id
{
  "text": "我爱北京天安门"
}

在上面我们把 “我爱北京天安门”写入到Elasticsearch中。我们可以通过如下的命令来检查写入的信息:

GET test/_doc/1/

我们可以看到如下的一个响应:

{
  "_index" : "test",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 2,
  "_seq_no" : 1,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "text" : "我爱北京天安门",
    "ml" : {
      "inference" : {
        "top_classes" : [
          {
            "class_name" : "zh",
            "class_probability" : 0.9999770918986735,
            "class_score" : 0.9999770918986735
          }
        ],
        "predicted_value" : "zh",
        "model_id" : "lang_ident_model_1"
      }
    }
  }
}

在上面,我们可以看到有一个叫做ml.inference.predicated_value的字段,它显示的语言是zh,也就是中文。

同样我们可以对其它的语言来做一个实验,比如日文:

POST test/_doc/2?pipeline=my-pipeline-id
{
  "text": "これはフリーテキストです。"
}

那么我们通过如下的命令来检查输入的文档:

GET test/_doc/2/

上面的返回的结果是:

{
  "_index" : "test",
  "_type" : "_doc",
  "_id" : "2",
  "_version" : 2,
  "_seq_no" : 3,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "text" : "これはフリーテキストです。",
    "ml" : {
      "inference" : {
        "top_classes" : [
          {
            "class_name" : "ja",
            "class_probability" : 0.999997979528696,
            "class_score" : 0.999997979528696
          }
        ],
        "predicted_value" : "ja",
        "model_id" : "lang_ident_model_1"
      }
    }
  }
}

在上面的ml.inference.predicted_value显示的是“ja”,也就是日文。

如果我们想使用pipeline的_simulate来测试的话,那么我们可以使用如下的命令:

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "inference": {
          "model_id": "lang_ident_model_1",
          "inference_config": {
            "classification": {
              "num_top_classes": 1
            }
          },
          "field_mappings": {
          }
        }
      }
    ]
  },
  "docs": [
    {"_source": {"text": "this is some free text."}},
    {"_source": {"text": "c'est du texte libre."}},
    {"_source": {"text": "Dies ist ein Freitext."}},
    {"_source": {"text": "これはフリーテキストです。"}},
    {"_source": {"text": "یہ کچھ مفت متن ہے۔"}},
    {"_source": {"text": "我爱北京天安门"}}
    ]
}

显示的结果是:

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "text" : "this is some free text.",
          "ml" : {
            "inference" : {
              "top_classes" : [
                {
                  "class_name" : "en",
                  "class_probability" : 0.9999832919481261,
                  "class_score" : 0.9999832919481261
                }
              ],
              "predicted_value" : "en",
              "model_id" : "lang_ident_model_1"
            }
          }
        },
        "_ingest" : {
          "timestamp" : "2020-03-04T05:51:26.380985Z"
        }
      }
    },
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "text" : "c'est du texte libre.",
          "ml" : {
            "inference" : {
              "top_classes" : [
                {
                  "class_name" : "fr",
                  "class_probability" : 0.996991483545992,
                  "class_score" : 0.996991483545992
                }
              ],
              "predicted_value" : "fr",
              "model_id" : "lang_ident_model_1"
            }
          }
        },
        "_ingest" : {
          "timestamp" : "2020-03-04T05:51:26.380988Z"
        }
      }
    },
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "text" : "Dies ist ein Freitext.",
          "ml" : {
            "inference" : {
              "top_classes" : [
                {
                  "class_name" : "de",
                  "class_probability" : 0.9999995801281829,
                  "class_score" : 0.9999995801281829
                }
              ],
              "predicted_value" : "de",
              "model_id" : "lang_ident_model_1"
            }
          }
        },
        "_ingest" : {
          "timestamp" : "2020-03-04T05:51:26.38099Z"
        }
      }
    },
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "text" : "これはフリーテキストです。",
          "ml" : {
            "inference" : {
              "top_classes" : [
                {
                  "class_name" : "ja",
                  "class_probability" : 0.999997979528696,
                  "class_score" : 0.999997979528696
                }
              ],
              "predicted_value" : "ja",
              "model_id" : "lang_ident_model_1"
            }
          }
        },
        "_ingest" : {
          "timestamp" : "2020-03-04T05:51:26.380992Z"
        }
      }
    },
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "text" : "یہ کچھ مفت متن ہے۔",
          "ml" : {
            "inference" : {
              "top_classes" : [
                {
                  "class_name" : "ur",
                  "class_probability" : 0.9999762917090316,
                  "class_score" : 0.9999762917090316
                }
              ],
              "predicted_value" : "ur",
              "model_id" : "lang_ident_model_1"
            }
          }
        },
        "_ingest" : {
          "timestamp" : "2020-03-04T05:51:26.380994Z"
        }
      }
    },
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "text" : "我爱北京天安门",
          "ml" : {
            "inference" : {
              "top_classes" : [
                {
                  "class_name" : "zh",
                  "class_probability" : 0.9999770918986735,
                  "class_score" : 0.9999770918986735
                }
              ],
              "predicted_value" : "zh",
              "model_id" : "lang_ident_model_1"
            }
          }
        },
        "_ingest" : {
          "timestamp" : "2020-03-04T05:51:26.380995Z"
        }
      }
    }
  ]
}

 

展开阅读全文

没有更多推荐了,返回首页

应支付0元
点击重新获取
扫码支付

支付成功即可阅读