ES使用记录
官方文档:Elasticsearch Guide [8.4] | Elastic 官方中文:序言 | Elasticsearch: 权威指南 | Elastic 社区文档:Getting Started(入门指南) - elasticsearch中文文档
一. ES安装相关
1. 使用docker安装es 和kibana
docker pull elasticsearch:7.4.2 存储和检索数据 docker pull kibana:7.4.2 可视化检索数据
2. 创建对应的ES挂载文件
mkdir -p /mydata/elasticsearch/config mkdir -p /mydata/elasticsearch/data mkdir -p /mydata/elasticsearch/plugins echo "http.host: 0.0.0.0" >> /mydata/elasticsearch/config/elasticsearch.yml
3. 添加selinux规则,将要挂载的目录添加到白名单
chcon -Rt svirt_sandbox_file_t /mydata/elasticsearch/config chcon -Rt svirt_sandbox_file_t /mydata/elasticsearch/data chcon -Rt svirt_sandbox_file_t /mydata/elasticsearch/plugins
4. 启动es
docker run --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e ES_JAVA_OPTS="-Xms128m -Xmx256m" -v /mydata/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /mydata/elasticsearch/data:/usr/share/elasticsearch/data -v /mydata/elasticsearch/plugins:/usr/share/elasticsearch/plugins -d elasticsearch:7.4.2
二 kibana安装相关
1. 启动kibana
#192.168.206.130这个ip地址换成虚拟机的ip地址。 docker run --name kibana -e ELASTICSEARCH_HOSTS=http://192.168.206.130:9200 -p 5601:5601 -d kibana:7.4.2
2. 启动问题
启动后kibana报server is not ready yet,使用docker logs kibana 查看启动日志。排查错误
常见ip地址找不到或链接错误:
docker inspect elasticsearch,查看es容器内部的ip地址,如172.17.0.5
将kibana启动地址改为 -e ELASTICSEARCH_HOSTS=http://172.17.0.5:9200 即可解决
三 ES使用记录
1. es常用查询
GET /_cat/nodes:查看所有节点 GET /_cat/health:查看 es 健康状况 GET /_cat/master:查看主节点 GET /_cat/indices:查看所有索引 类似mysql show databases
2. es更新时使用乐观锁
-
查出数据对应的版本号
{ "_index": "customer", //在哪个索引 "_type": "external", //在哪个类型 "_id": "1", //记录 id "_version": 2, //版本号 "_seq_no": 1, //并发控制字段,每次更新就会+1,用来做乐观锁 "_primary_term": 1, //同上,主分片重新分配,如重启,就会变化 "found": true, "_source": { //真正的内容 "name": "John Doe" } } 更新
-
更新时带上?if_seq_no=0&if_primary_term=1
192.168.101.164:9200/gulimall/order/1?if_seq_no=6&if_primary_term=1 更新失败会返回409
3. 查询示例
1.match查询
GET bank/_search { "query": { "match": { "firstname": "mill lane" #匹配指定字段包含指定值 包含mill 或lane的 } }, "sort":{ "balance":{ "order":"desc" #通过余额降序排序 } }, "from": 0, #分页 从第一条开始 "size": 20, #分页 取20条数据 "_source": ["balance","firstname"] #查询指定字段 }
2. match_phrase 短语匹配
GET bank/_search { "query": { "match_phrase ": { "firstname": "mill lane" #匹配指定字段包含指定值 包含mill lane ,不会将mill lane分词 } }, "sort":{ "balance":{ "order":"desc" #通过余额降序排序 } }, "from": 0, #分页 从第一条开始 "size": 20, #分页 取20条数据 "_source": ["balance","firstname"] #查询指定字段 }
3. text文本检索 使用match 数值查询使用term
4. should查询
应该达到 should 列举的条件,如果达到会增加相关文档的评分,并不会改变 查询的结果。如果 query 中只有 should 且只有一种匹配规则,那么 should 的条件就会 被作为默认匹配条件而去改变查询结果
## 匹配到genger包含M的数据,会提升score值 GET bank/_search { "query": { "bool": { "must": [ {"term": { "age": "32" }}, { "match": { "address": "Place" } } ], "should": [ {"match": { "gender": "M" }} ] } } }
## 只有should查询参数,则会将该条件作为匹配查询参数 GET bank/_search { "query": { "bool": { "should": [ {"match": { "gender": "M" }} ] } } }
5. 复杂聚合查询
#聚合查询 查出所有年龄分布,并且这些年龄段中 M 的平均薪资和 F 的平均薪资以及这个年龄 段的总体平均薪资 GET bank/_search { "query": { "match_all": {} }, "size": 0, "aggs": { "ageAggs": { "terms": { "field": "age", "size": 100 }, "aggs": { "genderAggs": { "terms": { "field": "gender.keyword" }, "aggs": { "balanceAggs": { "avg": { "field": "balance" } } } }, "balanceAgeAvg": { "avg": { "field": "balance" } } } } } }
6. es数据迁移
#数据迁移 POST _reindex { "source": { "index": "源索引库", "type": "account(源类型)" }, "dest": { "index": "目标索引库" } }
7. 手动创建映射mapping
字段映射类型在官网可以查看
1. 创建索引并指定映射
#创建索引并指定映射 PUT /my-index { "mappings": { "properties": { "age": { "type": "integer" }, "email": { "type": "keyword" }, "name": { "type": "text" } } } }
2. 添加新字段映射
PUT /my-index/_mapping { "properties": { "employee-id": { "type": "keyword", "index": false } } }
3.更新映射
对于已经存在的映射字段,我们不能更新。更新必须创建新的索引进行数据迁移
4. 查看映射
GET my-index/_mapping
四 ik分词器安装
1. 安装
安转完ik分词器后,修改ik分词器的权限
chmod -R 777 ik/
进入es 内部的bin目录elasticsearch-plugin list查看是否安装成功
elasticsearch-plugin list
2. 测试分词器
1. ik_smart测试
#测试分词器 POST _analyze { "analyzer": "ik_smart", "text": "我们都有一个家" } #结果 { "tokens" : [ { "token" : "我们", "start_offset" : 0, "end_offset" : 2, "type" : "CN_WORD", "position" : 0 }, { "token" : "都有", "start_offset" : 2, "end_offset" : 4, "type" : "CN_WORD", "position" : 1 }, { "token" : "一个", "start_offset" : 4, "end_offset" : 6, "type" : "CN_WORD", "position" : 2 }, { "token" : "家", "start_offset" : 6, "end_offset" : 7, "type" : "CN_CHAR", "position" : 3 } ] }
2. ik_max_word测试
#测试分词器 POST _analyze { "analyzer": "ik_max_word", "text": "我们都有一个家" } #结果 { "tokens" : [ { "token" : "我们", "start_offset" : 0, "end_offset" : 2, "type" : "CN_WORD", "position" : 0 }, { "token" : "都有", "start_offset" : 2, "end_offset" : 4, "type" : "CN_WORD", "position" : 1 }, { "token" : "一个", "start_offset" : 4, "end_offset" : 6, "type" : "CN_WORD", "position" : 2 }, { "token" : "一", "start_offset" : 4, "end_offset" : 5, "type" : "TYPE_CNUM", "position" : 3 }, { "token" : "个", "start_offset" : 5, "end_offset" : 6, "type" : "COUNT", "position" : 4 }, { "token" : "家", "start_offset" : 6, "end_offset" : 7, "type" : "CN_CHAR", "position" : 5 } ] }
结果:
能够看出不同的分词器,分词有明显的区别,所以以后定义一个索引不能再使用默认的 mapping 了,要手工建立 mapping, 因为要选择分词器。
五 springboot使用ES
目前最新版本为7.17,此处使用的7.4.2为项目中曾经的最新版本,仅作一个项目记录
1. 依赖添加
<dependency> <groupId>org.elasticsearch.client</groupId> <artifactId>elasticsearch-rest-high-level-client</artifactId> <version>7.4.2</version> </dependency>
2. 配置es连接客户端
参考ES7.4配置,其他版本的文档可以通过选择java rest client版本进行查看
@Configuration public class EsSearchClientConfig { public static final RequestOptions COMMON_OPTIONS; //共享请求配置 static { RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder(); builder.addHeader("Authorization", "Bearer " + "token"); builder.setHttpAsyncResponseConsumerFactory( new HttpAsyncResponseConsumerFactory .HeapBufferedResponseConsumerFactory(1024 * 1024 * 1024)); COMMON_OPTIONS = builder.build(); } //配置高版本客户端链接 @Bean public RestHighLevelClient restHighLevelClient() { RestHighLevelClient client = new RestHighLevelClient( RestClient.builder( //地址 端口 连接方式 new HttpHost("192.168.101.164", 9200, "http"))); return client; } }
3. 使用
3.1 添加
如果不存在索引,则会自动创建。如果不存在保存对象的字段,也会自动创建
//goodsInfoList为需要保存的实体类集合 public void saveData(List<GoodsInfo> goodsInfoList) { //创建请求 BulkRequest bulkRequest = new BulkRequest(); if (!CollectionUtils.isEmpty(goodsInfoList)) { goodsInfoList.forEach(s -> { //指定索引goods IndexRequest request = new IndexRequest("goods"); //将商品信息转为json字符串存入 request.source(JSON.toJSONString(s), XContentType.JSON); bulkRequest.add(request); }); } try { //保存数据 BulkResponse bulk = restHighLevelClient.bulk(bulkRequest, EsSearchClientConfig.COMMON_OPTIONS); System.out.println(bulk.toString()); } catch (IOException e) { log.error("保存es失败,错误信息==" + e.getMessage()); } }
3.2 查询
.... //省略业务处理代码 //构建查询req SearchRequest searchRequest = new SearchRequest(); //Without arguments this runs against all indices. 不传该参数,则会从所有索引中查询 searchRequest.indices(EsConstant.PRODUCT_INDEX); searchRequest.source(getSearchSourceBuilder(param)); SearchResult searchResult = new SearchResult(); SearchResponse response = restHighLevelClient.search(searchRequest, GulimallSearchClientConfig.COMMON_OPTIONS); ....... /** * remark:处理ESL查询语句 */ public SearchSourceBuilder getSearchSourceBuilder(SearchParam param){ SearchSourceBuilder builder = new SearchSourceBuilder(); //构建boolQuery BoolQueryBuilder boolQuery = QueryBuilders.boolQuery(); //1.关键词查询 if (!StringUtils.isEmpty(param.getKeyword())){ boolQuery.must(QueryBuilders.matchQuery("skuTitle",param.getKeyword())); } //2 过滤条件 //2.1 三级分类id查询 if (!StringUtils.isEmpty(param.getCatalog3Id())){ boolQuery.filter(QueryBuilders.termQuery("catalogId",param.getCatalog3Id())); } //2.2 品牌id查询 if (!CollectionUtils.isEmpty(param.getBrandId())){ boolQuery.filter(QueryBuilders.termsQuery("brandId",param.getBrandId())); } //2.3 品牌参数查询 嵌入式查询 if (!CollectionUtils.isEmpty(param.getAttrs())){ for (String attrStr : param.getAttrs()) { BoolQueryBuilder nestedBoolQuery = QueryBuilders.boolQuery(); String[] split = attrStr.split("_"); nestedBoolQuery.must(QueryBuilders.termQuery("attrs.attrId",split[0])); nestedBoolQuery.must(QueryBuilders.termsQuery("attrs.attrValue",split[1].split(":"))); boolQuery.filter(QueryBuilders.nestedQuery("attrs",nestedBoolQuery , ScoreMode.None)); } } //2.4 价格区间 if (!StringUtils.isEmpty(param.getSkuPrice())){ String[] split = param.getSkuPrice().split("_"); RangeQueryBuilder skuPrice = QueryBuilders.rangeQuery("skuPrice"); if (split.length>1){ skuPrice.gte(split[0]); skuPrice.lte(split[1]); }else { if (param.getSkuPrice().startsWith("_")){ skuPrice.gte(split[0]); }else { skuPrice.lte(split[0]); } } boolQuery.filter(skuPrice); } boolQuery.filter(QueryBuilders.matchQuery("hasStock",param.getHasStock()==1)); builder.query(boolQuery); //3 排序 if (!StringUtils.isEmpty(param.getSort())){ String[] split = param.getSort().split("_"); builder.sort(split[0],split[1].equalsIgnoreCase("asc")? SortOrder.ASC:SortOrder.DESC); } //4 分页 builder.from((param.getPageNum()-1)*EsConstant.PRODUCT_PAGESIZE); builder.size(EsConstant.PRODUCT_PAGESIZE); //高亮字段 if (!StringUtils.isEmpty(param.getKeyword())){ HighlightBuilder highlightBuilder = new HighlightBuilder(); highlightBuilder.field("skuTitle"); highlightBuilder.preTags("<b style='color:red'>"); highlightBuilder.postTags("</b>"); builder.highlighter(highlightBuilder); } //聚合查询 TermsAggregationBuilder brandIdAggs = AggregationBuilders.terms("brandAggs").size(10).field("brandId"); brandIdAggs.subAggregation(AggregationBuilders.terms("brand_name_aggs").size(10).field("brandName")); brandIdAggs.subAggregation(AggregationBuilders.terms("brand_img_aggs").size(10).field("brandImg")); builder.aggregation(brandIdAggs); TermsAggregationBuilder cateLogAggs = AggregationBuilders.terms("cateLogAggs").field("catalogId").size(10); cateLogAggs.subAggregation(AggregationBuilders.terms("catalogNameAggs").field("catalogName").size(10)); builder.aggregation(cateLogAggs); NestedAggregationBuilder nestedAggs = AggregationBuilders.nested("attrIdAggs", "attrs"); nestedAggs.subAggregation(AggregationBuilders.terms("attrIdAggs").field("attrs.attrId").size(10)); nestedAggs.subAggregation(AggregationBuilders.terms("attrNameAggs").field("attrs.attrName").size(10)); nestedAggs.subAggregation(AggregationBuilders.terms("attrValueAggs").field("attrs.attrValue").size(10)); builder.aggregation(nestedAggs); System.out.println("DLS--"+builder.toString()); return builder; }
3.3更多
更多查询参考 查询api
注意选择自己使用的版本。。。