2020年5月17日

使用SCF Trigger，30行代码搞定 Ckafka To Elasticsearch

昨天刚刚才开始研究Ckafka To Elasticsearch这个方案，如有不太对的地方还请大佬们指教。首先，我们来介绍下概念和场景部分：

SCF触发器：云函数SCF是典型的事件触发（Event-Triggered）形态的无服务器运行环境，核心组件是 SCF 函数和事件源。其中，事件源是发布事件（Event）的腾讯云服务或用户自定义代码，SCF 函数是事件的处理者，而函数触发器就是管理函数和事件源对应关系的集合。

Kafka： Apache Kafka 消息队列引擎，提供高吞吐性能、高可扩展性的消息队列服务。

Elasticsearch ：Elasticsearch Service（ES）是基于开源搜索引擎 Elasticsearch 构建的高可用、可伸缩的云端托管 Elasticsearch 服务。Elasticsearch 是一款分布式的基于 RESTful API 的搜索分析引擎，可以应用于日益增多的海量数据搜索和分析等应用场景。

众所周知，在日志处理场景中最常见的其实是ELK （Elasticsearch、Logstash、Kibana）, 腾讯云的Elasticsearch其实已经集成了Elasticsearch、Kibana 那在ELK这个使用场景下缺的应该是一个Logstash的角色，如果可以的话我们为什么不用SCF Ckafka的trigger来替代原有的Logstash呢？ 其实原理很简单，在trigger上拉到数据，然后原封不动传给Elasticsearch就可以了。

说干就干，千里之行始于足下，我们先搂一眼 Elasticsearch 的SDK：

感觉还是比较清爽的，不过这里选择版本的时候必须选择高于当前使用 Elasticsearch 版本的 SDK 否则会报406错误，血的教训。

我这里操作的话使用的 Elasticsearch 7.x 版本的 Python SDK 链接在这里：https://pypi.org/project/elasticsearch7/

Elasticsearch 对SDK的封装特别简单，基本就是对API简单包了一层，其实自己请求 POST 也是可以满足需求的，当然如果自己愿意在去封装的话。。

这里我直接用的 腾讯云基础版本的Elasticsearch服务作为演示，自建也是同理，只要搞对VPC网络就可以了，画个重点这块需要云函数与Elasticsearch同地域同VPC同子网。

搞完之后我们先看下怎么连接到腾讯云的ES把，腾讯云Es连接需要在Elasticsearch函数的参数中设置如下3个参数关闭节点嗅探：

sniff_on_start=False
sniff_on_connection_fail=False
sniffer_timeout=None

这里具体应该是这样的，填上自己的es信息就好了，简单的单条插入是这么写的：

from elasticsearch import Elasticsearch

es = Elasticsearch(["http://xx.xx.xx.xx:9200"],
      http_auth=('user', 'passwd'),
          sniff_on_start=False,
          sniff_on_connection_fail=False,
          sniffer_timeout=None)

res = es.index(index="my_index", doc_type="my_type", id=1, body={"title": "One", "tags": ["ruby"]})
print(res)

千万注意与云函数同地域，不然这块一定会报错。

事情远远没有那么简单，我们考虑的是ELK海量的处理场景，所以这块直接For循环es.index命令一定没法满足我们的要求，之前实验过顺序向es的my_index索引（该索引已存在）写入100条文档，却花费了大约7秒左右，这种速度在大量数据的时候，肯定不行。

后来，想到一个办法通过elasticsearch模块导入helper，通过helper.bulk来批量处理大量的数据。首先我们将所有的数据定义成字典形式，各字段含义如下：

_index对应索引名称，并且该索引必须存在。
_type对应类型名称。
_source对应的字典内，每一篇文档的字段和值，可有有多个字段。

首先将每一篇文档（组成的字典）都整理成一个大的列表，然后，通过helper.bulk(es, action)将这个列表写入到es对象中。然后，这个程序要执行的话——你就要考虑，这个一千万个元素的列表，是否会把你的内存撑爆（MemoryError）！很可能还没到没到写入es那一步，却因为列表过大导致内存错误而使写入程序崩溃！代码如下：

    for i in range(1, 100001, 1000):
        action = ({
            "_index": "my_index",
            "_type": "doc",
            "_source": {
                "title": k
            }
        } for k in range(i, i + 1000))
        helpers.bulk(es, action)

最后的最后，找到了一个方法，将生成器交给es侧去处理，不在函数处理！这样，Python的压力更小了，不过这里的es压力会更大，无论是分批处理还是使用生成器，es的压力都不小，写入操作本来就耗时嘛：

def Taobrss():
    """ 使用生成器批量写入数据 """
    action = ({
        "_index": "my_index",
        "_type": "doc",
        "_source": {
            "title": i
        }
    } for i in range(100000))
    helpers.bulk(es, action)

那么，得到的demo应该是这样的：

#!/usr/bin/python
# -*- coding: UTF-8 -*-
from datetime import datetime
from elasticsearch import Elasticsearch
from elasticsearch import helpers

esServer = "http://172.16.16.53:9200"  # 修改为 es server 地址+端口 E.g. http://172.16.16.53:9200
esUsr = "elastic" # 修改为 es 用户名 E.g. elastic
esPw = "Cc*******" # 修改为 es 密码 E.g. PW2312321321
esIndex = "pre1"  # es中已经创建的 index ，可以直接通过 es.indices.create(index='my-index111')

# ... or specify common parameters as kwargs
es = Elasticsearch([esServer],
      http_auth=(esUsr, esPw),
          sniff_on_start=False,
          sniff_on_connection_fail=False,
          sniffer_timeout=None)

def timer(func):
    def wrapper(*args, **kwargs):
        start = time.time()
        res = func(*args, **kwargs)
        print('共耗时约 {:.2f} 秒'.format(time.time() - start))
        return res

    return wrapper

def main_handler(event, context):
    action = ({      
        "_index": esIndex,
        "_source": {
            "msgBody": record["Ckafka"]["msgBody"] # 获取 Ckafka 触发器 msgBody
        }
    } for record in event["Records"])  # 获取 event Records 字段 数据结构 https://cloud.tencent.com/document/product/583/17530 
    print(action)  
    helpers.bulk(es, action)
    return("successful!")

在触发器拿到 event Records 字段 ，搞到ES 完事大吉。触发器这块的设置如下，可以直接在Ckafka消息转储功能中选择通用模板：

CKafka 实例：配置连接的 CKafka 实例，仅支持选择同地域下的实例。
Topic：支持在 CKafka 实例中已经创建的 Topic。
最大批量消息数：在拉取并批量投递给当前云函数时的最大消息数，目前支持最高配置为10000。结合消息大小、写入速度等因素影响，每次触发云函数并投递的消息数量不一定能达到最大值，而是处在1 – 最大消息数之间的一个变动值。
起始位置：触发器消费消息的起始位置，默认从最新位置开始消费。支持最新、最开始、按指定时间点三种配置。
重试次数：函数发生运行错误（含用户代码错误和 Runtime 错误）时的最大重试次数。

看一眼log和kibana，确认上传信息万事大吉：

代码下载：点击下载

Author:
Filed Under: Work

490 thoughts on “使用SCF Trigger，30行代码搞定 Ckafka To Elasticsearch”

  • Thanks for the recommendations you have shared here. Something else I would like to convey is that pc memory requirements generally go up along with other improvements in the technologies. For instance, if new generations of processors are introduced to the market, there is certainly usually an equivalent increase in the type demands of all computer system memory and also hard drive room. This is because the software program operated by these cpus will inevitably increase in power to make new technological innovation.

    回复

  • I used to be recommended this web site through my cousin. I am no longer positive whether this submit is written by him as no one else know such unique approximately my problem. You’re incredible! Thank you!

    回复

  • Can I simply say what a aid to find somebody who really is aware of what theyre speaking about on the internet. You positively know tips on how to convey an issue to mild and make it important. More people must learn this and understand this side of the story. I cant consider youre no more popular since you positively have the gift.

    回复

  • Thanks for the suggestions you have contributed here. One more thing I would like to convey is that laptop or computer memory needs generally rise along with other breakthroughs in the engineering. For instance, whenever new generations of processors are introduced to the market, there is certainly usually a similar increase in the size preferences of both the computer system memory plus hard drive space. This is because the software operated by means of these processor chips will inevitably surge in power to use the new technologies.

    回复

  • Coming from my research, shopping for electronics online can for sure be expensive, yet there are some principles that you can use to acquire the best deals. There are often ways to find discount specials that could help make one to come across the best electronic devices products at the lowest prices. Good blog post.

    回复

  • Coming from my investigation, shopping for electronics online may be easily expensive, nevertheless there are some guidelines that you can use to help you get the best things. There are constantly ways to come across discount promotions that could help to make one to hold the best consumer electronics products at the lowest prices. Interesting blog post.

    回复

  • In my opinion that a property foreclosures can have a major effect on the client’s life. Real estate foreclosures can have a Six to ten years negative affect on a borrower’s credit report. A new borrower having applied for a home loan or almost any loans for instance, knows that the particular worse credit rating is actually, the more difficult it is to get a decent personal loan. In addition, it may affect a new borrower’s chance to find a reasonable place to lease or rent, if that gets to be the alternative real estate solution. Interesting blog post.

    回复

  • Hi there! This post could not be written any better! Reading through this post reminds me of my previous roommate! He always kept talking about this. I most certainly will send this information to him. Fairly certain he’ll have a good read. Thank you for sharing!

    回复

  • Hello There. I found your blog using msn. This is an extremely well written article. I will be sure to bookmark it and come back to read more of your useful info. Thanks for the post. I will certainly return.

    回复

  • I have observed that in the world today, video games are definitely the latest phenomenon with kids of all ages. There are times when it may be unattainable to drag your son or daughter away from the video games. If you want the best of both worlds, there are many educational games for kids. Great post.

    回复

  • excellent post, very informative. I wonder why the other specialists of this sector don’t notice this. You must continue your writing. I am confident, you have a great readers’ base already!

    回复

  • Good ? I should certainly pronounce, impressed with your web site. I had no trouble navigating through all the tabs as well as related information ended up being truly simple to do to access. I recently found what I hoped for before you know it at all. Quite unusual. Is likely to appreciate it for those who add forums or something, website theme . a tones way for your client to communicate. Excellent task..

    回复

  • I was just looking for this info for a while. After six hours of continuous Googleing, at last I got it in your web site. I wonder what is the lack of Google strategy that don’t rank this kind of informative sites in top of the list. Usually the top sites are full of garbage.

    回复

  • Just want to say your article is as amazing. The clarity on your publish is just nice and that i could suppose you are a professional in this subject. Fine along with your permission allow me to seize your feed to stay up to date with impending post. Thank you a million and please carry on the enjoyable work.

    回复

  • Virtually all of what you mention is supprisingly accurate and it makes me wonder the reason why I hadn’t looked at this with this light before. This particular piece really did turn the light on for me as far as this specific issue goes. Nevertheless at this time there is actually one point I am not really too cozy with so whilst I make an effort to reconcile that with the main theme of your point, permit me observe what all the rest of your readers have to point out.Very well done.

    回复

  • You are so interesting! I do not think I’ve truly read through a single thing like this before. So great to find somebody with a few original thoughts on this issue. Seriously.. thanks for starting this up. This site is one thing that is required on the internet, someone with some originality.

    回复

  • I’ve learned some important things as a result of your post. I would also like to express that there may be a situation where you will apply for a loan and don’t need a cosigner such as a Federal Student Aid Loan. However, if you are getting a borrowing arrangement through a standard banker then you need to be willing to have a cosigner ready to help you. The lenders will probably base that decision on the few components but the biggest will be your credit worthiness. There are some lenders that will furthermore look at your job history and determine based on that but in almost all cases it will depend on your report.

    回复

  • Interesting article. It is quite unfortunate that over the last decade, the travel industry has had to take on terrorism, SARS, tsunamis, bird flu virus, swine flu, plus the first ever real global downturn. Through it all the industry has really proven to be strong, resilient in addition to dynamic, locating new tips on how to deal with trouble. There are always fresh complications and opportunity to which the marketplace must once more adapt and act in response.

    回复

  • Simply want to say your article is as astonishing. The clarity in your post is just excellent and i could assume you are an expert on this subject. Fine with your permission let me to grab your RSS feed to keep updated with forthcoming post. Thanks a million and please carry on the gratifying work.

    回复

  • Through my examination, shopping for electronic devices online can for sure be expensive, although there are some how-to’s that you can use to obtain the best bargains. There are often ways to uncover discount specials that could help make one to come across the best consumer electronics products at the lowest prices. Great blog post.

    回复

  • Hi there, just became aware of your blog through Google, and found that it is truly informative. I?m gonna watch out for brussels. I will appreciate if you continue this in future. Many people will be benefited from your writing. Cheers!

    回复

  • Attractive section of content. I just stumbled upon your blog and in accession capital to claim that I acquire actually loved account your blog posts. Any way I will be subscribing on your feeds or even I success you get right of entry to consistently fast.

    回复

  • Hi, Neat post. There’s a problem with your web site in internet explorer, would test this? IE still is the market leader and a good portion of people will miss your wonderful writing because of this problem.

    回复

  • I am no longer positive the place you’re getting your information, but good topic. I must spend some time finding out more or figuring out more. Thank you for excellent information I used to be on the lookout for this information for my mission.

    回复

  • Undeniably imagine that which you stated. Your favorite justification appeared to be at the net the simplest thing to understand of. I say to you, I certainly get annoyed even as other folks consider issues that they plainly do not know about. You managed to hit the nail upon the highest and also outlined out the entire thing with no need side-effects , people can take a signal. Will probably be again to get more. Thanks

    回复

  • There are actually loads of details like that to take into consideration. That could be a great level to deliver up. I offer the ideas above as basic inspiration however clearly there are questions just like the one you deliver up where crucial factor will likely be working in trustworthy good faith. I don?t know if best practices have emerged round issues like that, but I’m sure that your job is clearly identified as a fair game. Each girls and boys really feel the affect of only a second?s pleasure, for the rest of their lives.

    回复

  • Thanks for another informative blog. Where else could I get that type of info written in such an ideal way? I’ve a project that I am just now working on, and I have been on the look out for such info.

    回复

  • It is appropriate time to make some plans for the future and it’s time to be happy. I’ve read this post and if I could I wish to suggest you few interesting things or tips. Maybe you can write next articles referring to this article. I wish to read more things about it!

    回复

发表评论

邮箱地址不会被公开。 必填项已用*标注

− 6 = 4

Copyright © 2014-2023 CanMeng.

  • 本站由 提供计算与安全服务.

  • TOP