关于FaaS异步队列重试能力建设的二三事

经常用FaaS能力的同学肯定遇到过事件重试,目前为止SCF除了两个常见的出口触发器 APIGW/CLB和一个流式kafka同步触发器,大部分触发器其实都是都是异步的。如何保证异步触发器在出现各种报错的情况下合理的被重试和消费是这篇文章探索的主题。为了找到答案,几乎有段时间都是在焦虑中度过的,研发的挑战,客户的挑战,成本的压力,非准大客户完全相悖的使用习惯,几乎能被挑战的点都被我们趟了个遍。功能也马上就上线了,就用这篇日志分享下FaaS场景下,异步队列重试我是如何思考与建设相关能力的吧。

 

工欲善其事。我们先来讨论下腾讯云云函数在异步调用场景下的常见报错:

第一类 4xx,这里包括400(无效请求),430(用户代码执行错误),432(并发超限),438(函数关停)等等报错

第二类 5xx,这里主要就是500(内部错误),532(计算资源不足)

这么看上去其实很明确,也就两类报错内容。对比,参考友商后第一版的重试策略如下:

4xx错误:帮助用户重试两次,如果配置DLQ(死信队列)后将失败的信息投递至死信队列,如果没有配置DLQ则该事件会直接被丢弃。

5xx错误:考虑5xx一般为系统原因导致错误,故相关事件指数退避重试24小时,24小时后如果还未恢复,该事件将会被丢失。同样DLQ依然对5xx生效。

 

第一版重试策略沿用了很久,直到云函数将并发上移到用户可配置后,矛盾点爆发了。

首先,是对默认4xx重试两次的挑战,有的用户明确表示自己不希望做任何重试。这点很好解决,我们将默认4xx重试的次数交给用户,用户可以从 0-2 中任意选择所需的重试次数。

其次,是对SLA的挑战,在之前的重试队列是没有任何淘汰机制的,打个比方如果用户将保留配额(reserved concurrency)放到 0 ,并且在异步队列中塞进 100条事件内容。在之前的策略中,该100条事件将永远无法被消费,甚至成为永久的脏数据。所以我们需要一个统一的淘汰机制来保证队列消息不会被用户行为打爆。这里我们将异步队列设置一个时间长度,以最大6小时为限,超过6小时的事件将直接被丢弃,如果配置DLQ,会将相关消息投递至DLQ。

看起来这两个策略是完美的了,至少它可以解决我们现在的用户问题。但,这仅仅完成了最简单的部分。

我们都知道异步队列最大的作用其实是保证用户缓冲某个时刻大量的高并发数据。之前并发都是和 work match的,并发上移后启多少work就变成了一个很玄学的事情,打个最简单的比方,下图是我们经常会遇到的情况并发超限。并发超限按照第一版的重试策略会跟随全部4xx,重试两次后丢弃。

什么?数据丢失?并发配额上线那段时间几乎每天都有在抱怨丢数据。这在完全托管的异步队列是完全不能出现的情况,比较理想的状态就是用户有多少并发就启多少 work,这样超限错误永远就不会存在了。如下图

但是的但是,这里基本是不可能实现的,或者实现起来难度非常大,对资源调度的管理要求太高。那又有什么办法可以保证用户的数据不丢失呢?我几乎想尽了所有办法,还是AWS给了我灵感,如下是AWS并发超限时 Throttles 的监控图:

我恍然大悟,这块也是重试策略的能力啊,我们为什么不把超限错误重新投回队列进行重试呢?这样不久完美解决了超限数据丢失?其实根本不需要什么复杂的算法来解决超限啊。所以,最新的重试是这样的:

 

运行错误(含用户代码运行错误和 Runtime 错误):当发生该类错误时,函数平台将默认重试两次或使用配置的重试次数,固定间隔1分钟。在自动重试的同时,新的触发事件仍可正常处理。如果您配置了死信队列,重试两次失败后的事件将传入死信队列,否则事件将被函数平台丢弃。
系统错误:当发生该类错误时,函数平台会根据您配置的最长等待时间持续重试(默认持续重试6小时),重试间隔按照指数退避增加到5分钟。如果您配置了死信队列,重试超过最长等待时间仍失败的事件会被发送到死信队列,由用户进行进一步处理,否则事件将被函数平台丢弃。
超限错误:当发生该类错误时,函数平台会根据您配置的最长等待时间持续重试(默认持续重试6小时),重试间隔为1分钟。如果您配置了死信队列,重试超过最长等待时间仍失败的事件会被发送到死信队列,由用户进行进一步处理,否则事件将被函数平台丢弃。
调用请求错误和调用方错误:当发生该类错误时,平台将不会对该类其他错误进行重试,因为其他请求错误即便重试也不会成功。(超限错误(432)除外)

新版的重试完美解决了并发超限给用户带来的丢数据的风险,也让异步队列发挥了它应有的价值。异步调用的并发超限用户再也无需进行任何操作,在设定的最长等待时间内,函数平台会自动对并发超限错误进行重试。

 

这个探索的过程相当煎熬,也几乎和研发同学PK了无数轮。好在最后我们完成了目标,当然是在经过两轮的方案推倒重做后。当然也在探索这里关于更开放的错误码重试策略的事情,但总感觉为时过早还需要论证和调研,所以这里就不介绍了。

 

总结一些思路,寥寥数语,如有偏颇还请谅解。



71 thoughts on “关于FaaS异步队列重试能力建设的二三事”

  • Thank you for this article. I’d personally also like to express that it can end up being hard if you find yourself in school and starting out to establish a long history of credit. There are many individuals who are simply trying to survive and have a good or beneficial credit history is often a difficult element to have.

  • That is very fascinating, You are a very skilled blogger. I’ve joined your rss feed and sit up for seeking extra of your great post. Additionally, I have shared your web site in my social networks!

  • The heart of your writing whilst sounding reasonable initially, did not really work properly with me personally after some time. Someplace throughout the sentences you actually were able to make me a believer unfortunately just for a short while. I nevertheless have got a problem with your jumps in logic and you might do well to help fill in those breaks. In the event that you actually can accomplish that, I will surely be fascinated.

  • Another issue is that video games are typically serious in nature with the most important focus on understanding rather than enjoyment. Although, there’s an entertainment facet to keep children engaged, each and every game is generally designed to improve a specific expertise or programs, such as instructional math or scientific disciplines. Thanks for your article.

  • In my opinion that a property foreclosure can have a major effect on the applicant’s life. Real estate foreclosures can have a Several to a decade negative effects on a borrower’s credit report. A borrower who’s applied for a mortgage or any kind of loans even, knows that the actual worse credit rating will be, the more tricky it is to get a decent personal loan. In addition, it can affect a new borrower’s chance to find a really good place to lease or hire, if that results in being the alternative houses solution. Great blog post.

  • An additional issue is that video games are typically serious in nature with the most important focus on studying rather than enjoyment. Although, it has an entertainment part to keep your young ones engaged, each and every game is usually designed to develop a specific experience or course, such as mathmatical or technology. Thanks for your write-up.

  • An added important part is that if you are a senior citizen, travel insurance pertaining to pensioners is something that is important to really consider. The elderly you are, the more at risk you will be for having something awful happen to you while in another country. If you are not covered by some comprehensive insurance plan, you could have some serious complications. Thanks for revealing your advice on this website.

  • We would also like to say that most of those who find themselves without the need of health insurance usually are students, self-employed and people who are out of work. More than half with the uninsured are under the age of Thirty five. They do not feel they are needing health insurance simply because they’re young plus healthy. Their particular income is normally spent on real estate, food, along with entertainment. Many people that do go to work either full or part-time are not made available insurance by means of their jobs so they go without with the rising valuation on health insurance in the states. Thanks for the thoughts you write about through this web site.

  • May I simply say what a comfort to uncover an individual who really understands what they’re talking about online. You definitely understand how to bring a problem to light and make it important. A lot more people really need to check this out and understand this side of the story. I was surprised you’re not more popular given that you definitely possess the gift.

  • An outstanding share! I’ve just forwarded this onto a co-worker who was conducting a little research on this. And he actually bought me breakfast because I found it for him… lol. So allow me to reword this…. Thank YOU for the meal!! But yeah, thanks for spending time to discuss this topic here on your web page.

  • After I initially left a comment I appear to have clicked the -Notify me when new comments are added- checkbox and now every time a comment is added I get four emails with the same comment. Perhaps there is a means you can remove me from that service? Thank you.

  • Hi there would you mind letting me know which webhost you’re utilizing? I’ve loaded your blog in 3 different web browsers and I must say this blog loads a lot faster then most. Can you suggest a good web hosting provider at a honest price? Many thanks, I appreciate it!

  • This is really interesting, You are an excessively professional blogger. I have joined your rss feed and look forward to in search of more of your magnificent post. Also, I’ve shared your site in my social networks!

  • I have learned several important things as a result of your post. I will also like to express that there may be situation where you will apply for a loan and do not need a co-signer such as a Federal Student Aid Loan. But when you are getting that loan through a conventional creditor then you need to be able to have a cosigner ready to allow you to. The lenders are going to base their own decision using a few issues but the most important will be your credit standing. There are some financial institutions that will also look at your job history and decide based on this but in many instances it will hinge on your ranking.

  • Thanks for your publication. I also think laptop computers have grown to be more and more popular these days, and now are sometimes the only form of computer included in a household. This is due to the fact that at the same time they are becoming more and more cost-effective, their working power is growing to the point where they are as effective as desktop computers from just a few years ago.

  • Thanks for the diverse tips shared on this blog. I have observed that many insurance carriers offer customers generous special discounts if they favor to insure multiple cars together. A significant amount of households possess several motor vehicles these days, in particular those with more mature teenage kids still residing at home, as well as savings in policies can certainly soon increase. So it pays to look for a great deal.

  • I will also like to express that most of those who find themselves without health insurance can be students, self-employed and those that are out of work. More than half of the uninsured are under the age of Thirty five. They do not feel they are needing health insurance simply because they’re young as well as healthy. Their particular income is generally spent on real estate, food, and also entertainment. Most people that do work either entire or part time are not provided insurance by way of their work so they get along without owing to the rising price of health insurance in the states. Thanks for the tips you discuss through this website.

  • Thanks for the ideas shared in your blog. Something else I would like to convey is that fat reduction is not supposed to be about going on a dietary fads and trying to lose as much weight as possible in a few days. The most effective way in losing weight is by getting it bit by bit and obeying some basic guidelines which can provide help to make the most from the attempt to shed weight. You may be aware and already be following these tips, yet reinforcing awareness never damages.

  • One thing I have actually noticed is that often there are plenty of misconceptions regarding the banking companies intentions while talking about foreclosure. One myth in particular is the bank would like your house. The bank wants your hard earned cash, not your home. They want the bucks they lent you along with interest. Averting the bank will simply draw any foreclosed summary. Thanks for your article.

  • Thanks for your tips. One thing we’ve noticed is the fact banks and also financial institutions know the dimensions and spending patterns of consumers and also understand that many people max out and about their own credit cards around the holiday seasons. They properly take advantage of this particular fact and commence flooding the inbox in addition to snail-mail box by using hundreds of no interest APR credit card offers soon after the holiday season closes. Knowing that if you are like 98 in the American general public, you’ll hop at the possible opportunity to consolidate consumer credit card debt and shift balances towards 0 rate credit cards.

  • I have witnessed that intelligent real estate agents just about everywhere are Advertising and marketing. They are seeing that it’s more than just placing a sign post in the front area. It’s really about building relationships with these sellers who one of these days will become consumers. So, once you give your time and energy to serving these sellers go it alone – the “Law associated with Reciprocity” kicks in. Great blog post.

  • I have learned some new things by your website. One other thing I would really like to say is always that newer computer operating systems usually allow more memory to be used, but they furthermore demand more storage simply to perform. If someone’s computer could not handle extra memory as well as newest computer software requires that memory space increase, it is usually the time to buy a new Personal computer. Thanks

  • Thanks for another informative web site. The place else may I get that type of info written in such an ideal means? I have a venture that I’m just now working on, and I have been at the look out for such information.

  • You can definitely see your expertise within the paintings you write. The world hopes for even more passionate writers such as you who are not afraid to say how they believe. At all times go after your heart.

  • Pretty nice post. I just stumbled upon your blog and wished to say that I have really enjoyed browsing your blog posts. After all I will be subscribing to your feed and I hope you write again very soon!

  • One thing I’ve noticed is that often there are plenty of misconceptions regarding the finance institutions intentions when talking about property foreclosures. One myth in particular would be the fact the bank would like your house. The lender wants your hard earned dollars, not your property. They want the funds they loaned you with interest. Averting the bank will draw any foreclosed realization. Thanks for your article.

  • My programmer is trying to persuade me to move to .net from PHP. I have always disliked the idea because of the costs. But he’s tryiong none the less. I’ve been using Movable-type on various websites for about a year and am anxious about switching to another platform. I have heard great things about blogengine.net. Is there a way I can transfer all my wordpress posts into it? Any kind of help would be greatly appreciated!

  • I beloved up to you will receive carried out right here. The comic strip is tasteful, your authored material stylish. nonetheless, you command get bought an shakiness over that you wish be handing over the following. ill indisputably come further formerly once more since precisely the same nearly very frequently inside case you protect this hike.

  • Hi, I do believe this is a great site. I stumbledupon it 😉 I may come back once again since I bookmarked it. Money and freedom is the greatest way to change, may you be rich and continue to help others.|

  • Thank you a lot for sharing this with all of us you actually know what you are speaking approximately! Bookmarked. Please also visit my web site =). We can have a hyperlink exchange agreement between us|

  • Hey! Do you know if they make any plugins to assist with SEO? I’m trying to get my blog to rank for some targeted keywords but I’m not seeing very good results. If you know of any please share. Cheers!|

  • Does your site have a contact page? I’m having a tough time locating it but, I’d like to send you an email. I’ve got some creative ideas for your blog you might be interested in hearing. Either way, great website and I look forward to seeing it grow over time.

发表评论

邮箱地址不会被公开。 必填项已用*标注

84 − 78 =