捕蛇者说python内容(1)

任务队列 + `定时任务`

crontab
apscheduler

可以使用内存做 broker

多台机器，分布式注意任务重复问题

支持多种持久化方案

redis， mongodb..

celery

太过繁重

配置复杂

功能强大

huey

轻量级任务队列

多进程, 多线程, 协程

定时任务, 重试机制

兩種任務
    
    Huey.task()
    
    Huey.periodic_task()

from huey import RedisHuey, crontab, SqliteHuey

huey = RedisHuey("demo", host="10.0.0.206", port="6379", db=10)

@huey.task()
def add_numbers(a, b):
    return a + b


@huey.task(retries=2, retry_delay=20)
def flaky_task(url):
    return this_might_fail(url)



def this_might_fail(url):
    from random import randint
    a = randint(0,4)
    print(a)
    if a == 2:
        raise Exception()

def sync_all_data():
    print("finished data")


@huey.periodic_task(crontab(minute='1'))
def nightly_task():
    sync_all_data()
    

啓動消費者， 消費任務

    huey_consumer  demo.huey -w 4

立即執行任務  
    
    huey.immediate = True  # Tasks executed immediately.

huey = RedisHuey()

# Enable immediate mode. Tasks now executed synchronously.
# Additionally, huey will now use in-memory storage.
huey.immediate = True

# Disable immediate mode. Tasks will now be enqueued in a Redis
# queue.
huey.immediate = False

Gateway Interface `ASGI` vs `WSGI`

ASGI

ASGI (Asynchronous Server Gateway Interface) is a spiritual successor to WSGI, intended to provide a standard interface between async-capable Python web servers, frameworks, and applications.

需要全套的 asyncio 写法

WSGI

阻塞的

WSGI is the Web Server Gateway Interface. It is a specification that describes how a web server communicates with web applications WSGI provided a standard for synchronous Python apps

1  服务器主动向客户端发通知， 返回结果
    
     WSGI + HTTP协议 需要轮询, 消耗资源
     
     ASGI 不需要等待, 直接发 

2  异步任务(耗时任务)

     WSGI   添加第三方的任务队列 (celery , apscheduler)， 
            复杂
     
     ASGI   自带 事件循环, 直接添加任务进去

`uvicorn` + `uwsgi`, `gunicorn`

Python ASGI server

Uvicorn

依赖uvloop , makes asyncio 2-4x faster)

Python WSGI HTTP Serve

gunicorn

uwsgi

starlette框架

是基于 ASGI 的 web 框架

Python 3.6+

Seriously impressive performance.
WebSocket support.
GraphQL support.
In-process background tasks.
Startup and shutdown events.
Test client built on requests.
CORS, GZip, Static Files, Streaming responses.
Session and Cookie support.
100% test coverage.
100% type annotated codebase.
Zero hard dependencies.

Dependencies

Starlette does not have any hard dependencies, but the following are optional:

requests - Required if you want to use the TestClient.
aiofiles - Required if you want to use FileResponse or StaticFiles.
jinja2 - Required if you want to use Jinja2Templates.
python-multipart - Required if you want to support form parsing, with request.form().
itsdangerous - Required for SessionMiddleware support.
pyyaml - Required for SchemaGenerator support.
graphene - Required for GraphQLApp support.
ujson - Required if you want to use UJSONResponse.

from starlette.applications import Starlette
from starlette.responses import JSONResponse
from starlette.routing import Route


async def homepage(request):
    return JSONResponse({'hello': 'world'})

routes = [
    Route("/", endpoint=homepage)
]

app = Starlette(debug=True, routes=routes)



# 启动
uvicorn example:app

sanic框架

Flask Like 的 web framework

Python 3.6+ web server and web framework that’s written to go fast. It allows the usage of the async/await syntax added in Python 3.5, which makes your code non-blocking and speedy

from sanic import Sanic
from sanic.response import json

app = Sanic()

@app.route("/")
async def test(request):
    return json({"hello": "world"})

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8000)

更多Sanic和flask的对比

`pyflame` 性能调试

pyflame

使用：

查看调试的进程ID

ps aux|grep uwsgi

root      3013  0.0  0.7 258924 26556 ?        S    7月20   0:26 uwsgi /var/www/v7/uwsgi.ini

执行命令, 生成图片

pyflame -p 3013 | flamegraph.pl > myprofile.svg

追踪某个命令从开始到结束的完整执行情况

pyflame -t py.test tests/

python中的`super`

Python’s super() considered super

import logging

class LogginDict(dict):
    def __setitem__(self, key, value):
        logging.warning(f"setting {key} to {value}")
        super().__setitem__(key, value)


l = LogginDict()
l['a'] = 1

任务队列 + 定时任务

Gateway Interface ASGI vs WSGI

uvicorn + uwsgi, gunicorn