太长不看可以直接看文末加粗部分

问题描述

最近在项目中遇到了这样的问题,见代码

GAME_NAME = “”

def get_achieve_info():
        # ...
        global GAME_NAME
        # 更新全局变量 GAME_NAME
        GAME_NAME = soup.select(".profile_small_header_texture h1")[0].string.replace(":", "").replace("*", "")    
        # ...
def download(url, name, pic_path):
    print(GAME_NAME) # 并行计算的方法内输出全局变量
    if os.path.exists(pic_path):
        logger.warning(name + " exists, skipping")
        return
    req = request.Request(url, headers=HEAD)
    resp = request.urlopen(req)
    if resp.getcode() == 200:
        with open(pic_path, "wb") as f:
            f.write(resp.read())
        logger.warning("Downloaded " + name)
    else:
        logger.error("Get " + name + " returned code " + str(resp.getcode()))
        return


def download_pictures(pictures_q):
    print(GAME_NAME) # 非并行计算的方法内输出全局变量
    try:
        pool = multiprocessing.Pool(processes=multiprocessing.cpu_count() - 1)
        output = pool.starmap(download, pictures_q)
    except Exception as ex:
        logger.error("Download Pictures Failed!")
        logger.error("Error Message: " + str(ex))


def main():
# ...
logger.warning("Fetching Achievements...")
pictures_q, titles, descriptions = get_achieve_info()
logger.warning("Done.")
# ...
logger.warning("Downloading Pictures...")
download_pictures(pictures_q)
logger.warning("Done.")
# ...


if __name__ == "__main__":
main()

Windows 下的执行结果如下图

python 踩坑纪实 - 并行计算(multiprocessing)的全局变量问题

概括成一句话:并行计算时无法调用在其他方法内更新过的全局变量(全局变量为声明时的初始值)

原因

Continue reading “python 踩坑纪实 – 并行计算(multiprocessing)的全局变量问题”

logging 内定义的优先级为 CRITICAL > ERROR > WARNING > INFO > DEBUG > NOTSET

简单来说,根 logger 的优先级为“保底”

对于任何 handler ,如果其优先级高于根 logger,则按照 handler 的优先级显示日志

如果其优先级低于根 logger,则按照根 logger 的优先级显示日志

有如下代码:

import logging

format = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("test logger") # 根 logger

fhandler = logging.FileHandler("log.txt")
fhandler.setLevel(logging.DEBUG)
fhandler.setFormatter(format)

shandler = logging.StreamHandler()
shandler.setLevel(logging.WARNING)
shandler.setFormatter(format)

logger.propagate = False

logger.addHandler(fhandler)
logger.addHandler(shandler)

logger.debug("debug")
logger.info("info")
logger.warning("warning")
logger.error("error")
logger.critical("critical")


其运行结果为

python logging 优先级

文件日志的 handler 日志级别为 DEBUG,比根 logger 的级别 INFO 低,文件日志输出到 INFO 级别,没有输出到 DEBUG 级别

控制台日志的 handler 日志级别为 WARNING,比根 logger 的级别 INFO 高,控制台日志输出到 WARNING 级别,没有输出到 INFO 级别