太长不看可以直接看文末加粗部分
问题描述
最近在项目中遇到了这样的问题，见代码
GAME_NAME = “”
def get_achieve_info():
# ...
global GAME_NAME
# 更新全局变量 GAME_NAME
GAME_NAME = soup.select(".profile_small_header_texture h1")[0].string.replace(":", "").replace("*", "")
# ...
def download(url, name, pic_path):
print(GAME_NAME) # 并行计算的方法内输出全局变量
if os.path.exists(pic_path):
logger.warning(name + " exists, skipping")
return
req = request.Request(url, headers=HEAD)
resp = request.urlopen(req)
if resp.getcode() == 200:
with open(pic_path, "wb") as f:
f.write(resp.read())
logger.warning("Downloaded " + name)
else:
logger.error("Get " + name + " returned code " + str(resp.getcode()))
return
def download_pictures(pictures_q):
print(GAME_NAME) # 非并行计算的方法内输出全局变量
try:
pool = multiprocessing.Pool(processes=multiprocessing.cpu_count() - 1)
output = pool.starmap(download, pictures_q)
except Exception as ex:
logger.error("Download Pictures Failed!")
logger.error("Error Message: " + str(ex))
def main():
# ...
logger.warning("Fetching Achievements...")
pictures_q, titles, descriptions = get_achieve_info()
logger.warning("Done.")
# ...
logger.warning("Downloading Pictures...")
download_pictures(pictures_q)
logger.warning("Done.")
# ...
if __name__ == "__main__":
main()
Windows 下的执行结果如下图
概括成一句话：并行计算时无法调用在其他方法内更新过的全局变量（全局变量为声明时的初始值）
原因
Continue reading “python 踩坑纪实 – 并行计算（multiprocessing）的全局变量问题”