太长不看可以直接看文末加粗部分
问题描述
最近在项目中遇到了这样的问题,见代码
GAME_NAME = “” def get_achieve_info(): # ... global GAME_NAME # 更新全局变量 GAME_NAME GAME_NAME = soup.select(".profile_small_header_texture h1")[0].string.replace(":", "").replace("*", "") # ... def download(url, name, pic_path): print(GAME_NAME) # 并行计算的方法内输出全局变量 if os.path.exists(pic_path): logger.warning(name + " exists, skipping") return req = request.Request(url, headers=HEAD) resp = request.urlopen(req) if resp.getcode() == 200: with open(pic_path, "wb") as f: f.write(resp.read()) logger.warning("Downloaded " + name) else: logger.error("Get " + name + " returned code " + str(resp.getcode())) return def download_pictures(pictures_q): print(GAME_NAME) # 非并行计算的方法内输出全局变量 try: pool = multiprocessing.Pool(processes=multiprocessing.cpu_count() - 1) output = pool.starmap(download, pictures_q) except Exception as ex: logger.error("Download Pictures Failed!") logger.error("Error Message: " + str(ex)) def main(): # ... logger.warning("Fetching Achievements...") pictures_q, titles, descriptions = get_achieve_info() logger.warning("Done.") # ... logger.warning("Downloading Pictures...") download_pictures(pictures_q) logger.warning("Done.") # ... if __name__ == "__main__": main()
Windows 下的执行结果如下图
概括成一句话:并行计算时无法调用在其他方法内更新过的全局变量(全局变量为声明时的初始值)
原因
Continue reading “python 踩坑纪实 – 并行计算(multiprocessing)的全局变量问题”