Python multiprocessing crashed on my custom class. Took forever to figure out why.
So I had this script that processed a bunch of images. Image processing, right, perfect use case for multiprocessing. Split the work across 8 cores, should be way faster. First attempt: from multip...

Source: DEV Community
So I had this script that processed a bunch of images. Image processing, right, perfect use case for multiprocessing. Split the work across 8 cores, should be way faster. First attempt: from multiprocessing import Pool def process_image(image_path): # do stuff return processed with Pool(8) as pool: results = pool.map(process_image, image_paths) Looked fine. Ran it. Crashed immediately. Pickle error. Cannot pickle <class 'MyImageProcessor'> My what now? So here's the thing I forgot. Multiprocessing in Python uses pickle to send data between processes. Functions get pickled, arguments get pickled, results get pickled. Everything gets pickled. And my image processor class? Not pickleable. Tried a few things: Made the class simpler Tried with just a function Switched to threading instead Threading worked but was slow as hell because of the GIL. Image processing is CPU intensive, threading doesn't help there. Finally figured out the fix. The class held some state that couldn't be pick