Elixir - Process Pooling25 Dec 2018
there are several solutions which allow to process collections concurrently and to keep a constant amount of processes (pool) running at any given time.
each process is called a task (implemented as
this is the simplest solution which however must be sufficient for most use cases. what is important is that it’s built into the language - you don’t need to add another dependency.
Elixir v1.4 has something called
Task.async_streamwhich is a mixture of poolboy + task async. We have added it to Elixir after implementing Flow as we realized you can get a lot of mileage out of
Task.async_streamwithout needing to jump to a full solution like Flow. If using
max_concurrencyoption controls your pool size.
each process is called a worker (implemented as
Poolboyis a battle-tested Erlang pooling solution but still it’s a third-party library and hence one more depenedency so I’ll adhere to built-in
Task.async_stream/5solution for my future projects as long as it meets my requirements.
each process is called a stage (implemented as
Flowis geared towards large-scale data processing on par with, say,
Apache Sparkthough being not distributed per se unlike the latter.