Rowan
Repo:
https://github.com/vegabook/BeamQN
Use Cases:
1. Continuously growing lazy lists that can be operated on with BQN functions (streaming financial data from an external resource).
2. Operating on Erlang data with BQN functions without any strict latency requirements (on-demand analysis of data from an external resource).
3. Potential integration with persistent databases (OLAP or OLTP).
Prior Work:
1. https://github.com/cannadayr/ebqn
2. https://github.com/cannadayr/rsbqn
3. https://github.com/relaypro-open/gen_q
4. https://github.com/gordonguthrie/pometo
5. https://github.com/mlochbaum/BQN/blob/master/docs/bqn.js
Approaches:
1. A BQN virtual machine in Erlang (EBQN).
2. A BQN virtual machine in Rust, integrated into the BEAM as a NIF (RSBQN).
3. CBQN integrated into the BEAM as a NIF.
4. CBQN integrated into the BEAM as a port driver.
5. CBQN integrated into the BEAM as a port.
6. Compiling BQN to BEAM bytecode.
7. A BQN interpreter in erlang.
Analysis:
1. A BQN virtual machine in Erlang (EBQN).
Pros:
* This approach has the benefit of not requiring any native code.
* No cooperative yielding to prevent scheduler collapse.
* All terms are native erlang terms.
Cons:
* EBQN was extremely slow, in particular to run the compiler stages.
* Required using process terms as a heap.
Potential Improvements:
* The virtual machine could instead generate erlang code that is compiled, and benefits from the JIT (similar to Javascript BQN implementation).
2. A BQN virtual machine in Rust, integrated into the BEAM as a NIF (RSBQN).
Pros:
* Can be integrated into the BEAM using the production-ready rustler library.
* Can be modified to yield to the Erlang scheduler (see https://github.com/rusterlium/rustler/pull/232).
* Can make some guarantees about software bugs that could lead to BEAM runtime crashes.
* Significantly faster than EBQN.
* Concurrent garbage collector (https://docs.rs/crate/bacon_rajan_cc/latest)
Cons:
* Significantly slower than CBQN, especially with SIMD code operating on large arrays.
* Additional BQN implementation maintenance (CBQN is the defacto BQN implementation).
* Would require additional work to integrate.
Potential Improvements:
* There are projects that could potentially improve SIMD operations performance (https://github.com/minotaur-toolkit/minotaur).
3. CBQN integrated into the BEAM as a NIF.
Pros:
* CBQN is the defacto, high-performance BQN implementation.
* Significant work has gone into optimization.
Cons:
* Modifying to use enif_schedule_nif to cooperatively yield to the scheduler might not be feasible.
* Uses a mark-and-sweep garbage collector (cannot reliably make guarantees on collection time).
* Must manage its own resources.
* Would likely have to be a "dirty" NIF, running on its own thread, and interpreter handles would need to be managed from the BEAM.
* Would likely need additional work to properly free interpreter heaps.
* Fewer compiler level guarantees about crashing the BEAM.
Potential Improvements:
* Would have to familiarize ourselves with the CBQN source code, and determine feasibility of alternative GC and scheduler yielding.
4. CBQN integrated into the BEAM as a port driver.
Pros:
* Port drivers are asynchronous by default.
* Only open-source, in-production integration of an array language into BEAM uses a port driver (gen_q).
Cons:
* Port drivers seem to be unadvised in the community (https://erlangforums.com/t/use-cases-for-port-drivers-when-is-it-better-to-use-a-port-driver-instead-of-a-nif/1772), however its not clear if theres any plans to remove them as a feature.
Potential Improvements:
* Needs more research.
5. CBQN integrated into the BEAM as a port.
Pros:
* Safest integration of an external application into the BEAM.
* Simplest integration of an external application into the BEAM.
* Asynchronous by default, communicates via message passing.
* Will not crash the BEAM interpreter.
Cons:
* Significantly slower than a NIF or Port Driver.
* Potential non-trivial overhead of encoding/decoding terms.
* Potential non-trivial overhead of writing data to port handle.
* Potential improvements seem unlikely (upper bound on performance).
Potential Improvements:
* See above.
6. Compiling BQN to BEAM bytecode.
Pros:
* Has advantages of both native BEAM bytecode, and sticking close to the upstream implementation.
Cons:
* Way beyond my immediate familiarity.
* Might require modifying the BQN compiler to change output of intermediary representation - if it exists (IR).
Potential Improvements:
* Would be interesting as a research subject, but writing a bytecode compiler would take me significant effort at this time.
7. A BQN interpreter in erlang.
Pros:
* Has advantage of native BEAM bytecode.
* No scheduler collapse.
* No BEAM crashes.
* All native Erlang terms.
* Native BEAM term allocation and garbage collection.
Cons:
* Would require maintaining a separate implementation.
* Would require modifying pometo to support BQN.
Potential Improvements:
* Singeli could potentially be used to improve SIMD performance for BQN primitives.