Mles v2 published!
jq-rs, 30.12.2023
I am glad to announce Mles v2 has landed! More details on the Mles channel /c/mles.
Happy and safe messaging on Mles v2!
jq-rs, 30.12.2023
I am glad to announce Mles v2 has landed! More details on the Mles channel /c/mles.
Happy and safe messaging on Mles v2!
jq-rs, 17.8.2023, cross-posted from /c/mles
Mles originates from October 2016 and the Mles message structure has been the following since the v1.0 release
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ASCII 'M'(77) | Encapsulated data length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| CID (lowest 4 bytes of initial SipHash) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ SipHash +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| CBOR encapsulated data (uid, channel, message) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
It had pretty modern aspects included such as
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ASCII 'N'(78) | Data length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SipHash + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | JSON encapsulated id data (uid, channel) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Even though the fields are quite the same, the way we use them will be different and hopefully will solve all of the above issues:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data (raw bytes) ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+The first frame cannot hold more data than the JSON identifiers, but after that avoid we all the overhead allowing the application to decide the framing.
jq-rs, 6.11.2022
Mles protocol allows to create a distributed transparent data forwarding service. MlesTalk is a prototype mobile application which allows to use this framework in modern communication. MlesTalk itself is split into two parts, the UI and communication parts. The communication part is implemented as a JavaScript webworker which is easily replacable with any other webworker that implements its API. I am happy to notice that Zpinc Developers have implemented an alternative prototype webworker as a drop-in replacement, found here. In addition, they have also published an article which describes the overall messaging system on any pub-sub infrastructure, similar to Mles. Interesting! If you want to try it out, please check instructions how to compile your own instance. Happy messaging all!
jq-rs, 31.10.2021
I am happy to announce that MlesTalk has now full multichannel support included! Want to have just a simple messenger which allows you to send text and images without ads or giving a phone number or an email? And avoids the data collection of big corporations through services like Google Firebase? Here you have it, please check it out.
MlesTalk uses Mles WebProxy over WebSocket for messaging sessions. For more detailed information, please see app info.
jq-rs, 15.9.2020
I am glad to announce that Mles WebProxy 0.8 has been released in crates.io! It is based on Warp 0.1 and supports
If you have always wanted to have a simple static website which automatically renews TLS certificates from Let's Encrypt and also provides an access point into a federated messaging, this your chance now! Please check crates.io for step-by-step instructions. Thank you all!
jq-rs, 4.10.2017
I am happy to announce that Mles v1.0.0 has been released! Hurray! The most significant changes are related to API handling, now every API functionality is accessed through a structure with get()/set(), encode()/decode() which makes it simpler and more logical to use. Please have a look at the API here. Peer reconnection handling also had some fixes in just before 1.0.0 release.
I am very excited to be able to release this quite different and performant pub/sub data protocol with a reference implementation in Rust! Please see all the details at https://github.com/jq-rs/mles-rs, and please have a look to http://mles.io for more generic information.
jq-rs, 10.9.2017
Hi all! An issue with peer connections was found during 1.0.0-beta2 testing, thanks to all for testing efforts! The problem was that whenever a peer initiated a connection to a root server, it sent only the header of the first message to line. This could cause occasional instability to the peer server connections. The problem was introduced during code refactoring and was not catched by tests because the peer server connection handling did not unfortunately have automated tests. Now the issue is fixed with newly release beta3! The unit tests are also enhanced so that this particular problem won't ever happen again, but there may well be other issues. To ease up testing, mles.io has now open 8077 port a limited time before 1.0.0 release with "mles-devel-frank" key for you to have a try of the protocol. So why not give beta3 a go? Please remember to send feedback!
Another pretty exciting feature introduced in beta3 is the new resynchronization support between root and peer servers. Whenever a root server is restarted, it resyncs its state from peer servers' channel subscriptions, if they happen to have the history available for their channels (they could be configured to not to save any history). The resync is done with a newly introduced ResyncMsg-message. If you want to try it, you'll need to update your root and peer servers to beta3. This kind of distributed protection type of service is often very complex to introduce to any environment, however this resyncing feature was pretty swift (or should I say Rusty..) to implement. Moreover, it gives visibility to clients for the root server restart which was not seen earlier at all. I think it is a cool and quite unique feature to have in Mles - and now on board as part of forthcoming 1.0-release!
And of course all the details are available at https://github.com/jq-rs/mles-rs, please have a look if interested! PRs are welcome too for issues you may find!
Happy beta3 testing!
jq-rs, 24.7.2017
Thanks for the feedback during beta testing period! Some minor changes are now introduced to Mles protocol and server itself. The most significant is perhaps the ability to publish a message during connection initiation to Mles server. This allows to publish faster to subscribers. This is the only protocol change and server has been updated accordingly to support this functionality.
Moreover, the server functionality has been moved inside the library which allows to spawn new servers easily within programs and provides better characteristics for end-to-end automated smoke tests. A new APIs for server connections for receiving and sending messages are added and are facilitated in new unit tests - and in the new reference test (see below). Please check https://github.com/jq-rs/mles-rs and crate documentation for more information!
In addition to API changes, the licensing has been updated to Mozilla Public License 2.0 (MPL-2.0). As Rust libraries are statically linked by default, the usage of libraries under GPL is difficult with other licensing schemes than the GPL itself. For me the most important thing about GPL was really to underline that useful improvements should be introduced back to upstream to benefit the development community. MPL-2.0 motivates to do this without forcing the library users' licensing to anything specific. Even though you as a developer may think that it does not really make sense to publish some minor change because of the trouble involved, it is really worth it, as the improvements of all developers pile up in the long run!
I was also asked about differences between existing publish/subscribe networks and possible advantages of Mles. It is true that similar frameworks exist already, e.g. RabbitMQ provides ability to build a similar publish/subscribe service. In theory Mles server and clients should provide significantly better performance as it is simpler and focuses to this certain service type with minimal complexity. Real measurements are of course always a nice way to prove that the assumptions are correct. Thus, a new reference measurement has been added to utils library for RabbitMQ and Mles. It has one test for 100 threads receiving "Hello World!" and records the delay when all messages are correctly received by clients. With quick comparison, Mles is roughly over twice as fast compared to RabbitMQ with this use case: 300 samples with RabbitMQ gave 231.87 ms mean performance (lower is better) with 18.3 ms variance compared to 97.47 ms mean with variance 2.3 ms Mles performance on the same 6 core i7 VM system. More fine-grained tests that measure raw messaging performance may be added later for further reference. If you have an idea how to better compare the performance, please give feedback! And of course if you want to contribute to reference tests (or to any Mles development), feel free to send a PR!
Because of above mentioned changes are quite significant, a new beta2 is released! Feel free to try it and please give feedback!
jq-rs, 29.6.2017
After four weeks of alpha testing, a couple of issues came up that needed some more detailed investigation. First one was loop handling, which was not that good in the alpha-release. Mles users can certainly control that they do not connect to peers at all which obviously prevents loops, so one might think that this is not really an issue. Nevertheless, if peer is configured, the user cannot really control where the peer might be connected to and would that connection path eventually create a loop back to user's original Mles server. To solve this problem, a new connection id (CID) field with random value was introduced to Mles protocol header. It can be used by the server to identify possible loops between the peer servers and block those connections.
Other problem that was noticed was the limitation to login at most once with same UID from different equipment. In many cases, one would prefer to use the same UID from different devices at the same time. Now sequential connections with same UID on the same channel were blocked. We were lucky here as this UID problem can be fixed with the same just introduced CID field! After some test rounds with the new solution, it should be now in good enough shape for other to try. So, Mles reference implementation is now 1.0.0-beta released! Just grand! Feel free to try and please give feedback! You can find more info at http://mles.io.
jq-rs, 31.5.2017
After two weeks of testing, Mles reference implementation is now alpha-released. Grand! Feel free to try and please give feedback! You can find more info at http://mles.io.
If you wonder how can 1.0 be so close as underlying dependencies have not reached 1.0 yet, the short answer is that with crates.io, it should not matter. A longer answer is that it of course depends of the depencies: after some slightly philosophical discussion on relevant channels, e.g. Tokio should be ready for 1.0 release any time. The stable APIs are the key here and filling general expectations of 1.0 releases. Thus, I think we are close to 1.0.0, if everything goes well with testing the alpha.
jq-rs, 14.5.2017
I have followed the path of Rust from early days with enthusiasm. As my own background is heavily C-language oriented, with emphasis on networking, I think I have some insight to all the problems C has, it indeed being "a quirky, flawed and enormous success" [1]. There is a tremendous effort of running additional tools to at least somewhat guarantee that large programs do what they are supposed to, if they are written in C. To me, selecting or even keeping C-languange as a primary language to implement almost anything nowadays does not make sense as we have other better options: Rust as a safe, concurrent and fast systems language is a very good such option, solving most of the C-problems at compile time. And with the zero-overhead combining C-programs and Rust-programs, even more so.
Is Rust then ready for solving networking problems and implementing new protocols? Instead of just pondering, I decided try is it able to do a new lightweight and distributed publish/subscribe data service protocol which I named Mles (Modern Lightweight channEl Service). Without any worthy earlier Rust experience, it would be a challenge that would really tell something about Rust's networking support.
The journey took more time than I anticipated initially, but it was certainly worth it. All implementation was done on my free time (usually when it was raining (which was often, to be honest)). I ended up writing first synchronous version of the Mles server reference implementation. Serde with Serde CBOR crate provided a great framework to build on the first versions of the Mles protocol implementation. The synchronous implementation was a good training for Rust in general, recommended to anyone familiarizing to Rust network programming.
As my final target was fully non-blocking and lock-free server implementation, I quite quickly started to re-implement the synchronous version with Rust asynchronous networking crate Tokio. And there was some learning curve there, as Rust/Tokio has cool features like closures, futures and tasks which I had not encountered earlier. If you know e.g. Haskell, many Rust's modern features may be quite familiar to you.
Today, I think I am close to 1.0 release of Mles reference implementation and Mles protocol at the same time. All relevant features are implemented and additional WebSocket proxy support was added in the recent days, which will make the implementation maybe more relevant to users who already have a WebSocket client available. After finalizing the WebSocket proxy, I'll prepare to publish 1.0.0 alpha-release. Probably in the near weeks, if nothing major comes up. Feel free to try and please give feedback! You can find more info at http://mles.io.
I really enjoyed the journey and Rust will be my number one choice in the future. During the development, the absolutely great thing was the Rust community! I was not alone learning and solving the problems encountered. Thanks to all who helped, e.g. on the #rust IRC and Tokio Gitter channels. The Rust Book [2] is awesome! The answers to StackOverflow and Reddit are also just wonderful and complete for Rust, grand stuff. A special thanks to @pyfisch, @carllerche and @tanriol, you made a difference in this project, I appreciate it!