Antimatrix
Sync and BTSync Detailed User Manual

(Formerly BTSync Detailed User Manual)

Версия на Русском

See also:

BTSync User Guide

Note: This manual no longer focuses exclusively on BTSync, and there are numerous reasons for it. We will, from now on, focus on sync process as such and will cover all the available sync alternatives. We will also start working on collecting the information base for creation of an open source alternative to BTSync.

And it is a pleasure to mention that even at this point there exists enough information to create a truly open source BTSync alternative, and even more than that, we already have at least two working versions of the BTSync alternatives.


Note: This manual is also available via btsync.

See: How to get this manual via BTSync


You can ask the questions, bring some proposals or simply chat with other participants who are on line and have BTSync running and everyone will see it within seconds.

To do that, you need to create a new BTSync folder for chat page with the following r/w key ("secret"):


AR333HZMSCBUQ3XVO3SIAVZAXEXZTAL5Y

You can just keep this page loaded in your browser and simply refresh it periodically and you will almost immediately get new messages. So it becomes like a chat.

Contents

Open source sync design

Existing open source sync alternatives

Syncthing (Pulse)

Ego politics

Other open source sync alternatives


BTSync

Problems and solutions
Installation and configuration

O/S Independent issues

Linux

Principles of operation
Edit History

Contacts: Where to get help?

If you have any problems using the BTSync program, send the email to here: preciseinfo [@] mail [.] ru.

Get involved, instead of sitting on your butt

Get involved, instead of sitting on your butt, hoarding your creative energy and waiting for some "Messiah" to come and "save" you. If you only take, but never give anything, you will never get enough to satisfy you finally. Because in that case you are simply wasting your life in endless turns of the Wheel of Karma, as they call it, or are "spinning your wheels, getting nowhere really fast".

No matter who you are, you do have something to contribute. For this is the Law. The Law of Life itself! All you need to do it to look inside yourself and pay attention to that which "turns you on", something that attracts you, something that excites you and makes you want to be.

And then you simply give whatever "turns you on" without worrying of condemnation and denials of others, done, primarily, out of their of blindness and self-denial. You shalt see this one day. It is just a matter of time WHEN you do, at least if you are sincere and are honest enough to yourself.

If all those of you, who really understand and know something, simply add what they know to this manual and make sure to add the new chapter/topic to the TOC, can you imagine how much you help all sorts of people struggling "out there" with artificial problems that only exist because they simply do not understand how it all works? Well, all that wasted energy can be redirected to CREATIVE things, instead of wasting it, struggling with the paper dragons.

So, let this manual become truly alive with every drop you contribute to it, and, with it, YOU will. For it is the Law, the law of Life itself as expressed by the Infinite Multidimensional All-Pervading and Ever-Unfolding Intelligence.

The purpose of this manual

The purpose of this manual is not to merely duplicate the information available elsewhere, including the "official" BTSync manuals. But to provide the comprehensive and most detailed coverage of some of the tricky issues and describe as many different aspects of BTSync behavior and its effects as possible.

This means that not every single conceivable issue is covered here, even though we attempt to cover as much as possible.

The second point is to hopefully give an example to other software manufacturers or developers of how do you write documentation which covers all the minute details of your products in such a way that even users with the most basic understanding of computers could use your products without going to all sorts of forums and asking about those things that should have been explained in the first place.

Many, if not most, software manufacturers and developers in most cases provide only scratchy documentation that leaves way too many questions in the minds of ordinary users. Instead, they keep placing almost all of their efforts on developing all sorts of bells and whistles and "features" of all kinds, as a result of their never ending market department driven attempts to grab a bigger and bigger "market share", and BT is about as perfect example of it, as it gets.

So, instead of covering the most important issues in their documentation and fixing the critical bugs, they dedicate most of their efforts on adding more and more "bells and whistles" to their product while urgently needing resolution issues remain unattended. Their basic product is not fully described in clear and concise and detailed way.

If you look at their blog, you might get amazed at the amount of hype they keep producing in their massive marketing efforts.

http://blog.bittorrent.com/

And if you subscribe to their RSS feed, you'll be getting the announcements on all sorts of "bundles" packaging all sorts of zombie "entertainment" nearly every other day.

http://blog.bittorrent.com/rss/

But if you look at their BTSync forums, you'll be amazed, if not "blown out of your chair" outright seeing the amount of problems users experience, and plenty of those problems are of CRITICAL nature, the issues that have to be resolved ASAP. But many, if not most of them, simply remain unattended as you can see by the age of some of those threads.

Sync Troubleshooting
http://forum.bittorrent.com/forum/108-sync-troubleshooting/

The result of this is the products that work in most cases, but in some cases of "this should never happen" kind they either do not behave correctly or simply do not work - period.

For example, we have observed a countless cases where new users join the share but never actually start downloading it. So, they sit there for a while and then leave, never to be seen again. At the moment, we are seeing over half of the nodes on one of our main shares not being in the state of 100% completion, and they stay that way permanently, never completing, at least as it looks to other nodes. We have also seen cases where some nodes begin to download, but then stop for some reason and never resume the download and never completing the initial download. And we have seen the cases where they do manage to complete downloads if you restart BTSync on a master share. And we have seen plenty of cases where nodes permanently remain in a locked state as a result of time difference issue between the nodes, which is also one of the most critical issues that had to be resolved immediately, but remains unresolved for nearly a year now.

Now, every single issue above is of CRITICAL level and has to be resolved immediately. Because the program simply does not work or works in fundamentally incorrect or totally unexpected ways.

The users install the program, try to join some share and sit there for few minutes waiting for something to happen. But, since nothing happens, they simply leave the share and probably simply deinstall the BTSync thinking that "this stuff does not work". And what could be MORE critical of an issue than not being able to perform even the initial download? How is this even possible technically?

They do not seem to understand that about the MOST important criteria in any product is not necessarily the quantity of bells and whistles, but the robust and predictable behavior and the ability to recover from any situation, conceivable or "impossible".

And the second most important property of any product is precise and detailed documentation that could be understood by "mere mortals", and not just some one-liners of a smart kind that could be understood only by professionals.

Documentation should be as easy to read as you read a newspaper. You should not be sitting there, scratching your head and wasting your mental and creative energy on trying to think "what do they mean by that?".

So, by reading a good documentation you effectively become and expert as far as product use goes. Simple as that.

And, finally, if not most importantly, the overwhelming volume of problem posts on their forums, for the most part, is a result that users simply do not know or do not understand how things should work under normal circumstances, simply because it is not documented in the manuals. When we were introduced to the BTSync, we could not quite believe how scarce and sketchy is their documentation and how little of some of the most critical issues of the program behavior were described.

Forums are not the place to provide the documentation. Else, you see the large amount of traffic on them that should not be there if it all described somewhere. As a result, their customer support on the forums is simply overwhelmed with amount of questions and issues.

The end result of poor documentation is wasting a lot of energy that could be used for creative purposes and not merely wasting one's time struggling with paper dragons and purely imaginable issues that shouldn't be there to begin with.

Building the sophisticated information distribution systems

You can build the sophisticated information distribution systems of a synthetic kind where one part of your system exhibits the private network behavior and is not visible to others, who are not the part of your private network, and the other part of your system exhibits the purely public aspect and can be accessed by anyone as long as they have a key to some share/top folder/collection.

Here is one example related to "hot" and "not so hot" information distribution via web sites:

In this example we will use the combination of BTSync and Syncthing (Pulse) working together.

Assume you have a web site on a server somewhere and you have a working system, possibly at home. You might be doing all sorts of updates and edits on all sorts of information in all sorts of subdirectories of your site which you would like to sync to the site as soon as you push the save button in your editor. This way, you do not have to worry about remembering which exact subdirectory of your site the information belongs to and you don't have to waste time on using the FTP client to update the server, which requires a number of steps before you can actually transfer the files.

So, you can use the BTSync via privately known key to update the site automatically and in "hot mode", meaning that as soon as you push the save button in your editor, the information gets shipped to the server within seconds, and without any further action on your part beyond pushing the save button in your editor. The rest of the process is fully automatic.

But then, you would also like to make this site available on the sync basis to the general public so that they could receive the "latest and greatest" updates nearly instantly. Except there is one "but". In case you are working with some document in active mode, which might imply a lot of updates and save cycles, possibly every few seconds, you would not necessarily want to update the sync clients as often as you update your site.

So, what you do is to create another folder on your disk which contains the same exact copy of the site, but in "not so hot" mode, so that you could decide at some point: ok, we are kind of done with this cycle of updating this document and now it is ready to be synced by the whole world.

To do that, you establish another connection to that 2nd folder, but with using the Syncthing this time. What it does is to feed your 2nd folder from the "hot" site back to your home box.

Finally, you feed that 2nd copy of your site to the whole world via BTSync, only on a different share/folder. So, as soon as your site on the web server is updated, it feeds the info back to the 2nd folder and is served to the whole world via BTSync.

So, to make that 2nd folder to be not so "hot", all you need to do is to stop the Syncthing that feeds back the information from the site to your 2nd folder. Then, once you feel this edit session is complete enough, you enable back that Syncthing channel and it will bring back the site to the 2nd folder that is serving the world now.

One other benefit you receive with this approach is that you also assure a certain guarantee that your original information does not get damaged because of the actions of other users from the general public that might do some crazy things with their copy, which, in turn, might update all the other nodes with wrong or bad information. This is quite possible with BTSync and we have seen it many times in different situations, either because of program misbehaving or because some users do things not really understanding what they are doing. And we have also seen plenty of cases when some users intentionally try to attack the share. In this case, under no circumstances would you like your original information to get damaged. That is why you establish a private channel only between your working system and your server. In this case, you know 100% that you do on that link only those things that will not cause the damage to the original information.

So, the private side of your system can be controlled by you in a predictable and reliable manner, because you control and have access to both ends of that link, but the 2nd folder, which is connected to the general public, might and is likely to get damaged, sooner or later, because of all sorts of reasons. Moreover, some degree of damage may be observed on more or less periodic basis. We have seen some shares getting damaged every few days, mostly because not that many people really understand what they are trying to do and the consequences of it.

But you can recover from that damage since you also control the 2nd folder and the original state of the data can be restored within seconds simply by copying the original data over the damaged copy. This can be done with a single command. You just need to remember to copy the entire directory tree preserving the file modification times.

So, what you have effectively, is a synthetic system that exhibits the behavior of totally private connections, but, at the same time, any part of your information may become public any time you want and all of it will be updated automatically without any need for you to do any additional things in order for the entire system to function as a whole, and in any mode of operation you might think of, "hot", "not so hot", or even in "slow motion".

Interactive information systems with nearly instant delivery

What the above mentioned synthetic information system also means is that you can get really creative and can concentrate primarily on the creative aspects of working with information, instead of being bogged down by purely technical aspects of information management and the routine work. This way, your creative energy could be used at its peak, without worry about all sorts of routine technical things and tasks.

Once the system is set up, you no longer have to worry about many things that assure the whole thing works. Everything, but the creative aspect itself, becomes fully automatic and nearly instant delivery and distribution of information becomes guaranteed and fully automated.

In other words, now has come the time, finally, "to get down with it" and investigate and expose the REAL issues facing this planet, as you no longer have any excuses of not "doing a good job" because of some bizarre or overwhelming technical aspects and issues.

What else can we say?

What does sync have to do with New World Order?

Note: the following chapter(s) were originally written in the context of conversion of Syncthing to Pulse and, shortly after that, change of copyright license, which has the profound effects on the entire project. So, the implied reference to be kept in mind is the Syncthing/Pulse project.

Well, in a single word: EVERYTHING! Believe it or not.

In relationship to Amazon.com showing up in the whois request, whatever that means or implies, one can mention this kind of thing:

If, by any chance, Amazon.com is in fact involved in any of this, then the question arises: But who stands BEHIND the Amazon curtain?

For example, it is alleged that behind the BTSync stands no one less than the NSA, and that is pure grade evil and global, all pervasive surveillance and identification of the "undesirables" to be placed on the so called "red list" of those individuals that are considered "dangerous" in the view of the powers of evil and this NWO thing.

Interestingly enough, even the veterans returning home from the war zones are considered to be "the most dangerous" kind, if you can imagine such a degree of evil. Those that are promoted as "heroes" and "fighters for freedom and democracy", in reality, are in fact considered to be the "most dangerous" of all. And those soldiers do not even realize any of this until they return back home and cold and brutal reality hits them between their eyes.

Most people childishly believe that there simply must be some limits to evil and it can not go against the most basic things. Well, sooner or later, they'll have a chance to realize that the evil has no limits whatsoever, ethical, moral or otherwise". That is its very nature. It does not even have a concept of some bounds or limits, just as shown, for example, in a book called "Atlas Shrugged" by Ayn Rand, who was one of the mistresses of Rothschild and this book, according to John Todd, a former member of "the council of 13", was created as per request of Rothschild as indoctrination book for the Illuminati. Kind of mind programming book in a psy-ops kind of scenario.

John Todd's introduction to Atlas Shrugged
(about the famous Illuminati mind conditioning book by Ayn Rand)
http://antimatrix.org/Convert/Books/Ayn_Rand/Ayn_Rand_Atlas_Shrugged.html#John_Todd_introduction_to_Atlas_Shrugged

And that opens the whole Pandora box that leads to this thing called the NWO. Actually, anything one deals with in the modern world, eventually and inevitably leads to the same thing, and that is takeover of this planet via "The Plan" of planetary takeover, which has been nearly completed in full at this junction.

And so let us diverge a little bit from the "main subject", and attempt to cover just a few basic things about that scheme, because, first of all, these are some of the most vitally important things that are happening on this planet, by far. Anything else simply pales in comparison, in terms of significance or the impact, at least from the standpoint of the global information war raging on as we speak, and if one underestimates the consequences of that, then, as they say "you are as good as dead".

And about the sorriest thing is that not that many people even realize what is REALLY going on. They just buried their heads in the sand and "play dumb", like it is "not my problem". So...

For example, on BTSync forums there was one thread that specifically addressed the issues related to the NSA, and it was one of the most active threads. Interestingly enough, the BTSync staff has not made any public statements on it or about it in order to clarify some issues.

Then, all of a sudden, that thread was simply deleted, without saying a word of comment or providing a reason for it! Why would that happen and what would it imply?

Furthermore, one participant on that thread has said that he definitely has seen some "strange behavior" of some things, except he did not wish to disclose exactly what was it, and for quite obvious reasons, once you realize what you are dealing with.

Are claims of security and encryption real?

And the idea behind the so called "red" and "yellow" lists is that all those that end up on the "red list" are to be exterminated first, or thrown into the concentration camps according to the NWO plans. In the USA alone, at this junction, there exist over 100 concentration camps, with gassing chambers, efficient furnaces to burn the bodies and the rest of it. The "red list" as of couple of years ago contained about a MILLION people, if you can comprehend the grandness of the scale of this operation code named "Project ENDGAME". Barak Obama has issued a secret order to activate that operation back in 2009. This information is publicly available, from Alex Jones, for example.

"Project ENDGAME" is a plan of 2003-2012 of detention of 775,000 specially selected citizens in the concentration camps

US Orders Project ENDGAME To Begin, 775,000 Americans Targeted For Arrest

"The bottom line" is that one can not forget that we live in the world of evil most profound, and all the so called governments throughout the world and even the international organizations, such as UN, EU and so on, are merely the puppet theaters, stuffed with puppets that sold their souls for the benefits of stuffing their own bellies at the expense of all others. For example, Obama bluntly, and even jokingly states: "there is a theatrical aspect to politics" and governing. One can only add that it is all MOSTLY theatrical. The reality aspects of it are minuscule.

Look for FEMA concentration camps. Btw, hundreds of thousands of caskets, made out of special plastic and capable of storing at least 3 bodies in each one are stored all over the USA. Over 30 THOUSAND guillotines are stored in two states in the US alone. With that many guillotines you can "process" people at the rate of chickens on a chicken factory. So, we are talking about MASS murders of genocide scale, unseen in the entire history, done in the name of "protecting freedom and democracy" and "fighting terrorism". Can you imagine millions of "terrorists" or "extremists" in the USA alone?

So, the same exact people that were called the "dissidents" before, are now classified as "the enemies of people", the "extremists" and even the "terrorists", all of which is pure grade lies and fabrications, just as was explained by Nicholas Rockefeller to Aaron Russo in their friendly meetings.

(Note: you can find the interview with Aaaron Russo by doing a search on "Aaron Russo: Reflections and Warnings - Full Transcript". In this profoundly shocking interview he talks about his friendly meetings with Nicholas Rockefeller, who, for example, explained to him the scam of 911 and how it was meant to create a totalitarian state in the USA. He also talks about the mechanisms used by "the banking mafia" to rob the world and create mountains of fake "money", not backed by anything other than thin air.)

This is not just a bad joke, by ANY means. Except you need to study these subjects to even BEGIN to comprehend what might be taking place with this seemingly strange "conversion" of Syncthing into Pulse brand and all the alleged "support" by the corporate "sponsors".

The problem for the NWO "elite" is true information that discloses their evils and exposes their scheme of world takeover. That is why it is being suppressed on a global scale and the so called governments, who, in reality, are nothing more than mere puppets of the NWO puppet masters, have already passed all sorts of "laws" to suppress the true information by classifying it as either outright "terrorism" or "extremism". Interestingly enough, most of those "laws" and "executive orders" go against The Universal Declaration of Human Rights in the most direct and blatant ways.

We are in the global information war right now and of the magnitude never seen by the mankind in its entire history.

And that is precisely where sync technology comes into this picture.

Sync on P2P basis is the most urgent need in today's Information Technology

It is very unfortunate, and even sad, that even the "advanced crowd", the developers, do not seem to be "getting it". They seem to look entirely oblivious of the facts and events in the world that have the most profound significance and the impact on the present and future of the entire planet. They watch the puppet theater on TV and think that this is something that represents or corresponds to reality. Because via mass media their minds have been zombified to believe so. And, since their minds have been programmed with the ideas of "I don't care, this is not MY problem", they believe that "their world" extends to the surface of their skin, nothing more than that.

It seems that "the smartest of all", at least as they think of themselves, the developers, are either "playing dumb" and pretend that the issues of public distribution of information on P2P basis do not have any merits and "use cases", or have simply burred their heads in the sand, pretending that it is all nothing more than "none of my business", "let it all go to hell for as long as I can save my own skin and stuff my own belly" kind of thing.

It is like the only things that everyone is interested in is to sync their laptop to their camera, their server and a chip in their shoe, if not in their brain. That is ALL they seem to "care about".

Yes, all this sync stuff is good and all that, and it does serve some local, individual purpose. So, now you can push a single button and you get all your devices automatically updated with the latest collection of the zombie images, sound and video files and you name it. Push one more button, and it all gets delivered directly to your ears and eyes, so that you could get zombified faster with all sorts of images of the most horrendous violence, wars, killer "heroes", conmen of all kinds and the rest of the zombification menu of the NWO grand "plan", of which very few even know about, or even care or suspect that such a thing actually exists, and even if it does, "hey, what does it have to do with me and 'my interests'? It is not MY 'problem'"!. Then WHOSE "problem" is it? Do ANY of you know, by any humble chance?

And so, it is not entirely incomprehensible that the "developer-in-chief" even gets vicious in his utter denials of the very need for public information distribution issues, which is probably the most significant development this planet is going through at this junction, as far as purely technical issues are concerned.

Because how can you defeat the evil if you have no clue of what it is, how does it work, what are its tools and methods, who stands behind what, what are the "plants" and what kinds of schemes have been developed and implemented for centuries?

There is an ongoing massive attack on the Internet going on as we speak, and all the corrupt puppet governments and their agencies issue orders to block more and more sites, and even worldwide, without any court orders or proper legal proceedings in gross violation of the The Universal Declaration of Human Rights, articles 10, 11, 18, 19, and 30.

http://www.un.org/en/documents/udhr/

Because the powers of evil that have already taken over this planet for the most part and control over 80-90% of the world's gold, precious metals, diamonds, energy supplies, banking system, media and publishing, and have taken over the money printing business for all the major world currencies with the help of the FRS scam, know very well, that their demise and inevitable defeat begins with true information, disclosing the nature and the mechanisms of evil and principles of world takeover.

They were able to suppress and destroy all sorts of most vital information about the nature and mechanisms of evil, and instead, via the zombification machine called mass media, were and are feeding everyone lies, perversions, deceit and the ideology of violence and destruction, and have programmed the human minds to be interested only in the most idiotic, perverted, corrupt and violent things, promoting greed as some kind of "objective necessity" and a basic mechanism of "survival", and so they were able to do to this planet anything they wish in pursuit of their final goal of taking over the world via the scheme known as the NWO.

But, even despite their most vicious methods of destruction of information and the authors and publishers, all of a sudden, it all floated back up to the surface, seemingly out of nowhere. And that is their most deadly problem, because it has spread out the waves of energy of awareness throughout the planet and the people throughout the world have begun to revolt against the entire satanic doctrine of Luciferianism, that has been ruling this planet for thousands of years.

But to stay well informed about methods and history of evil, you now need the tools and methods that allow you to switch to a different mode of operation. Instead of attempting to access the sites and collections of information directly, you need to be able to access it no matter what, on P2P basis. There is simply no other way to prevent and stand against all the tools and methods of blocking the information of all kinds even bypassing the law itself, and unless millions upon millions of people know exactly what they are dealing with, there is no way they will be able to defeat the most powerful and sophisticated system of evil that has been developed and perfected for thousands of years.

Before you can stop the evil, or cut off the parasitic entities that suck all your resources of all kinds like a black hole, and restore the environment of this planet and human dignity and real fairness in all sorts of things and activities, you need to KNOW how that system of evil works. You need to study it all, which used to take years until now, but now can be done in months, if not days and some things - in minutes.

Or do you expect that some "noble knight on a white horse" will come and "save" your sorry rear end, while you are sitting on it, doing nothing and pretending you are merely dumb as a piece of wood, not to insult the piece of wood?

WHO is going to come and "save" you if you are not willing even to move a finger to save your sorry ass which is on fire, regardless of whether you realize it or not?

For how long you, the false "kings", are going to deny the utmost need and urgency of implementing the global public sharing of all sorts of information via sync technology, which is the only technology that allows you to stay current even with the immense amounts of information via single collection and a single key?

The thing is that even torrents can not do that. Because torrents are static. Once created, that's it. They can not be updated, extended, modified and so on. How many times do you need to be told all this before it clicks in your "sophisticated" minds?

Look, you "smart guys", even a FIVE year old child can comprehend it. Why can't you? Are your brains so profoundly damaged and zombified to the point that you can not put two and two together? Are you going to ask "but what are the use cases for that"? Try to "prove" that you need to even bother about two plus two. "Who NEEDS it"? What are the "use cases" for that?

Well, what can we say but "it depends on how dumb you are", or pretend to be! It is not inconceivable that by now even Life as such is nothing compared to all the images of death they feed you non stop, any place you look.

Yes, the situation on this planet has never been as pathetic and as hopeless as it is right now, and that is precisely why you have a chance to correct it, once and for all. You need to come to a state of utter hopelessness and desperation before most of you even move a finger, and even then, only for the exclusive purpose to "save their own ass", pathetic as it is.

This is all you get from us in this transmission. Either you wake up, or be prepared to face the consequences of your denials, utter insensitivity towards each other, and even Life itself, and utter absence of care within you. Simply because your minds have been totally zombified with the ideas and images of death, destruction, violence, parasitism, cunningness, corruption and the rest of it.

It is up to YOU to decide which world you wish to live in. No one can do it for you. Because it is against the Laws of Free Will and Free Choice to interfere in your affairs, whatever they might be, unless it has all come to its final, logical fruition, at which point, it can no longer be allowed and the forces beyond visible simply have to get involved directly and in the most tangible ways conceivable.

Comparison of file synchronization software

Here's the link to Wikipedia article about different sync packages.

Comparison of file synchronization software
https://en.wikipedia.org/wiki/Comparison_of_file_synchronization_software

Open source sync alternatives

Introduction to open source sync alternatives

This is just a stub for now. This section will begin to accumulate the information about the open source alternatives to BTSync.

It is our opinion that BTSync, at least the way it is right now, has no future, and for quite a few reasons:

  1. The last updates from 1.3.109 to 1.4.72 and higher include the major GUI change and the position of the management is "this is the way it is going to be from now on, regardless of anything anyone says or thinks". And that means a catastrophe. The "latest and greatest" GUI is web browser based and it is a huge step back from the previous design, which provided a reasonably detailed view on dynamics of nodes update.

    In particular, what used to be the Transfers tab was very useful to see which file is sent/received from which node. You could see the simultaneous transfers to/from different nodes and that gave you a sufficiently good idea of what is going on. That tab is gone now and you can not see the actual transfers taking place.

    The Devices tab in old design would show one all the shares/folders and all the nodes participating in those shares. So, in a single screen you could have a pretty good overall idea of what is going on. That screen is gone now and you have to click on things just to open up another screen or semi-dialog, where you still would not see what you could in previous design. And so you have to click more and more times to see anything useful, and even there, its usefulness is questionable and information provided is clearly insufficient.

    This fact alone is a killer. If that is the way it is going to stay, then BTSync is dead. Quite a few people have decided to go back to previous version.
  2. It is a closed source software which implies that you do not really know what happens "under the hood" and you can not trust a word about claims of encryption of traffic or increased privacy, first of all because no claims of encryption or increased security or privacy can be made in a closed source code, pretty much by definition. Any security specialist knows that as good as 2+2=4.
  3. The incredible number of bugs in BTSync and its generally poor stability and consistency of behavior, combined with the fact that it is a closed source code, makes it impossible for other developers to fix piles upon piles of bugs, problems and the issues.
  4. BT strategy on BTSync seems to be a desire to corner the sync market via distribution of various "bundles" of zombie/biorobot level of entertainment. As a result of this, and, possibly because BT may have some contract arrangements to get "the piece of the pie" of total volume of traffic in the "entertainment industry", it is only "natural" for them to hard-wire the mechanisms of tracking the traffic of any node running the BTsync. Therefore, whenever you run BTSync you leave your footprint at BT.
  5. There are reasons to believe, or at least suspect, the cooperation of BT with NSA and other agencies engaged in global surveillance and you can find some references about it in this manual.
  6. BT is primarily interested in creating as much hype as possible, instead of fixing bugs in the software to make it stable, reliable and have a predictable and consistent behavior. Instead, they keep developing all sorts of new bells and whistles to entrap more and more unsuspecting customers or clients in order to control as much of global sync traffic as possible. They regularly brag about the mind boggling statistics of global volume of sync traffic, and to have that ability, they'd have to hard-wire the mechanisms of informing BT about any and all sync traffic that is tracked by BT servers, trackers and relays.

    In fact, you can not even turn off the ability of being tracked even if you disable their trackers and relays. Because the program simply downloads the list of servers automatically, without informing the user. Even if you disable their trackers and relays, still, for some magical reason, you can discover the other nodes, even if you disable the DHT.
  7. BT keeps bringing up new and new features and "improvements" in a product which they themselves classify as Beta, which implies that the product is not only becoming more stable, but just the other way around, less and less stable. In beta stage, no development or new features are allowed. Otherwise you will never get to the state of the final version. Because bringing in new "features" will bring in new bugs. But they not only keep bringing new "features", but even do some major and quire radical redesign of the most basic mechanisms, which is an absolute "no, no" in genuinely Beta phase of development.
  8. BT is not willing to disclose the necessary information to create the compatible products, which, by itself, is an evidence of their desire to monopolize the sync market.
  9. The project management, supervision and technical competence of their developers are of such a low grade that it isn't even laughable. It is simply pathetic. For experienced developers, some of the design errors and incompetence in technical sense are simply obvious, and would have never be allowed if level of technical competence would be on the competent, professional level.

    Just looking at one obvious example makes things pretty obvious. As it stands, BTSyncs begins to "forget" the nodes it knew about on a particular share if you keep the program running for several hours. But this should never happen, simply because once the node discovers any other nodes there is no logical way for it to "forget" them. How could you possibly forget something you knew? This is evidenced by the fact that if you restart BTSync it will again find those nodes that it "forgot" about before you have restarted.
  10. There are other numerous examples of poor or even bad design in GUI, and we have mentioned a few of examples of it on BTSync forums months ago, but no improvements can be seen even today. In other words, they did not even move a finger or even looked at any of it, like it is something utterly insignificant. But to the users it would provide a much better and more complete and detailed picture on the whole thing and some of the ideas we proposed would even help to identify numerous bugs much quicker then it is possible now.

We could continue this list for quite a while, but what's the point?

As a result of this, it is pretty clear that one can not expect BTSync to become a stable and reliable product, at least in the foreseeable future.

That is why we began to accumulate the information on developing the open source alternative, and it will start right in this manual, which is no longer dedicated exclusively to BTSync, but, rather, to the sync technology as such.

Highly desired features

This is just a stub for now. This section will begin to fill with those features of sync programs that distinguish it from the "stone age" of syncing.

First of all, it is not a bad idea to keep in mind that syncing, as such, is still quite an immature technology. Furthermore, it is a highly complex from the standpoint of all sorts of logical cases or consequences of all sorts of conditions and events, and there are so many of those, that we can not quite enumerate them all.

Some of the most critical parameters of the sync process are as follows:

Additional problems of an open source implementation

This is one of the aspects of the open source that could be easily overlooked, which might cause the most drastic problems. It is applicable to the situations where the information exchange does not necessarily happen within some trusted group of people or between different devices belonging to one individual. And this is particularly applicable to any public information exchanges, regardless of anything else.

The thing is that in open source alternative you have to be extra attentive as to how you design your algorithms and various data structures passed between the nodes and how do you identify various things with confidence that they are not faked.

For example, in BEP (Block Exchange Protocol) definition there is a concept of a local database, in essence, which is sent out to all other nodes upon request. Now, in r/o nodes, there might be some files that are not present in the original master node (r/w), and so, various decisions might be made by other nodes as to ways in which that additional information may affect the other nodes. Because that information may look like a new version of some existing file, or a new file that has been stamped as being present on the master node, while it was merely faked.

In that case, some attacker, who does not even have a key to be the master node, may effectively cause an update even of other master nodes, just because, for example, he might stamp some files with some flags that indicate that they originally came from some master node, which is not true. In that case, one master node may decide that some other master node have added a new file, which may cause a download from the r/o node of any garbage files created for exclusive purpose to cause damages to the share/folder/collection.

Therefore, an extra rigid update logic needs to be implemented in order to assure the validity of any files and data structures floating around, and and the overall consistency of data across all the nodes in a share.

Generally, no r/o node should be able to update any r/w nodes under ANY circumstances whatsoever, even if some file looks like it was obtained from some master node by faking some bits in the data structures passed around.

Secondly, no file may cause an update of other nodes unless there is a guarantee that it definitely originates in some master node. That means that the file hash on the originating master node has to be generated in such a way that faking the originating source becomes impossible, because the originating flag (master version) is encoded in the very hash and could be checked after its decoding and, possibly, comparing the hashes of the file pieces or even the entire physical file against the hashes.

Basically, no information, bits or statuses of nearly every single data structure passed around may be trusted unless they are encoded with the necessary identification stamps.

The only data that can be trusted is the data that is guaranteed to come from the master nodes, or guaranteed exact copy of it.

This all means that even if someone modifies the sources for the program and recompiles it and then tries to use his version as an attacking weapon, he should not be able to do any harm or damage to the data, nor to inject, delete or modify the originals. At the most, he might cause some extran network traffic, which is also undesirable as this could be used as a permutation of some bandwidth attack that might choke the nodes with utterly fake data.

Guaranteeing the master version

In order to fulfill the requirement of data consistency across all the nodes, about the only realistic alternative is to make sure that the only files that are propagated around, including from the r/o nodes, are the files that are GUARANTEED to be the exact copies of the master version. And that is easier said than done in an open source setting. Because the attacker may modify his version of the program and set any elements of the data structures and any flags in any way he wants.

This means that no transfers of any files should be possible unless there exists a guarantee that either at least one master node is on line, so things could be checked against it if needed, or there exists a copy of the master node database, which has all the necessary information about files, hashes and so on. That means that every r/o node first of all obtains a master database, which may even come from some other r/o node if there is a guarantee that it was not faked, which is possible. The master database needs to be stored permanently on each node, so that it survives across the program restarts.

One thing we know about the master nodes that make them different from the r/o nodes is the KEY. No r/o node knows about that key. Otherwise, it can become a master node any time it wants.

This provides us the hook via which we can assure that the things really originated at the master node. So, when we hash or encrypt things on the master node we can use the master key similar to security certificates, private and public. The mechanisms are pretty much the same.

We may not even have the latest version of the master database on our node because no master nodes are currently on line. But that is not a problem. We can still operate and transfer files even between the r/o nodes that might even update the other r/o nodes with new files, because those files are stamped in unforgeable way to prove that they came from the master node.

But when at least one master node comes back on line, we can update our r/o node databases with the latest and greatest version and possibly update some files that are newer or fresher than those we currently have.

With this strategy we can be sure that even if some attacker uses a custom version of the program, modified for the purpose to attack, it is not going to work. Because he will not be able to encrypt the things the same way as the master nodes do, simply because it does not have the master key.

Synthetic synchronized torrent approach

One of pretty interesting and not quite apparent approaches might be simply using the existing open source torrent code and extending it with sync functionality.

For example, if we take some portable torrent client, written, for example, in Java, such as Vuze/Azureus, then we already have the bulk of the work done, including the DHT discovery, UTP protocol implementation and so on.

From there we have several alternatives:

Basically, there are variations on this theme and things need further consideration and a more detailed analysis. But what we achieve with this approach is quite a bit. First of all, we would already start with a fully functioning torrent program, and, if it is portable across several platforms, we can immediately start working on implementing the sync functionality in small incremental steps instead of starting the whole thing from scratch.

Single modifiable torrent approach

Let us consider the single modifiable torrent approach to syncing. The goals of this approach is to minimize the amount of development effort to get the initial version going, while, at the same time, assuring that all the nodes are updated to the "latest and greatest" version.

Furthermore, it has to provide the guarantee of data consistency across all the nodes, meaning that all the nodes contain the correct and "latest" version of all the files. (Note: latest is in quotes because the very notion of file time stamp is quite illusory, at least in cases of a deliberate "time stamp" attack on some share.)

Also, it would be interesting to investigate the possibility of uniting the ordinary torrent functionality while extending the torrent client program to also do the sync operations. There is quite a bit of common information GUI-wise between the regular torrent operations and syncing, which makes is worth of investigating it further.

It might turn out that the easiest way to implement such an architecture would be to stay as close to the original torrent transfer concept as possible, and that is - to use the torrents for the entire collection rather than doing it in a more fine grained manner by constantly constructing smaller torrents for only a part of a collection. But there might be potential performance degradation in case of using the single torrent for the entire collection versus a number of smaller torrents that only contain the modified or new files.

But single torrent approach might exhibit the additional difficulties of synchronizing the updates in case there is pending transfer of some torrent while some file was updated in the middle of transfer operation. But this is one of those more "esoteric" aspects which, by the way, would have to be dealt with even if we handle transfers by construction a number of smaller torrents representing the deltas only.

This approach assumes that all the top level communication exchanges begin with top folder hash. Top folder hash is the combined hash of all the files in a share/top folder/collection. Any further action is required if those hashes differ on different nodes. Else, the sync state has been achieved and there is nothing to do beyond periodic scanning to recalculate all the hashes and, ultimately, the top folder hash, which is the main or "key" hash for the share.

About hashing.

The idea is that you gradually and incrementally reach the state where all the nodes on a share contain the same collection, represented by the same torrent file. That is the stable and final state of the share and it implies that all the nodes are in sync.

But there might be a difficulty here because there is a contradiction. On one hand, we are trying in effect to create a single torrent that represents the master collection. That would imply that all the nodes would eventually end up with the same torrent file when they are all fully synchronized.

But, since any node may ignore some files from the master collection, that would mean that their version of the torrent might be, and is likely to be, different than on a master node. But there might be a solution to that, which allows for a SUBSET of the master node files to be distributed. Because it still valid, correct and current with the exception that it is not complete.

But it is just one node's view. When you contact the other nodes, you are likely to see a slightly different version of the same collection, except THEIR ignores are likely to be different. But it does not matter. What matters is that they might have some files that are missing in the first node, and that is what matters.

Because you enlarge the view on the complete collection by using the union operation on all the data on all the nodes on line. The more nodes, the more chance that your own version will be as close to the master version as possible.

But the files themselves are exactly the same as on the master node(s). This needs further analysis. One of the solutions might be to have the ability to cancel the pending transfers once we detect the change in the file system and then restart some transfers once new torrent file is constructed.

This is just a rough sketch which will require much more detailed analysis of all possible cases. But this is something that might turn out to be the easiest and most effective and simplest way to go about it by starting with some open source torrent client code base.

Single modifiable torrent approach via magnet link

One interesting aspect of using a single torrent file for the entire collection is that if that torrent file is in standard .torrent format, then it can be either made directly available to download by anyone, or even better, you might refer to it via magnet link, which is effectively the same thing as a key to the collection, except it does not have the torrent file itself. But the magnet link itself provides further information for you to recover the torrent file itself. The link is the same, but the content of what it points to is different. That makes it like GUID of some sort. The link itself is unique.

That means that you simply provide the magnet link and it is eventually resolved into a torrent file. In the most general case, the magnet link contains only the hash or a key for the collection. The peer discovery happens via DHT protocol. But it has the provisions to provide the addresses of several trackers, just like torrents. If no trackers are provided in the magnet link, the only way to discover the peers would be via DHT or via exact or the fall-back exact address of a torrent file. You can specify those in the magnet link text.

Basically, the magnet link differs from the direct torrent approach in that it represents the mechanism of mapping a unique torrent content to the universal and ever valid identifier to the collection. There is really no need to carry around the static torrent files that inevitably get outdated. All you need is the universal one-to-many mapping mechanism, which is what the magnet links attempt to provide.

The actual torrent file corresponding to the magnet link is obtained directly by the peers. This means that in our approach we do not need to pass around the torrent files themselves, and, secondly, as soon as we discover at least one other peer, we will obtain "the latest and greatest" torrent file from them directly. Now, once we discover at least one peer, from then on, regardless of how "old" or outdated our own torrent file is, regardless of how we obtain it, we, from then on, are guaranteed to eventually obtain "the latest and greatest" version of the torrent and collection.

This approach could be quite interesting to consider. Yes, it would work fine if collections do not change. Otherwise, you have to make sure that torrent clients always refer to this share via magnet link, and not via torrent, every time they join this.

So, this means that by simply modifying the magnet link text you effectively refer to a different torrent file. This needs more analysis though.

Sync general design issues

Several things may be noted here.

The basic idea used by BTSync is to use torrents as the underlying mechanism and an engine to do the transfers and updates.

This is a pretty good idea, and for several reasons. First of all, there is no need to "reinvent the wheel". The entire transfer mechanism is well developed by now and is available in open source versions.

The basic idea is simple enough. You simply use the same torrent transfer engine except you create the dynamic and temporary torrents on the fly instead of creating a single torrent of fixed contents for the entire share or collection of information.

The MAIN disadvantage of static, per collection, torrents is that it can not be updated, extended, modified and so on. Yes, its main advantage is that it guarantees you the integrity of a collection and gives you an assurance of reliability of data and guarantees that no data within the collection may be modified. But that is not what is needed in the modern information age.

The nature of information in general is dynamic. It changes all the time. And that is why torrents get outdated. But the problem with them is that they can not be changed once created. You'd have to create new and new torrents for new versions of information, which is highly undesirable. It would be preferable to keep a single version that could be updated, extende and so on.

Now, since torrent transfer engines and mechanisms are a mature technology by now, it does make sense to use them instead of inventing the whole engine again. And that implies that at least 30 to 70% of the work is done once you can use the same technology and create the dynamic and temporary torrents to perform the transfers.

What remains at that point is the high level control logic and GUI. At that point, you have about 90% of all the work done. The remaining mechanisms are key mapping to share/collection, node discovery, such as using the DHT or "predefined hosts", used as a general purpose trackers, and various hashing algorithms and traffic encryption. And that is all there is to it, pretty much.

None of it is that "esoteric" once you start looking at it with a competent enough eye.

So, this is the direction we will address in this document.

Do we need to be BTSync compatible and why?

Well, first of all, since BTSync source code is closed and no information is released that will allow to create a compatible product, and no description of either their protocol or various key algorithms, used to construct all sorts of hashes, is available to the public, making the fully compatible product would be very difficult if not practically impossible. Some of the algorithms are so weird that there exists no realistic way to make something fully compatible without using the reverse engineering techniques.

But WHY do you need to be fully BTSync compatible to begin with? What does it buy you?

hobarrera said:

Here's a couple more points:

* I belive the BT Sync protocol is propietary too, not just the implementation.

Well, let them be as proprietary as they wish in their attempts to control the sync market. But you don't HAVE to be compatible to them. In fact, it is THEIR "problem" to try to be compatible with you, if your higher level logic is well designed and YOUR version is more stable and reliable and more consistent and assures the database consistency across ALL the nodes regardless of anything.

Because, at least currently, their protocol, if they have such a notion to begin with, is NOT consistent and does NOT assure the data consistency across the nodes. And this is a fact, which I have seen personally in numerous cases. In fact, I can provide a number of cases that could be easily verified independently. I can even claim that I have never seen anything so inconsistent as their "protocol", if there is any. From what I was able to gather so far, they are simply using the UTP protocol and all its mechanisms. I have my reasons to doubt they even have such a concept as a higher level protocol and logic. It all seems to be wired together in ad-hoc manner.

So, by the sheer fact that you are an open source and can provide as much of protection against snooping, security and so on, YOU become the standard by which others are to measure. Because your source is open and can be verified, and, which could be even more important, ANYONE can improve on it, extend it and you name it, if they have what it takes to improve on a tank, and if this is acceptable by other competent developers, it will be wired into the next version or so. Why not?

The "bottom line" is you do not HAVE to be compatible with BTSync, or anyone else for that matter, any way you look at it. If your system works and is reliable why should you even bother about BTSync? Sure, if they ever decide to go open source, which they won't, at least in the foreseeable future, and they have proposals on how to improve things, then it is a different matter. And even then, you are not forced to follow up on their proposals if you find they do not actually "buy" you anything of real value.

I'd be willing to discuss this issue with anyone having an argument.

hobarrera said:

* Pulse has local network announcement (eg: nodes in the same LAN can communicate offline). BTS needs to be online or have access to a tracker to find nodes.

Well, these things are minor issues as far as I can see, and so are the issues with network protocols, such as TCP or UDP, or even ICMP, need be.

In fact, you can utilize the synthetic approach and use some protocol for a specific operation only, and not system-wide, that gives you the most benefits and provides better performance.

For example, you may decide to do UDP broadcasts or send packets containing a slice of information on the current state while contacting other nodes you know about at this point.

Update logic engine

One of the most critical aspects of sync process is stability and predictability of data in the context of update.

It is a tricky process if you allow all sorts of funk and bells and whistles, like it is done, for example, in BTSync. The main problem with their update logic is that if the r/o node modifies the files in any way, then it will no longer be updated from the r/w (master) nodes. And this is a disaster, at least from the standpoint of knowing what is going on, and the effects of such a design are profound and totally unpredictable. BTSync logic is described in the transfer table.

BTSync behavior table

Current file update and propagation logic problems

It is proposed to use a much more strict update logic, based on few very predictable criteria that assures the stability and consistency of data across all the nodes. It is based on several principles:

Higher level logic engine issues

Core principles

First of all, we will use the term "system" instead of a "program", as in our understanding, the programs better be viewed as systems and not merely the collections of code. The very definition of a system implies much more rigid requirements, the main of which is system STABILITY. No matter how "sophisticated" is the system, if it is not stable under any conditions whatsoever, it will, sooner or later either degrade or collapse entirely.

The second criteria, which is of vital importance, is data consistency. In case of a sync system, consistency is applicable across the nodes and means that there is only one valid copy of data floating around. That is, in a stable state, meaning no updates are in progress at the moment.

It is highly undesirable, and in fact fatal in some sense to have different versions of the same file be available on different nodes, even though during the file modifications or updates we do have some files that are inconsistent with the rest of the system. But those states are transition states to a new stable state, and are subject to the most rigid update logic to make sure we do not propagate the inconsistent data around.

This, first of all means, that the r/o nodes are not allowed to modify, delete or add any files that are not present on the master (r/w) nodes. This is what we call here the "hard core" logic. If this requirement is not followed, then disastrous consequences are pretty much inevitable, and for quite a few reasons.

For example, in BTSync, as of versions 1.4.nn, the r/o nodes are allowed to modify, delete and add files, and under some circumstances, as a result of what seems to be clearly a bug, they may even update the master nodes, which is nothing more than the invitation to disaster. Their reasoning is based on a rule "the user knows better", which is fine and dandy, except it does not work, and for quite a few reasons.

So, in case, for example, when some user deletes some file, which immediately creates a data inconsistency across the nodes, his deletion is considered valid, except it is only applicable to his own node. This means that this file will remain in its own database but will no longer be updated from the other nodes. This, in turn, creates a nightmare. Because, first of all, the master node can no longer report the status of completion of that particular r/o node an so, the "main" node - the master node, does not really know what is going on with the share, and from several different points of view. But let us not diverge from the main thing here. BTSync update rules and logic is described in:

BTSync behavior table

Higher logic operational components

First of all, the higher level logic engine is the most complex part of the system. Basically, the operation of higher level logic extends to scheduling a transfer. Beyond that, higher level logic is no longer applicable end we enter into a phase of plain, ordinary issues of of assuring the delivery of a correct copy of some file. So, once the file is queued for transfer, that's it, "we wash our hands".

So, in effect, the higher level logic deals with the issues of discovery which files are to be updated and scheduling of updates.

Now, the issues of discovering which files need to be updated are, for the most part, the issues of efficiency and performance. Because fundamentally, there is a "brute force approach", which means, that during the periodic scanning to find changes, we may simply go through every single file of the share and compare the copy (of the file and block/piece hashes), available on disk, with the copy available in the database, and if there is a difference, then transfer is scheduled. Yes, it is the least efficient way of doing things, and yet, it is most reliable and trustworthy.

Master and local databases

In our design, we chose to store two separate databases for each share/folder/collection. First of all, we must have the local database, which contains the latest state of the share from the standpoint of local view on this particular share. But we also chose to store a master database on a local node as well. This is quite valuable because of "hard core" logic, which says: You transfer ONLY the files that are guaranteed to be the exact copies of a master node(s) version. No file transfers take place if we do not know for fact that what we are about to transfer is an exact copy of a master version. Else, the eventual inconsistency is pretty much inevitable.

But we do not need to store the other nodes version of their database on any other nodes, but their own, on a permanent or even on a temporary, per session, basis. Because "we could care less" of what they have. About ALL we "care" is whether we are consistent with them, and if not, then do they have something to contribute to us and vice versa.

This could be done via procedure of incremental discovery (of differences) by first examining the top level hash for the entire share, and then, if it is different, comparing the hashes of subfolders. If any of them differ from our version, then we descend into those folders and now check the individual file hashes. This optimizes the overall performance and reduces the need to carry around useless information about anything.

And contribution does not require a complete knowledge of their entire database. It only needs deltas, the differences, which are, in vast majority of cases, are but a very small part of total database. Furthermore, it is virtually useless to store their entire database on our node. For what? Most of it is exactly the same as our version.

So, what kind of information is stored in the database? Well, we certainly need a file name, actually full path relative to the base folder of the share. Then we need the file hash, and, possibly, for efficiency reasons, the individual block/piece hashes for the file. We also need to store several state bits.

If we decide not to store the the file pieces hashes in the database, then we will incur an additional performance degradation and, in some cases, may not even be able to make the correct decisions as to what to do next. Because there is no guarantee that the master node is on line, and so, we can not even query it from the r/o node in order to get the individual block/piece hashes for this file (This is applicable if we do NOT store the master database on each node. Otherwise, we could query any r/o node about their view of the master version and base our decision on that information).

Secondly, even if we could, we would create the otherwise unnecessary network exchanges and traffic, which could significantly impact the overall performance, especially for the cases of small files. With large files, the additional network traffic to get the block hashes could be minuscule compared to total file transfer time.

(Again, most of this logic is related to efficiency issues and not something fundamentally necessary or required.)

Yes, if each r/o node also stores the master database on a permanent basis, then we do not really need to refer to the master node directly. We can obtain this information from any other r/o node provided that its version is guaranteed to be correct. This is the key. We do not store, nor carry around anything that we can not trust.

File state flags

So, this is pretty much all the information that needs to be stored in the database about each file.

Cryptographic information signing

One of the issues of critical significance in the context of public information sharing and open source nature of the system is the authenticity of things. The assumption here is that everything may be forged by the custom build version of the system. Therefore, things of critical nature need to be processed via some cryptographic technique, such as PGP for example. In this light, let us see which information is of critical nature and significance and needs to be signed.

File scanning

File scanning is an attempt to find out if any files have changed locally on a particular node. It is scheduled on a periodic basis, and, even if our O/S supports the file system notification events, some of them might have been missed, if, for example, the program was not running and file modification happened during that time. So, regardless of anything, we do need to schedule the periodic scanning which is meant to detect any file changes with certainty. No other mechanism will provide us the definite answer on any file modifications.

File scanning is also performed on the program start as a default. During that startup scan we verify the validity of our local database for this node. Alternatively, we may decide not to perform the startup scan, but rely exclusively on the periodic scans. It really does not matter eventually, because sooner or later we still have to rely on the periodic scans.

Mostly, the strategy of scanning reduces to optimization of efficiency, because, otherwise, we could simply use the "brute force" technique and perform the physical reads of the files on disk and compare the piece/block and file hashes calculated during the read to the hashes available in the database that tell us what was the state of any particular files. If our newly computed file hash is different from the one in the database, then we certainly have the file modification case. The same thing applies to pieces.

Basically, during the scanning we are looking for the changed, new or deleted files. In case of deleted files everything is very simple. We simply can not find some file on disk that is present in our database. Therefore, we need to pass this information to other nodes and let them decide what they wish to do with this case. In particular, it depends on whether the deleted files are or are not on the other nodes ignore lists.

In case of new file creation, things are still relatively simple. Some file did not exist in our database and now it is present on disk. Or, alternatively, it did exist in our database, but was marked as "not existing on disk", which means that it did exist at some point, but then it was deleted. No matter what, we need to pass the new state of this file to other nodes and let them decide whether they want it or not.

In case some file was modified, things get substantially more complex, especially in the light of "hard core" logic we adopt here. Several things may help optimize the process:

The special case of detection of file modification is the O/S file system events, such as delete, modify and so on. But some operating systems do not have such a facility, so this becomes merely a performance optimization and we can not rely upon it as a general case.

Once we detect the file change(s), we simply schedule the node notification mechanism. But we are no longer concerned with how and when it is done.

One other optimization is to try to schedule several small files and pack them into a single delivery via building a single torrent for a number of small files. Because the network overhead, especially with TCP connections, may be significant if we keep bouncing back and forth about every individual file. But again, this is merely optimization.

Polling queries for updates

We may also perform the polling queries to see if some other nodes have something different than what we have. Since we are dealing with on/off line situations all the time and any node may come and leave any time they want, and so we are, we might like to explicitly query the other nodes to see if we are not in sync with them.

For efficiency purposes, we don't poll for every single file in a share. First of all, we poll for the top folder hash and only if that hash is different than our own, we need to perform the more detailed discovery and request the sub-folder hashes, and if at least one of them is different than ours, then we request the individual file hashes for that folder.

In most cases, polling for updates is a waste of resources since we are in sync with all the other nodes most of the time. But it does not hurt, at least if used in efficient way.

If we find some file hash "out there" different than our own, we then request all the different particular pieces of that file.

Basically, file transfers happen upon request. No node forces any other nodes by transmitting anything to them, unless those nodes specifically request things. It is up to that node to figure out where it is and request for any further information to clarify the states of the particular files.

Polling can be performed either on periodic basis, like scanning, or during the system startup.

The first choice to poll is the master nodes, if there are any on line. If we are consistent with ANY master node, we do not need to do any further polling, at least according to "hard core" logic. Because once we are on line and any node has any changes, we will be informed by it as to those changes. So, there is no need to poll.

But during the startup, if we find that we are inconsistent with our own database, meaning that some file modifications, additions or deletions were performed while the program was not running, then we may chose to do some hard polling in order to reconcile with all other nodes as soon as possible.

Share/Collection degradation problem

The main disadvantage of the sync approach compared to torrents is the collection degradation problem. Which means that once all the master nodes are gone and no longer appear on some share/collection then there is no certainty that any r/o nodes that visit this share do have the entire collection as it existed on the master nodes.

Furthermore, an extra measure of care needs to be taken in order to make sure the new r/o nodes may even start downloading the share. For example, if it is required for the master node to be present on line in order for the r/o node to construct the correct database, then the r/o node can not even start downloading anything because it never has a chance even to build the database.

Now, if we carry the cryptographically signed "master version" flag in a database for each file, then any r/o node may contribute its part to the new r/o node database, even if their part is not the complete master node database. This means that even if some r/o node on line has a single file and no other r/o nodes are on line at the moment, the new r/o node can still build its database even though it contains a single file, and it can download that file since we know for fact that this is an exact copy of the master node version. Later on, when some other r/o nodes come on line, then the database for the new node gets expanded.

Basically, with "hard core" logic, we can assure the theoretical maximum fitness of the resulting share to the share on the master nodes, which can assure that the resulting database will indeed be a union of all the individual files on all the nodes that ever visit this share. We can do that because in our logic we are guaranteed that the only files that propagate are those that are cryptographically signed to be the exact copies of the master node version.

This means that the very notion of share degradation becomes somewhat inapplicable. Because the very fact that we do no longer have any r/w nodes on this share, even permanently, we still have the latest snapshot of the master node version of the share, which, in turn, may be logically equated to the "latest version of this torrent ever released" in torrent terminology.

And again, probably the main advantage of this approach is the fact that we can start downloading the new node(s) even in utter absence of the master nodes on line, and so this share in reality "never dies" or "never degrades" as it will live on in as close version to the master node as possible.

Actually, the behavior would be exactly the same as if you got some partial torrent. Meaning that at some point when the torrent seeding node was last seen on line, there was only one other leecher node that was able to download only a part of total torrent. From then on, if no seeder node ever comes back on line, this torrent will remain partially complete until the end of times, regardless of how many other nodes join that torrent, unless they have some files to contribute to the incomplete torrent.

This is excellent and quite an unexpected result that removes probably the main disadvantage of the sync approach vs. pure torrent approach. Furthermore, it removes the second disadvantage of the sync approach, which is the collection is not guaranteed to be healthy, unlike with torrents. With torrents, all the files that were downloaded completely are guaranteed to be healthy, which would be impossible with other sync designs that do not utilize the cryptographic signature of the master version.

This design once more reasserts the design principle of "no r/o node, under ANY circumstances, may distribute any non cryptographically signed versions of the files" guaranteed to be the exact copies of the master node versions.

Furthermore, no r/o node even informs any other nodes about the fact that it may have some other files in the share directory tree that are not a part of the master collection. Those extra files are considered to be a part of the local version of the file tree in the file system and have nothing to do with the share as far as sync process is concerned.

Any existing file(s), that were renamed to the new names, effectively "move out of the share" and no information about them has to be kept in the share's database, and, if they are renamed back to their original name in the future, they are still to be re-downloaded as though they did not exist in the share. Because once they are renamed to the new name, they "fall off the share" and we loose their cryptographic signature kept in the database only.

The logical requirement for such seemingly harsh treatment of renamed files (vs. merely renaming them back without the need to re-download them again) is that we also need to keep their cryptographic signature when they are renamed, if we expect them not to be re-downloaded once they are renamed back to original name. So, if we keep them in the node's database with their signatures, then we create the new files with the master node status and they become subject to the propagation, unless we utilize a special mechanism to handle that case possibly with the help of some extra bits in the database. One way to resolve this conflict would be to also mark the newly named files as "non-updatable" and "non-propagatable" in the database.

In that case, even though we do keep those files in the database, we no longer either update them, nor propagate them to other nodes, nor do we even inform any other nodes that we do in fact have these files under new file names. But probably the simplest and most straightforward solution would be to simply move them out of the share and "forget about them" as far as this share is concerned, because unless those are huge files, re-downloading them again should not represent such a big problem. Plus, the solution becomes much simpler and we do not introduce any probable inconsistency issues if we have to juggle a couple of extra bits in the database. But this can be argued.

Share/Collection index and database

There is a need to keep the state of the shares/collections on per file basis because during the time the program is not running there may be some changes in the file system, compared to what we knew about the share, and we need to detect the changes since the last time the program was running and watching things.

When the program starts up, it can perform the startup scan to determine "what's new out there". During this scan we can use either the database for the share and check and see if the physical files in the file system are the same as in the database. We may also be interested in scanning the file system and check it against the database in order to discover new or modified files. We can not discover the new files if we only use the database as a reference, that is, we only check those files that do exist in the database. We have to do it the other way around as well.

We need a high performance database engine because we may have some huge shares with hundreds of thousands of files, which is not uncommon. Also, the files may be huge in themselves. So, we need a high performance indexing mechanism to quickly check some file and get the information we need to see what to do next.

Each share/collection has its own individual database. When some share is deleted, all that happens is deletion of its database. No actual files are deleted for variety of reasons. If user wants to delete some share entirely, first he has to delete its key/folder in the program and then physically delete the files from disk.

The database needs to store the file tables using the file path (relative to the root folder of the share) as a primary key. Here is what we need to store in the file table:

When new share is created on the master (r/w) node, the initial indexing is performed and the database is constructed for the share. The process of construction is gradual and some sync operations may start even before the share is fully indexed. This is important for the very large shares that may take minutes, hours or even days to fully index. Basically, as soon as we fully construct a row/record in the database we can start using this file for sync operations.

On the r/o nodes, the database may be constructed even if no r/w nodes are present on line at the moment (if we cryptographically sign each file in the database to indicate that it is the exact copy of the master node version).

Basically, the r/o node database construction goes like this:

One of the other reasons the databases are needed is to minimize the amount of core memory we need and release any and all the memory occupied by the things we have already done. This is important in large systems with a number of shares, some of which might be huge.

Otherwise, we may have the memory leaks which will eventually not only lock up the program, but may even bring down the entire O/S or JVM (Java Virtual Machine), just because we ate all the available memory.

(In particular, Syncthing/Pulse as of version 0.10.5 suffers from the memory leaks, it seems, and the program has to be restarted every few hours).

Initial index and database creation

One of the tricky aspects of update logic engine is the initial index and database creation. The complication primarily arises in case where there are no master (r/w) nodes on line while some new node initially joins the share. It is further complicated by the fact that the new node may actually contain most if not nearly all the data from the existing share at the moment when it joins some share. For example, if, for whatever reason, the database was damaged, there might be a need to remove some share from the sync and recreate it again in order to recover the correct state of the database. But all the physical data is already on that damaged node. So, in case of very large collections, rebuilding the entire index and a database may imply re-downloading the entire collection, which is the last thing in the world anybody would be interested in.

So, this means that even if you need to rebuild the index, you need to be able to determine that you do not actually need to download the files that are already there.

This means that you should be able to join some share and reconstruct the index and database in the most efficient way.

But the question is: what if you create some share in some folder that already has some files in it and what do you do with those files as far as adding them to the database? Those files may be a part of the collection as it is known to all other normally functioning nodes, or it may have some extra files on the disk that are not a part of this collection. What do you do about those files and their index?

Well, logically speaking, you do not want your database to contain any files that are not a part of the collection. Otherwise, you might incur some overhead in processing and may create logical inconsistencies across the nodes, where one node thinks this and the other node thinks that, which is an invitation to eventual disaster.

In order to solve this problem we can adopt the following strategy:

It should be noted that we should be able to build the database and the index for new node even in absence of any master nodes on line. This means that the r/o nodes MUST only contain the files in their database that are the part of the collection and are guaranteed to be "good" (the exact copies of the master node versions). Otherwise, if we are the r/o node, we will not be able to start downloading the share if there are no other master nodes on line, which is a disaster and is totally unacceptable.

Share/Collection hash and a key

There are two separate notions related to each share/folder/collection. One is a folder hash and the other is folder key (which is also a hash, but not uniquely derivable from the folder hash).

Folder hash is used exclusively for node discovery and can not be used to gain access to the share/collection.

Folder key is the key that is used to actually be able to access/join some collection and download it, at the very least. Key can not be used for the purpose of node discovery, just as folder hash can not be used to gain access to the share.

So, folder hashes may be publicly distributed without concern that some undesirable party may gain access to the share/folder.

But folder keys give one the ability to access the data for the share.

Selective sync

For now, this is just a stub with the most basic idea. It needs more thought and work, but it is something to think about indeed.

Selective sync means that you can selectively sync only those files that you are interested in. It is particularly applicable to the situations where some web pages or even the entire web sites may be blocked, (in particular, by the powers of evil to explicitly prevent the distribution of truthful information about the evil "ruling this planet" and its nature, tricks, tool and methods it utilizes).

Many people might not be aware that what we have at this junction is a global information war raging on as we speak. Quite a few very important documents and even the entire sites have been blocked by now and it is only going to get worse and worse, beyond any doubt. It is a huge subject to cover and we are not going to do it here. We have done it in other places and some of it elsewhere in this document.

But, what we would like to discuss is the mechanism to access only the selected web pages on some site as one of the most typical examples of the selective sync mechanisms. Or, we might wish to view some video or listen to some audio files even without downloading the entire collection. Actually, there are quite a few cases for selective sync.

Assume there is a great site somewhere, but it was blocked, at least for some potential visitors either in their country or by their ISP or globally for that matter. So, that site becomes inaccessible and the information is lost for all practical purposes. Because you can not access it by going to that site directly.

But the information is not really lost. It is still available at the source, be it someone's home computer or elsewhere. So, what can be done in this case in order to be able to access the information from the blocked site is to distribute the sync key to the collection. In that case, the site becomes accessible again on P2P basis, directly from some sync node(s). Except the potential viewers might not wish to download the entire site, but would simply like to look at some selected web pages.

In order to do that, you need to provide the ability to selectively sync only some files out of the entire collection. In that case, when you join some selective sync share, first of all, it automatically downloads the main TOC file for the collection. It may also download the common necessary files and/or subdirectories, such as common CSS files and images or other things. Then, once those files have been downloaded, you may open the main TOC.html file, which we call The Navigator and it will show you the list of all the web pages in this collection. Then, you might simply click on any of those links and the program will selectively download only the files necessary to view that page.

There are variations on this theme and different ways to implement this functionality. For example, there is MeshedSites and the XTTP plugin that is meant to work with BTSync as it stands right now. We tried to install it but for some reason were unable to get it working properly and we do not have much time to spend on it at this time. Basically, everything seems to be working, the XTTP plugin installs and when you click on some link for some site it is indeed added to the BTSync share collection. But no other nodes could be seen and, therefore, no download starts. Who knows, it could be because there are actually no nodes present, at least when we tried it, which seems to be strange. But here it is:

MeshedSites
https://www.meshedsites.com/

Basically, once you enter a key for the selective sync folder, there may be a checkbox in the edit box for this functionality. If that checkbox is checked, then sync does not start automatically except for the common files, such as CSS, images or anything else.

You may provide the text file with well known name which contains the list of all the files that are automatically synced.

But if you selectively sync only some HTML, PHP or some other file, then it may contain the references to other files that are required to properly display that page. That creates quite an issue and will require parsing the page to extract the links to other required files and so on, which opens up a Pandora box. A compromise might be to download any subfolders in the directory where this file is under assumption that you have some common subdirectories for images, css and so on. But there is no straightforward and reliable solution to this set of issues.

But, at least static HTML pages should at least display, which may be sufficient in case of some important information where the page formatting is not as significant of an issue compared to contents.

Unfortunately, for dynamic web pages this isn't going to work, at least to some extent or functionality.

But, in the worst case you may simply sync the whole collection or at least select which subdirectories you would like to sync. All of this needs more thought and analysis, but it might be something to at least think about.

Uncensored chat and blogs on P2P basis

Some of quite interesting applications of sync are chats and blogs on P2P basis. For one thing, while working with BTSync we have observed all sorts of problems with other nodes and in some cases they would sit on some share/folder/collection but things were broken on their end, partly because of all sorts of bugs in BTSync that would put them out of sync forever, and they did not even realize it, sitting on some share that would not update for them from then on.

The sorry thing about it is that you can not even tell them about their problem and can not advise them how to fix the situation. Because there is no chat facility available even though sync application could handle things like chats and blogs easily.

There are some attempts to implement the chat facility, like, for example, Vole, which works with BTSync underneath, and there is even an effort on the part of BT to provide a P2P based chat facility as a separate program. But why does it have to be a separate program when you already have the sync mechanisms in place with BTSync?

So, we can say quite a few things about desirability of chat facility combined with the sync main program in a seamless manner, and there are different ways it can be implemented.

But what we can say in summary is that - yes, such a facility is highly desirable, even for the simple practical tasks of communicating between the nodes possibly telling them about the problems they have and communicating all sorts of other useful information.

One thing to be kept in mind is not to worry about full-fledged web-like behavior as far as dynamics of the page update. Even if the display is done in the most primitive way, without any text formatting or styling, it is still quite acceptable, at least at the initial phases of implementation. Because the value of exchanging even the simplest kind of text messages is already quite something.

We've been experimenting with chat in the context of straight BTSync program by simply creating a separate folder/share for the chat. Then we would create a standard HTML page for chat messages. It even has any kind of standard HTML formatting and styling available. It all works fine and syncs between the participants within seconds. But its major limitation is that you need to edit that web page in your text or HTML editor and try to find the right place to put your message in and type all the things besides text itself in order for it to look good. But this is way too primitive to be called the real chat.

In the most primitive way, you can create the simple text pages and drop them into that chat folder and they will sync with all the participants. Except they'd have to periodically look into that folder to see if there are any new text files available. But it does work and we were able to help a few people with their issues by merely editing their text file and providing some answer or advice.

All we can say is that such a facility is highly desirable and quite useful. But what is the best way or a mechanism to do it remains to be seen and investigated.

Preliminary Update logic engine design and strategy

Here, we will present some preliminary ideas on design, architecture and key strategies for the engine. Following posts were made on the Pulse (previously known as Syncthing) forums:

Update logic engine design overview

Forum thread: How is Pulse different to BitTorrent Sync?
https://pulse-forum.ind.ie/t/how-is-pulse-different-to-bittorrent-sync/1083/15?username_filters=alberto101

Note: I was hesitant to reply to this, but I'd like to investigate some issues in the context of some info that we are putting out about sync as such.

calmh said:

[Note: calmh is the main developer of Syncthing (now Pulse).]

Syncthing would be a bad choice today for public distribution due to a bunch of protocol decisions.

Well, if you mean the network protocols, such as UDP or TCP, I can tell you that BTSync, for example, can perfectly work with both.

I am not sure what you imply by "protocol decisions" and I can comprehend that there may be a bunch of places in the code that need to be looked at, at least.

calmh said:

Mainly, since the sync is entirely bidirectional both sides need to know what the other side has.

Sorry, I do not see that.

1) One node does not have to know what the other node has in its totality.

2) About ALL it has to know, at least after the initial handshake is whether their main folder/share hashes are the same. If they are the same, nothing to talk about to this particular node.

3) If they are different, then one node may request the subfolder hashes (of all the subfolders in the main folder/share). It would have to also request the individual file hashes of the top folder.

4) When it receives the subfolder hashes, it may "realize" that there is a NEW folder. In that case, it requests the individual file hashes of that folder and schedules the requests for those files to be delivered.

5) If top folder hash is different, that means something has changed in the share, which, in turn, means that even if all the folders are present, there must be at least one folder with a different hash. When it finds that folder, it requests its file hashes as in 4).

6) If it finds that it does not have one of the hashes present, that means that file is absent, and it requests that file to be delivered.

7) Otherwise, if ALL the file hashes are present, at least ONE of them must differ, which means the file was updated, in which case it requests its delivery as in 6).

8) All of this means that when I come on line I do not have to know the complete state of any other node. The process is gradual discovery and is cumulative. This means that when I see some node, I do not have to have to know anything about it initially, except of the top folder hash, and ONLY if that hash differs, or is utterly absent, further decisions will be required on my end.

9) This means that when I come on line I do not request the other nodes entire database or index. This is, first of all, too much of unnecessary work and a waste of resources for most cases. Because most of the nodes will normally be in a state of total sync, provided no funk done manually and not gross configuration errors exists on those nodes.

10) This means that I only deal with nodes that differ from me, assuming they are synced to at least some degree. Otherwise, I am a supplier to them and it is their "problem" to request whatever they wish to know about this share/folder/collection.

Do you see any problems with this approach?

calmh said:

This means if you use a central point for distribution,

WHO says I have to have some "central point for distribution" and what is it needed for in a P2P framework?

11) Sure, if you imply the REFERENCE information, such as either the reference node or a master (r/w) node, then yes, my primary choice to get the most reliable picture of the entire share is to get the information from the r/w nodes. But it does not imply that I have to do the actual transfer from them. About ALL I need from them is the HASHES for every folder and, possibly, every file in a share.

12) So, the process is incremental discovery of the modified data. The very fact that I have obtained the folders hashes from the master node does not imply that I MUST get the data from the same node(s). From then on, I may refer to ANY node I know of and see if his subfolder hash is the same as on the master node, and if so, that node is as reliable of a supplier to me as the master node itself.

calmh said:

it needs to know and care what the potential millions of clients have.

13) First of all, millions of "potential" clients is just a fiction in my mind. The busiest torrents I have seen in my life had about 300 nodes. Out of those, the biggest number of active nodes was less than 30 at any given time, which is entirely manageable even using the TCP protocol, not even mentioning UDP.

14) Since I am not obliged to know about the other nodes and their state beforehand, ie, before actually contacting them for whatever reason, I, therefore, care ONLY about the nodes that are on line at this time.

15) Furthermore, I don't have to even know any details about any other node if I request (via UDP, for example) a top folder hash and it happens to be exactly the same as mine. That means we are in sync and I don't have to know even a thing more about them.

16) So, we are in practice dealing with no more than 30 nodes and with those, yes, we do have something to talk about.

calmh said:

Keeping all that state, not to mention an open TCP connection to each of them, would be prohibitive. In short, it's not in the design from the start, and now it fits badly.

17) I am not sure I follow what you mean by "Keeping all that state"? WHO "keeps" what state and where?

Edit: First of all, you don't have to keep the TCP connections permanently open to all the nodes you know of. For what? Once the "job" is done with them, you can just close it. The overhead in opening a TCP connection compared to the total amount of work to be done doing actual transfers, and that is probably the only time you need to have the TCP connection to begin with, is so small, that there is not much to even talk about.

Secondly, the only open connections you need are with those nodes with whom you have some real business to do, and that is transfers. All other work can be done via UDP, need be, or even the temporary TCP connections, which might incur some unnecessary overhead.

You do NOT open any TCP connections, and especially the permanent ones, to ANY nodes you have no active business to do, that is transfers. Simple as that. For what do you need to keep those connections open? What does it buy you? At best - efficiency. But even there, you need to look at the overall load and the amount of work to be done.

You can not just state, point blank: no, I do not want to use the TCP because of its connection handshake overhead.

But the TRANSFER overhead with TCP connections is much lower than with UDP, and it provides you guaranteed delivery and verification mechanisms. So, when you start comparing things on this level, you can't loose the bigger picture from your sight. Well, actually, this can be argued, because the amount of additional traffic on the TCP connection going back and forth is theoretically higher and there are some schemes to wrap the TCP in the UDP, and precisely for this reason. So... Arguable indeed.

18) Well, it is entirely understandable that you had a specific idea in your mind when you have started, and now it might even look frightening to even start thinking about a much more flexible architecture, becase you know your code better than anybody else and may intuitively have a sense "well, this looks like too much work". But I thought you are not exactly the kind of guy who is easy to frighten, at least with purely technical issues.

calmh said:

So, in the end syncthing, or Pulse, or whatever is open source. As you know I don't think this fits really well into syncthing, and it's not a feature I need so *I'm* not going to develop it. But I'll be more than happy to be proven wrong; patches with useful features that don't screw up other things are always welcome, and it wouldn't be the first time syncthing does something I was sceptical to from the beginning. ;)

Well, what can I say? I can not force you into something you are not excited about. First of all, I have no such a right to violate your space or impose anything on you. About ALL I can do is to present a different point of view. If it makes you feel better and it looks like it opens some doors in your perception and makes it interesting and exciting to consider something else.

Meanwhile, just by writing this post, I have gone through some things that are of interest to me. We'll have to digest it better and chew on it more and any feedback anyone wishes to provide would be interesting to see.

Now, you are not obliged to follow further on this post and I do not even have to remind you of that.

Good luck.

So far, "the force beyond visible" is with you, at least as far as I can sense at this junction.

Node discovery and trackers

One of the important issues is node discovery. For that, you are welcome to use quite efficient DHT network approach, which allows for distributed storage of node discovery information across all the nodes of the network. So, this way, you do not carry ALL the DHT traffic on every node and do not need to store the entire DHT tables on your box. You just carry the traffic and a part of the DHT tables of the nodes "closest" to a particular hash you are requesting.

And so is the issue with trackers. Technically, you do not have to provide some centralized, prewired tracker. You MAY, but it is not a predefined requirement. The trackers need to be configurable on per installation basis. Sure, you may provide a default and "guaranteed to work" tracker with a release. But even that one may stop working for whatever reason, and could even permanently go off line in the future.

I also like the approach of any node behaving like a tracker and upon ping would tell the requesting node about all the nodes it knows about for a given share. There could be even two different pings - global - per node, and per share only. You may even decide to utterly prohibit the per node global queries, purely for security and privacy purposes, again, configurable on per node/installation basis. But this needs more thought, especially in the private network applications, where you know for fact that your data for some share(s) is strictly private and confidential and you may not even let the other arbitrary nodes to query some information about those particular shares.

I also like the idea of keeping the node history. Meaning, once you see some node on line, it goes into your history file. So, even if that node goes off line for any length of time, you can still have its information. So, in cases of static IP/domain:port situations, you can periodically query the nodes you saw before to see if they respond. This is a fallback strategy to node discovery.

Basically, I am of the opinion that node discovery needs to be fully decentralized, or at least give you a configurable option to do so. Nodes and networks may have their own trackers or they may share some of them and should not, at least in most cases, rely on some fixed central tracker.

But when you see that the TCP connection may be more beneficial for some specific operation(s), then why not use the TCP? The TCP connection does not HAVE to be permanent to any node, if you find for any reason that you would not like to keep a huge number of connections simultaneously. But I think on modern hardware, you can easily utilize at least 50 simultaneous connections without winking an eye. The additional TCP handshake and connection overhead is not that significant once you start a transfer session. It is only inefficient in cases when you have burps of information, such as various queries that do not necessarily cause any further communication with that node.

Furthermore, BTSync can fall back on alternative protocol. This could become important for example if the company/organization network blocks some protocols for whatever reason. I recall some such discussions on BTSync forums.

Basically, you don't have to get stuck with specific protocol, and even there, for the most part it is just an efficiency issue and not something fundamentally important.

So, all these broadcasts, handshakes and all sorts of bells and whistles can be implemented in variety of ways. But fundamental things, like higher level update logic engine can not and remain the same, regardless of anything. Because they are designed like a tank to handle the craziest conditions imaginable and guarantee the data consistency across all the nodes.

Hope this helps.

Assuring the valid copies of the files in a share/collection

This is a pretty critical issue in public applications. It has to do with validity and integrity of files across all the nodes and making sure the correct version of the files is distributed.

Probably the main problem with public applications is that there is no guarantee that some node will not do some things that are damaging to the entire share/collection. They may not realize what they are doing and do some things the effects of which will be propagated to all other nodes, such as deleting or modifying their version of the file, adding new files and so on.

Or, some share may be deliberately attacked, either in order to create chaos on it or to destroy it entirely. We are seeing this with BTSync every day, literally.

Therefore, there is a need to assure that only the correct copies of the files are distributed. The most "correct" copy is the copy by the original supplier of information, which we call the reference node. Then we have a concept of a master node, which means a node that is guaranteed to have the exact copy of ALL the files on the reference node and has the r/w permissions and is allowed to update the other nodes with newer versions of the file(s).

Beyond that, there exists no logical way to assure that the information is valid.

So, before you can accept any files from the r/o nodes, you MUST be sure that these are not merely fakes.

So, what do you do in that case? Well, you know for fact that the original supplier of information is the only reference version of it. The 2nd in validity would be the master (r/w) nodes. Beyond that, nothing can be guaranteed.

But what if there are no reference or master nodes on line at the time you wish to transfer some files? Well, then you need to assure that any time the original version of the file is being transferred, you also carry a validity flag with it, which indicates that this version can be trusted as valid. But that flag can not be carried separately from the file hash. Otherwise, since it is an open source program, anyone may change how things work and set any flags they wish to any value they want.

That means that when you transfer a file from the reference node to other nodes, including the master nodes, you also include the validity flag encoded in the very key. That means that when you are calculating the file hash, you also prepend or append the validity flag as a part of the file itself, and so the hash is calculated accounting for the flag. The hash itself includes the value of the flag. So, when that key is decoded, the value of that flag can be trusted as not the faked one. This means that even the r/o nodes will carry and transfer the valid file versions ONLY. There is simply no way to transfer the incorrect versions of the files in this design.

Furthermore, with the "hard core" logic, there is no way to transfer the files unless they are guaranteed to be the exact copies of the originals. So, even on the r/o nodes, once the file hash is decoded, the value of the flag may be tested to see if this is just a fake copy that might have come from some other r/o node that modified his own version of the file(s) for whatever purposes, intentionally or not does not really matter from the standpoint of validity of data.

Decentralized node discovery

Here's one article that addresses the node discovery via UDP broadcasts in the context of multiple network interfaces:

I have a program that uses UDP broadcast to compile a list of available servers on the LAN... it's pretty straightforward: the user clicks the "Query Available Servers" button on the client, the client sends out a single broadcast UDP packet on a well-known port, and any servers that receive the UDP packet reply with a (unicast) UDP packet back to the client's IP address (as reported by recvfrom()).

When the client receives a UDP packet from a given IP address, it adds that IP address to its list of available servers in the GUI.

UDP broadcasts vs multiple interfaces?
http://developerweb.net/viewtopic.php?id=5085

The reply to the initial UDP broadcast needs to contain the currently active listening port. So, you send out the broadcast packet on a well known port and all the available and willing nodes reply to it with their listening port. Then, after the followup handshake on P2P basic, the requesting node AND the node that have responded are said to have established the communication and may add the other side to their "nodes seen" lists.

Dynamic configuration and node discovery options

jamesgecko said:

This is incorrect. BitTorrent Sync has LAN broadcast. BTS also allows you to disable tracker use for all your shares or on a per-share basis.

Yes, and I think it is a good idea. As far as I can see, all these things need to be configurable on the user level, right from the GUI, and it would be desirable if switching at least some options or choices of configuration could be done without the restart. Because the restart operation causes a "boot" process and you have to restart all sorts of things and make essentially duplicate and unnecessary communication exchanges that might not even be relevant to the parameter or option you have changed.

The approach I like better is that the configs are updated as soon as you close the dialog either with the OK choice or close button. They are not saved ONLY if you push the Cancel button. So, the program parameters may be partially reinitialized, but not necessarily all of them, and not necessarily the whole program needs to be reinitialized, at least in most cases.

Transfer and higher level logic engines

These are two of the most important subcomponents of the system. The "good news" is that the transfer engine code may utilize a well known and well tested existing code, which is available in the open source. Let us look at something known to be well tested by time, such as libutp library. Again, the idea here is that there is no need to "reinvent the wheel".

The UTP protocol was introduced in uTorrent and since then has been implemented in numerous torrent clients. It works like a champ and it is a high efficiency and high performance protocol that can handle any realistic transfer speeds without problems, which implies that it is highly efficient.

The libutp library is one of the most important parts of the system if developers decide to go the route of utilizing the underlying torrent based transfer mechanism, which would provide the bulk of the engine code to the entire sync application.

There is a compelling reason to consider a torrent based approach because, first of all, there are open source versions available and secondly, and could be primary, it provides the bulk of the code for the entire program. All you need on the top of the torrent transfer engine is hooks that dynamically construct the torrents for one large or many small files in a single torrent and simply pass them to the torrent based transfer engine. So, the amount of development work reduces to bare bone minimum.

The basis idea is simple. Let us consider a model which includes the database of all the files in a share/top folder/repo/collection. Once some share is created, the program then goes through all the files in all the subfolders of the main folder and computes a simple hash for each file and stores it in the database.

Besides the file hash, you also need to carry some state information as to the state of the file, such as we describe in BTSync related transfer table logic. Specifically, once you refer to some file, you need to know whether that file physically existed in a file system the last time your referred to it. If what you see in the file system is not the same as you see in the database for that file that means that this file was either modified or deleted or restored after being deleted or renamed. These are the MAJOR states you need to carry in the database for any file.

So, once some share is created, the first step in this model would be to go through each file in the file system for this folder and calculate the file hash. Then you need to set the file state bits as "physically present in the file system", "propagatable", "updatable" (from other nodes). We have considered it in relation to BTSync logic problems here:

Current file update and propagation logic problems

Now, once you have gone through each file and indexed it, you can start the actual transfers if there are any other nodes present on this share. Actually, you can start transfers even before you finish indexing the entire share once you know the file hash and the state for a given file. But that is certainly something desirable, but not a necessary requirement. It translates into simple inability to start the actual transfers until you completely finish the indexing, which is nothing more that slight inconvenience.

So, once you index the share, then you can start constructing the torrents, in case you are the supplier of information, and tell the other nodes that you have "the latest and greatest" version of some/all the files.

Actually, the process is a bit more intricate as you can initiate transfers either by announcing, if, for example, you have detected a local modification of some file. Or it may be initiated by requesting the information about any given file from the other nodes on a periodic basis. So, the process of initiation of transfers needs to be considered as a separate issue and in quite a detail since the results may be slower than expected frequency of updates. For example, it is higly desirable to start updating other nodes as soon as the O/S level file system events are detected, such as file update or removal.

Torrent construction

For efficiency reasons, you may construct a single torrent for many files if those files are too small and would unnecessarily create too much of a network traffic activity for not much of a benefit. So, you may decide that "I am not going to start the transfers if I have more than one file to transfer and its size is less than say, 1 MB". Why would you want to create the unnecessary network traffic if you have tens of very small files that could be packed into a single torrent? Because a torrent may carry ANY number of files without any problems.

UTP transfer protocol

And now let us look at the UTP transfer protocol, which is available as open source.

Since UTP is a well established and tested with time high efficiency protocol, this definitely deserves a pretty serious look at. Again, this could provide to be the major part of the transfer engine, aside from torrent construction and update higher level logic.

The advantages of using the torrent based engine are multiple. First of all, there is no need to "invent the wheel". Basically, this would provide the core components of the transfer engine. You just need a higher level sync engine logic on the top of the torrent engine and "you're done", almost... Well, not exactly, because you also need at least GUI, which, if written in Java, would provide an excellent, very nice looking and as powerful UI as you can imagine, considering that it would be portable across the multitude of platforms.

It is our opinion at this time that it would be highly desirable to create a well defined sync logic engine which can communicate to all the major blocks or subsystems of the rest of the program via well designed programming interface. Basically, probably the most portable and clean interface between subsystems would be the use of the interfaces. This would clearly and cleanly separate the subsystems and reduce the poking of one subsystem into another to the utmost minimum. There should be no peeking and poking across the modules outside of the interface mechanisms.

The sync logic engine is the highest level logic of any sync operation, and that logic could get extremely complex in a hurry. Because there are myriads of cases that need to be well designed and handled.

This is one of the major problems with BTSync. They simply do not seem to have even a notion of a high level sync logic. As a result, the behavior of the program is highly inconsistent, unpredictable, and many logical cases are not handled properly which results in inconsistencies of data across the nodes and in respect to the master or reference nodes.

The sync logic table needs to account for any modifications, updates, deletions, restorations of files imaginable. Unless every cell of this table is filled with the exact, precise and consistent behavior, you are bound to end up with database inconsistency issues, which means disaster as far as data goes. Unless you assure and GUARANTEE the consistency of all the files in a share/folder/collection and across ALL the nodes, regardless of the fact that some of them may be on or off line at any given time, and for any conceivable length of time, your system will eventually and inevitably degrade as far as data consistency goes.

Some nodes will not be updated, or updated with incorrect version of the file, or they may even update the other nodes with older version of the file, which might look "newer" to the system if, for example, someone has decided to attack the share by modifying the files with any kind of garbage information and then setting the file update times to the later time than the original or reference versions of the files, in which case the garbled files might be, and are likely to be distributed to all other r/o nodes since these garbage files will have the exactly the same name but will look "newer" to the system. This can create havoc across the nodes. This case certainly has to be considered. Otherwise, any user can obliterate the entire share if you, for example, do not refer to the master or reference nodes to see it any file is actually a valid file as available on the master node(s).

And we had a chance to observe such a bizarre and highly damaging behavior with BTSync in too many cases to even mention here. Basically, we can observe all sorts of damages to the data or incorrect reporting in BTSync pretty much all the time. In fact, we can not point to a single share that was not damaged in one way or another and/or its reporting being partially or totally incorrect, and in the most profound ways. This is just to accentuate the overall significance of this issue.

There might be enough reasons to have not only a notion of a master node, but the reference node as well. The difference between the master nodes and the reference node is that the reference node is the ORIGINAL supplier of the information, and, if ANYTHING whatsoever does not reconcile between any nodes, then the only guarantee of an authentic version could be obtained from the reference node. The master node is a node that is guaranteed to have the same exact copy of a share, (including every single file), except it is not the original supplier.

Yes, there exists a tricky part of the notion of the reference node in cases where you have several master (r/w) nodes, each of which might also be an original supplier of some information. But, on per file basis, there could be only one and one only original supplier. So, in the database you may have some files where one node is the original supplier, but the other files were originally supplied by some other node. But this seems to be solvable issue, if, for example, we mark the reference node status on per file and not per node basis.

Why this difference between the reference and master nodes? The difference is that in case the reference node is off line, how do you determine that your share is not merely attacked with garbage files, or being damaged inadvertently? Any r/o node might become corrupted at some point, intentionally or not, and it may look like it has the "latest and greatest" version of some files. For example, if user touches some or all of the files with his current clock time, then all his files become "newer" and would have to be resynced.

The same happens if some node, discovering some error, decides to copy some files from the sync archive to the main folder, but he does not copy the files with preserve flag, in which case the new copy of the file will have an update time equal to the time when it was copied, which is now. Thus, the newly copied version will look like "the latest and greatest" version of the file despite the fact that the file is actually an old version.

Yes, if file hashes did not change than you do not actually transfer the files to the other nodes. All you have to do is to update their time stamps. Well, in this case, you can refer to any master node.

So, this is a little extra overhead related to consistency. Before you do any updates, you need to verify the file against the reference or master nodes. That might create some extra network traffic overhead. But it will dramatically increase the consistency across the shares.

There may be many master nodes, but there is only one reference node for a specific file at any given junction (again, the reference status is on per file bases, and not per node global).

But what if the reference node goes off line permanently and never comes back? Well, in that case you need to design a strategy of passing the reference node status to one and one only master node, and there are various ways of doing it. But let us skip it for now because these points are too fine at this junction.

Now, some files may never be updated if, for example, some user modified his local copy or deleted the files or restored the previously deleted file, after which they can not resync because the database "thinks" these files have been deleted and permanently marks them so, just like BTSync does it, which is probably one of its most critical design flaws as far as update logic goes.

There are many more pretty critical cases and issues to be considered and we will try to cover some of it if we find some spare time and we can see that this thing is going forward and is alive.

So, the sync engine higher level logic needs to be well designed and account for the most minute cases, "possible" or "impossible" - does not matter. In real life, those things that "should never happen" DO happen. Therefore, the sync logic must be as reliable as a tank and not miss or fail in any situation whatsoever. This is probably the core issue of a reliable sync system.

The MAIN rule in ANY system is STABILITY. There isn't anything more important in the system as stability, no matter how simple or complex that system is. Because if you can not guarantee the stability under ANY circumstances whatsoever, the system will crash, in one way or the other, or may perform in incorrect ways. And you can not possibly assure the stability if you can not guarantee the consistency of data across ANY nodes. Therefore, this rule is VITAL. Else the system will collapse or degrade, and inevitably so. It is just a matter of time...

So, the sync logic engine is the main subsystem that assures the consistency across all the nodes of any given share.

Finally, here's the code for libutp:

libutp
https://github.com/bittorrent/libutp/

Go language version:

https://github.com/andradeandrey/GoUTP/tree/master/libutp

About trackers

Any node should also have the ability to perform as a tracker. This means that any node which has been discovered by any other nodes by ANY means should pass the list of all the nodes it knows about for the particular share to the node sending a node discovery request.

For example, in BTSync, if you specify the "predefined host(s)" for some share, then those nodes will also send you the list of nodes they know about on this share, except a predefined host can behave like a tracker ONLY if it contains a physical copy of the data of the share, which is incorrect of a design.

(Actually, in BTSync ANY node behaves like a tracker in terms of passing the information about the nodes it knows about on this share.)

ANY predefined host should be able to behave like a tracker REGARDLESS of whether it has a physical copy of the data or not. You simply have to specify it in the configuration for the share. For example, in GUI edit box for this share you simply click on a check box "use as a tracker only", which means that this node does not have to have the actual data for the share, and yet, it can behave purely as a tracker node and pass the list of all the nodes it knows about to any requesting node, simple as that.

Furthermore, it should be able to maintain a list of previously seen nodes that are currently off line. Because now they are off line, but they have the static IP and the same port and may come on line ANY time. So, when they will come on line the next time around you should be able to get to them even if no other means of node discovery are available, such as DHT or "predefined hosts". Technically, it is no problem to remember ALL the nodes some node has seen on some share in the entire history. Because all you have to remember is the IP and the port, which means that you can remember hundreds of hosts without much problem.

About hashing

Well, hashing could be effectively used for identification of information purposes. For example, the base-32 SHA1 cryptographic signature is "good enough for the poor man like me" identification purposes.

You can identify even large amounts of information by simply calculating a hash of the entire file, block or a very long message. Yes, the guarantee of uniqueness of the hashed information is not 100%, but the statistical chance if misidentifying it could be easily 1 to a million cases.

You may calculate the hash not only to the entire file, but to individual blocks of that file. This may allow you to identify the partially transferred but incomplete blocks within the file, resume the file transfers of very large files by updating only the blocks with incorrect hashes and so on. File hash may be calculated out by going through the list of all individual block hashes, which could be useful for performance purposes.

Furthermore, to increase the efficiency of the entire transfer logic engine you may compute the folder hashes as well. This may significantly increase the performance of updates and minimize the amount of network traffic in order to figure out which files need to be transferred to other nodes or which files need to be requested from them.

So, in order to see what has changed in a share/top folder/collection, you don't really need to get their entire database or index and then compare it to your own in bulk. What you can do instead is to calculate the top folder hash by calculating the individual subfolder hashes by going through all the files in them, calculating the file hashes, and then calculating the folder hashes out of those file hashes. Then you calculate the top folder hash out of all subfolder hashes.

Basically, you would still have to periodically rescan your collections even if you do handle the file system events, such as delete or update. Because some nodes may be off line and during the time of event they won't be able to get information and do the appropriate thing. So, no matter how automatic or "great" is your mechanism from the standpoint of efficiency or performance, there are still countless cases where some things or events are going to be missed. This means that you can calculate all your hashes by default during the periodic scans. And, if you properly handle the file system events and recalculate the individual hashes as soon as you see a need for it, then you might obtain the maximum efficiency, performance and reliability of any file related operation.

As a result, to see if you are in sync with some node, all you need to do is to request the top folder hash first. If that hash is the same as yours, you have no business to do with this node and you don't even need to communicate with it any longer, except during the periodic global scans.

But if top folder hashes are different, then you need to obtain the individual folder hashes either by going through all the subfolders you know and requesting the individual hashes for those, or by asking the list of subfolders available to the node you are communicating with. And so you descend deeper and deeper into the folder hierarchy in the process of discovering which files were changed, deleted, added or renamed.

So, hashes allow you to dramatically reduce the amount of CPU load and network activity in the process of discovery what has been changed in all the nodes you know about. That is the overall idea.

About logging

Logging is one of those often overlooked facilities to assist in all sorts of reporting, including the operation status, transfer completion, various warnings and errors.

Well thought out logging system is a tremendous help from nearly any angle you can look at it and, quite often, is more effective then debugging techniques.

Basic principles of logging are relatively simple and do not have to be something "fancy" or too sophisticated. There is no need to overcomplicate things that are basic in nature.

First of all, logging needs to be done to a file, and not just to the screen, and that file should be created automatically as a default, on, for example, per session or daily basis. This way, in case you find some damages or errors, you can go back in the logs, possibly even from the previous sessions, and find out exactly when and why this problem had occurred. But the logging needs to be sufficiently precise and give you full paths to the files and things like that. You might have some files having the same file name, but either located in different places or belonging to different shares. So you need to be able to distinguish between them.

Furthermore, there may be several logs. One of them could be more or less detailed description of operations and the results. It is worth noting that the these things need to be logged in as complete way, as possible, providing full paths of file names and full identifying details of the source and destination.

Also, status information about success or failure of some operation is very helpful, including as complete details, as possible, so there would be no doubt what exactly happened, duing which exact operation, to which exact files and so on.

Then there may be a warnings/errors log. If you utilize the convention of categorizing errors and warnings by the degree of severity, then even looking at very large logs represents not much of a problem in order to find out what exactly was the issue.

Each log message needs to be date/time stamped and start with some easily searchable and identifiable string, such as

The number of "*" or ">" characters at the beginning of a message would correspond to severity or significance of the message. So, to quickly find out the most critical problems with possibly huge transfers, all you'd have to do is to search on ">>>>> ERROR: " string in the log file.

About portability and User Interface

Syncing as such, by implication, requires the highest degree of portability because it is desirable to be able to sync all sorts of devices including the popular NAS and mobile devices. The bigger the variety of devices the program supports the broader its applications.

One noteworthy application is using the sync program on a headless box, such as virtual hosts or other servers. That might place a set of additional requirements on the programming language from the standpoint of supporting GUI-less operation, or, alternatively, the browser based GUI where program is run without the need for the native GUI support, but with support of some other interface to the external programs, such as browsers, using some well established notation or data exchange format, such as JSON, for example.

Basically, the program must be able to run in a headless configuration.

Extensible architecture

Also, providing the well defined interface to the key program functionality and mechanisms will allow to hang on the top of the basic engine and program a GUI that fits the user's taste and needs the best. "There are no friends in tastes and colors" they say.

So, the program core is the same, but you can have several GUI packages, including the native mode O/S dependent high performance GUI or alternative and "universal" browser based GUI that could be accessed remotely.

Another thing here is that the users may create their own plugins for the core of the program to perform whatever crazy things they wish to do. So, if the program core is fully exposed from the standpoint of any logically possible and discrete and self-sufficient operation, then the program becomes not only merely an open source, but open and extensible ARCHITECTURE, and that is really something, which is entirely possible and not that complicated if you design your interface to the external programs really well.

About development strategy

One of the complicating factors of development strategy is the fact that one needs to have both ends functioning, at least to some degree, in order to be able to proceed with development in effective and productive manner. Because both ends of the sync need to work, at least to some extent.

What is a further complication is that many operations need to be developed symmetrically. You can't have just one end of the wire work and the other one not being able to cooperate.

This means that the very strategy needs to be stable and steady. You can't add some significant amount of code and/or functionality if you can't have both ends functioning and cooperating.

So, the very approach would be slow and steady, without creating too much of a radical change at any given point. Well thought out logging mechanism would help tremendously in this process. Again, because it might not be even feasible to rely on break points and single stepping. Else you might need to single step on both ends, and those could be drastically different, such as your PC based system and a headless server.

The main rule of reliability and stability

Well, this is one of the most common errors made by the programmers. They keep adding new things and all sorts of bells and whistles but never quite test all the conceivable and even "impossible" situations. So, what happens as a result is that they are being hit by those "impossible" situations nearly on unending and constant basis. But, instead of stopping and fixing the problem right then and there, once they see it, they just keep ignoring it because they are "busy" with other "great" things and features.

And, what is even more significant, is that at some point, if you do not fix things ASAP, as you first noticed them, then the amount of problems will simply accumulate. At some point, you might simply get overwhelmed by the innumerable number of problems, just as we see with BTSync, for example. It is amazing to see their main tech support guy on forums hasn't gone insane, dealing with piles upon piles of problems many of which are not even easy to replicate in a test lab setting.

Because once you get overwhelmed with problems, the whole project begins to give in and degrade, if not collapse. If that happens, any and all development has to be stopped entirely and every single effort and resource to be allocated to fixing the bugs and in a prioritized manner from the standpoint of stability, reliability, predictability and assurance of the consistency of data across all the nodes.

Otherwise, the end result in most cases is that the program works and/or performs well only under ideal conditions. But the "real life" has NOTHING to do with some "ideal conditions". The most unimaginable things happen ALL the time, whether you like it or not.

So, the "moral of this story" is quite simple: as soon as you see some bug or some malfunctioning or anything that represents ANY kind of problem, FIX it, right then and there, and you will never have to regret doing it. Because at any given junction your program will be stable and reliable and will work under any "crazy" conditions imaginable, which will only make you feel good, instead of forever feeling uncomfortable constantly seeing these seemingly small and insignificant things which you chose to "ignore for now" in the name of some "greater good" which is one of the grandest illusions there are.

Related useful open source repositories

The following repositories might contain the bulk of the necessary code:

Note: if you use the Windows version of github then very large repositories might not get cloned, no matter how many times you try. So, if you have access to Linux system, install git package on it and then run:

git clone https://github.com/repo_name

That should complete fine. If not, install the screen package, create a screen session via screen -S session_name. Then run git command from it, and in case you get the the puTTY disconnect or do not see the shell prompt for a long time, just duplicate the session on putty and do: screen -d -r session_name. That should show you that git completed and you'll see the shell prompt.

Then just zip that entire repository folder, FTP it to you Windows box and unzip it into the same exact directory. Then start the git for windows, and simply drag that depository from the web page, so it starts adding it. Now, since you have it all already, it will simply recognize that it already exists.

But, since your dowload was fresher than the repo, then it will look to git that you have the updates for all the files. Just deselect the "Files to commit" checkbox so it unselects all the files, then right click on the same column where you saw that message and click on "Undo all changes" choice. That should put you back in full sync with that repo.

Basically, github for windows is a bit flaky, so you may have to play with it...

Other synchronization systems:

BTSync 1.4.72 disaster warning

Warning: Think well before you decide to install or upgrade to BTSync 1.4.72 version or higher, and especially if you had a previous version installed, such as 1.3.109, which is the latest more or less working version.

And if you do decide to upgrade the version, make sure to back up your entire btsync configuration directory (C:\Users\username\AppData\Roaming\BitTorrent Sync directory on Windows). Specifically, make sure to back up the settings.dat and sync.dat files in that directory. Also, make a backup of all the database files (all the files with long names in capital letters and digits, *.db, *.db-shm, *.db-wal.

Because it 1.4.72 is going to override your previous installation and if you decide to go back to it you won't be able to.

Quite a few users wanted to go back to 1.3.109 which is easier said than done. We have spent two full days trying to recover and are still not recovered fully. It is hard to describe by anything other than a nightmare.

There are several very significant issues with 1.4.72. Probably the main one is a new GUI, which is IE (Internet Explorer) based, and is probably the most primitive UI you have ever seen in any stand-alone program. Because IE was not designed for creating a full fledged GUI to begin with, and, even if it can do some things that LOOK like something GUI-like, it is as far from the real GUI as you are from the planet Mars. We do not have time now to talk about it in specific details. Just a couple of points in passing:

  1. There is a compatibility issue of 1.4.72 with some previous versions (do not recall which ones). This means that if you have some nodes with older versions of BTSync they won't sync. This issue is a killer in a general public setting where you don't have access to each node directly and can not even tell the users how to solve the problem. It simply won't work and they won't know why.
  2. There is a compatibility issue with Windows XP/Vista/Server 2003 users.
  3. With version up to 1.3.109 your main operating tab would be Devices where you could see all your shares and all the users for each share on one screen. That is the most comfortable way to work with the program. With 1.4.72 you would have to move your mouse, select some nearly invisible dots to the right and then click on some share to see its users. Now, while you are watching it, you can not see what is going on with your other shares or other things.
  4. You can not resize the columns in a way that makes sense.
  5. You can not rearrange the columns.
  6. You can not copy any text from GUI to the clipboard which makes it impossible to copy some status and other messages, long strings, possibly written in foreign language, long hash strings with random letters you can not possibly remember and so on.
  7. There is too much white space wasted on the screen and so, if you try to make it smaller, you won't be able to see anything of real value left. The classical example of a well designed GUI and screen space utilization is the Windows O/S itself. Everything matters down to a single pixel and no pixels on the screen are simply wasted. That way, you can pack a lot of information in a relatively small window or a dialog on the screen and then resize it if you need to, just like you do with Windows Explorer or some other file browser.
  8. There is no File Transfer screen. This thing alone is nearly a disaster. Because when some transfers occur, and there could be several at the same time, in the previous versions you could see which exact files are being transferred and to which users, which is probably the easiest way to estimate the progress of things especially when it gets busy. With 1.4.72, you are virtually blind and can not see what is transferred and to whom and what is the progress of things.
  9. Dialogs are modal, meaning that once it is open, you can not do anything else with the program until you close it.
  10. There is no OK and Cancel buttons in the dialogs, which means that the values and parameters and settings are modified immediately, just as you type. So, if you make a mistake and close the dialog, it will have the wrong values left, because you can not cancel out, nor you can undo some possibly wrong values entered.
  11. Dialogs can not be resized. They just sit there in a fixed place and with fixed size, just like dummies, and you can not even move them out of the way if you need to look at other things on the screen. This is simply pathetic.
  12. Text contrast between active and inactive items is too little and makes them virtually indistinguishable. For people with poor eyesight or with poor screen lighting conditions you won't see any difference between active and inactive items.
  13. Generally, there is too much of a mouse move and click to see something useful, and even there, you still can not see one useful screen that pretty much gives you the whole picture of all your shares at the same time. You have to constantly move your mouse and click on some things that are not even easy enough no notice and open up yet another dialog box that does not necessarily tell you much. Again, pathetic is probably the best word to describe such a primitive UI. It is definitely the worst this writer has ever seen in any stand-alone program UI design, even on the amateur level.

BTSync 1.4.72 has a totally redesigned GUI and it runs under IE "under the hood" and users are not even informed and/or not aware of it, or may not even suspect it, but what they see on their screen is IE running. For some users it is a matter of principle NOT to run anything from the Microsoft beyond the O/S itself. We have asked: WHY are users forced to use some 3rd party software? - No answer beyond ridicule.

There was a court case in EU against Microsoft shipping IE as a part of a Windows as a default, thus slanting the opinion of people in favor of running IE instead of installing some independent browser. Microsoft lost that case. But with BTSync, we not only have the same exact issue, but users a FORCED to run IE and thus the "message" from BT is essentially "get used to it". What do you call this?

First of all, it is not even clear what are your security risks when you run some program via IE. Why would you want to have another questionable issue with such programs as BTSync where you already have more than enough of questionable security or privacy related issues and some even allege that BTSync works or cooperates with the NSA and all these "encryption" and "security" claims are as fake as three dollar bills, just because it is a closed source program?

And why do you have to run a bloatware web browser program just to run the BTSync? What is the "message" here? - You can not exist without Microsoft? Any place you look there is Microsoft with backdoors to the NSA?

Also, in 1.4.72 some things have changed as far as special files are concerned. In particular, the .SyncArchive directory, where all the previous versions of files are stored, is now located in .sync subdirectory of the main directory for the share. Also, .SyncIgnore and .SyncId files are no longer located in the root directory for the share, but are also moved to the .sync subdirectory and their names are different now. Also, the file format for all those files was changed.

There are other things in 1.4.72 that have a profound effect and need a detailed analysis and understanding. But we do not have time to cover it right now. So..

Perpetual beta excuse for not fixing critical errors

To make the long story short, all you have to do is to ask yourself a question:

What would I do with my car which is broken forever, to the end of times?

If you see an answer to this question, that is all you have to read from this chapter. But, in case you are still curious, here it goes:

One of the main excuses for not fixing piles of bugs is called "beta excuse". It allows one to do anything they want in terms of adding new bells and whistles instead of doing those things that MUST be done in beta stage - BUG FIXING. This means that when you see people complaining about tons of most critical bugs, you simply blame it on "beta", like beta is like a rubber condom that could be stretched to the size of a zeppelin. So, let us first define a beta release.

What is beta release?

Beta means the product is nearly ready for the final release and, for the most part, is as good as final, with minor exceptions. At beta stage, no development of ANY kind is allowed and no feature additions are possible. In the places like the Silicon Valley, which is recognized as world's capitol of software development, if product manager, chief architect or director of development allow ANY feature modification or addition, they are simply fired, as soon as VP learn about it, pretty much on the spot with "no ifs and buts".

Because it is an invitation to disaster that may lead to eventual bankruptcy of the entire company, not even talking about the company image and reputation, which will be tarnished forever. Software business is one of the toughest and unforgiving businesses to be in where people are fired for such seemingly innocent things as testers contacting developers directly, without going through their manager. Because it wastes the developer's time, and that time is about the most valuable time in the entire company.

And what we see on BTSync forums is ALPHA-tester monkeys going as far as insulting, attacking, ridiculing, mutilating the posts, killing and banning the most senior level developers of the architect level, and about the most disgusting aspect of it is that they were told who are they dealing with and had plenty of opportunities to verify things, just by looking at the posts of the author, for example. This is insane asylum level. And that's a BAD, BAD news for BT, and it is going to cost them probably more than they can swallow.

Beta is strictly bug fixes. Nothing else is allowed, under ANY circtumstances whatsoever. Else, you'll never have a final version of the product. Simple as that. You can not bring in NEW errors and problems into the product at beta stage. And ANY changes to the product features or behavior, as of necessity, will bring the NEW bugs and problems. And that is why BTSync is in this "perpetual beta" phase for a year now. Have any of you heard of a product in a beta stage for a YEAR?

At some point in alpha state, there is a feature freeze, which means that no new features could be added to the product from then on until the next MAJOR version. New features, as a rule, are not added to the MINOR versions. Only minor things targeted mainly at improving the product stability and, possibly, correcting some functionality issues that could be not strictly bugs, but, rather, the imperfections or deficiencies of behavior.

That means, that once the product exits the alpha phase, no feature additions and no radical redesign or change in product behavior is allowed.

As far as BTSync goes, such a major release as 1.4.72, which COMPLETELY changed the GUI and whole interface behavior, means that this product can not even be classified as alpha. Essentially, it is in PRE-alpha stage, because even in alpha, at some point there is a feature freeze.

This fact alone is enough to bring up a claim that BT is engaged in misrepresentation for the purpose of entrapment and cornering and monopolizing the sync market. And these are as bad of a business practices as it gets. And the issue of monopolizing the market is self-evident in the sheer fact that BT does not disclose the information necessary to make a compatible product and has no intentions to do so the way it looks so far. They claim it is a "free" product, but do not release the necessary information for other players to join the market and thus prevent the healthy competition and attempt to hold back the progress of technology for the benefit of mankind as such. And that is pure grade EVIL.

The end result of such business practices is that the other developers can not fix the piles upon piles of bugs, even if they have all the necessary competence. That means that BT INTENTIONALLY stands on the way of progress of one of the most important and critical technologies in today's information driven world. And there can be no bigger evil than this one as far as business practices and moral or ethical issues go. Because it is an indication of a PROFOUND dishonesty and an attempt at domination in the modern information world. BAD, BAD news for them.

Basically, BT is disgraced utterly and completely, and the level of trust to them is falling as rapidly, as it gets, which is evidenced in numerous posts in their own forums where people are nearly begging for someone to start developing an open source version of an equivalent product. And it does not have to be FULLY compatible to BTSync. There is simply no need or requirement for it. Sure, it would be great if there was a 100% compatible product, but it is not a must, by ANY means.

The support staff, with exception of one guy, usually provides some scratchy and technically incomplete or imprecise answers, at best, and if you question any of it, they usually pull their standard "excuse": "But it is still in beta", and even try to ridicule you and make you feel guilty of something, even though it is THEIR problem. And if you keep trying to clarify the issue, the procedure is: ridicule, insult and then delete, kill and ban. Simple as that.

They even invented a new term for this pathetic excuse - "perpetual beta". So, the thing can be broken forever with this excuse and you can not even expect anything other than their excuses, including "but this is a free software". So what? Linux also is free software. Does it mean it has to be as flaky as BTSync, which is probably the flakiest piece of software you have ever seen in your life?

Like it means beta implies broken or half-working something. Well, BTSync has been in this "beta excuse" trip for a year now, and even today to call it beta is misrepresentation and entrapment, essentially, and luring more and more unsuspecting users into using a half-broken program, with hidden motive to corner the sync market. That is probably the closest interpretation of it all.

Why entrapment? Because most people would be willing to try out the beta version in understanding that this is something nearly as good as the final version, and people would never even suspect that this isn't even close to beta. Even to call it alpha is misrepresentation. And not that many people would be willing to try the alpha version because it is nearly assumed that there are likely to be problems with it.

It creates this eternal excuse for not really doing a good job on fixing tons of things that are broken, and instead, they forever keep adding all sorts of useless bells and whistles to entrap more and more customers and users with all sorts of "new features", while under the hood even old features do not work correctly or in a logically consistent manner.

If you just look at BTSync forums, you will find such a huge number of problems that it is hard to believe someone can even release such a thing as a product. At the rate they are going, it is probably going to take at least a year or two before this thing becomes real beta.

With such a huge list of problems, and we are talking about some vital issues of top priority grade in any self-respecting software outfit, is not acceptable in any professional product, or even in some open source project. And we have mentioned some of the key problems months ago, but they have not moved a finger to fix most of them, even the most critical ones. And we are talking about things that make the whole thing not to work in some situations, as bad as that.

In particular, the so called timediff problem, that totally prevents some nodes from syncing, have been brought up and we were even able to determine the exact technical issues related to it, and even proposed a solution to this nasty problem. They said that they will see what they can do to help the users in the future release, and since then there has been about 10 releases, but the problem is still there, and we see it every single day with several nodes on different shares. Not a thing has changed.

Furthermore, this problem is so nasty that it can screw up the shares on the nodes so badly that they won't be able to sync at all, or their stuff will get all messed up. To someone who understands the issue and its significance it should be a top priority item, to be fixed before anything else. And we are seeing its effects in the nastiest possible ways right now and have been struggling to fix things for days, LITERALLY, and having no luck, no matter what we did.

It looks like a market department driven organization with extremely poor management and technical oversight. And when companies start running according to the marketing department ideas, those companies quite often end out of business. Because they never fix things to make their product stable and reliable, and instead, keep adding more and more complications and all sorts of bells and whistles, thus loading the technical departments with things that take away their resources from things that do need to be done, and quite often, urgently. But there is no time for it left. Because marketing is always in a hurry.

Furthermore, some of the most obvious programming errors one can see nearly everywhere in BTSync, going down to such a basic level as GUI design, simply indicate that the level of technical competence of their coders is on the level of a three months class in some community college - total incompetence.

For example, let us take probably the simplest case of item selection in a list of some dialog. If you select some item, such a device name in Device tab, or items in the Transfers or History tab and the list is updated, then the item you have selected is no longer selected. What is selected instead is the item with the same index position from the top of the list, which is not even laughable and a sign of total incompetence, but it is simply pathetic. Any programmer with a few months of education could do it correctly within minutes. This author has never seen any program that can not handle this case correctly. All it takes is to select the item not by its index in the list, but by the key. That is all it takes.

And this simplest thing in the world creates a nightmare when you try to see what is happening with some device and devices come and go all the time. It is nothing short of a madhouse to try to keep up with it. It is simply a circus level.

We could provide here a list of the most obvious programming errors that could be fixed in minutes by someone who knows what he is doing. We have mentioned some of those things in their forums months ago. But they simply ignored it like something so insignificant that it does not even deserve an note or a response on their part. And if we look at the issues of much more fundamental and critical level, it might look like they simply have no clue of the most basic "no-nos" in multi-threaded programming.

The results of it, just as reported by users on numerous threads, is that the program starts eating the file handles and memory. After a while, it simply chokes your comp and it grinds to a halt and the program may even crash or grind to a stop. Because some program threads deadlock because of improper use of the synchronization objects and violations of the most basic rules of synchronization. So, they simply abandon those threads and keep creating more and more threads and/or file handles. It is too technical to discuss here and there is really no need. What will it change?

When we brought some of those things on the BTSync forums and asked some questions on how to recover a totally broken installation after trying to run 1.4.72, there was only one semi-intelligent response from one of the users. But all we got from the BTSync related personnel, such as "alpha testers", was nothing short of an insult and ridicule and by no means a precise and technically correct answer.

Eventually, they have started to simply delete the messages without addressing the issues and trying to cover it all up and make it look like those messages are not "on topic" for that thread. And this is not the first time they are doing it. It seems like their favorite practice to handle the most touchy issues. We don't even know how many other messages from other people they have removed trying to hush things down.

We include just a couple of those messages below. The first one is all that was left of them. All other messages were simply deleted. Not a single answer with sufficiently precise explanation, description of anything or instructions was provided by BTSync staff. They even went as far as to insult the writer and make him look stupid by saying things like "well, I do not know how to make it simpler for you", while they have not answered the question or addressed the issue in a competent enough way.

And it happens that some "alpha tester monkey" was insulting the most senior level developer trying to make him look stupid, instead of providing the precise answer and technically competent instructions, throwing around the lunacy terms like "you have to remove the configs" and things like that. Well, WHAT constitutes the configs? There are configuration files, there are configurations in the Windows registry, there are databases, history files and so on. So WHAT EXACTLY do you have to remove? Which key in the registry? Which exact files and located where? What kind of answer is this?

And, mind you, we are dealing with fatal level issues. The entire program no longer works. NOTHING works. But it worked perfectly well just before the installation of 1.4.72! And THAT is how you treat your users, you funky monkeys with three months of programming class in some community college, at best, while talking to some of the best developers of the architect's level there are with equivalent education and experience level of a Phd.?

I have a question for you: Are you REALLY insane, or just pretending?

I did ask one of them a question: are you a sadist? But he did not have guts to reply. Because he certainly is. Just look at his/her favicon and try to figure out his/her gender and "preferences". It is not even clear if we are dealing merely with the transvestites or outright mutants and hybrids from the "dark side" directly. Looks like Martians are here already.

GreatMarko. Alpha tester of BTsync.

GreatMarko. Alpha tester of BTsync. Hey, what planet are you from?

Hello... from "the dark side"!!!

To psychologists, this is a typical picture of a suppressed degenerate sadist. Stalin has ordered a study of physical indications of degeneracy to be conducted by the top medical experts. They have identified about 40 typical indications of degeneracy expressed in birth defects. Several of those indications are clearly present in this picture. But this is a totally different subject. So, we'll skip it.


So, here we go with the last message that survived this butcher-mutant-sadist-wannabe something big. How many of other on the point messages has he/she deleted? How many people did he/she insult and how many accounts did he/she/it close? Guess...

Messages on BTSync forums about 1.4.72 upgrade disaster mutilated and eventually deleted by the "moderators" and alpha test monkeys

So, basically, what you see on BTSync forums is a highly censored and mutilated version of reality. Yes, once in a while they do allow some pointed questions or comments, probably to make it look like a balanced view on something. But we know for fact that there are so many most critical issues in BTSync and there are so few posts addressing them that this can not possibly happen in a world not filled with lunatics and complete zombies.

Basically, what this predetermined commitment to stay with IE, no matter who says what, means is that this product has no future. Even without this IE "improvement" the product has quite a list some of the most critical issues that do not seem to be even scheduled for resolution. It seems that as long as there are new bells and whistles offered every few days, that is all that counts, pathetic as it gets. The most important thing to be done seems like producing as much hype as possible.


Existing open source sync alternatives

Syncthing (Pulse)

But the good news is that there begin to appear similar products. For example, there is Syncthing which already works for private networks. That means that if you only want to use it to sync between several devices to which you have access, everything should work for you as good as BTSync. Syncthing is also quite portable and supports a variety of operating systems and devices even though some people complain about absence of support for some esoteric devices. You'd have to look at it yourself to determine if it supports your situation.

Unfortunately, they do not allow just any user to join the shares automatically. It all depends on manual approval by other nodes, which means that you have to sit there by the screen 24 hrs./day and watch the requests to join showing up. But from the standpoint of a new node that wants to join some share but can not until he is approved, the whole thing might look like "well, this thing isn't working as advertised" to the new users and they are likely to abandon the whole share or even the program. After all, how many hours or days would you be willing to sit and wait and see nothing happening?

For general public applications, where you use the program to distribute a collection of information, joining the share or repository should be fully automatic, just like with torrents. Once you have the torrent file, you can start downloading stuff immediately assuming there are other nodes present on that torrent at the moment.

That is the MAJOR problem with Syncthing, even though there are other desirable things to be implemented, but none of them are critically important for the most part.

So, the program seems to be reasonably well suited for the private networks, but not for the general public use. But those guys have begun to discuss the easier ways to join the shares and the main developer has promised that version 1.0 will have that feature working. We'll see.

Links to their forums and release page.

Syncthing/Pulse Releases

And here is support forums:

Documentation

Support

Development

Using Syncthing for public distribution of information

Warning: Do not even attempt to use Syncthing/Pulse for general public distribution of information, regardless of anything else. Simple as that.

First of all, our PRIMARY interest in Syncthing/Pulse project is the public distribution of information applications, just like with torrents, for example. Yes, the program works, more or less, with purely private applications, such as syncing your home box with your camera, laptop and so on. And yes, these applications are useful. But that is not interesting to us in the context of the most urgent work of information distribution that has profound significance and affects ALL the people, regardless of who or where they are.

So, this is the primary context for any information about Syncthing you will find in this section.

Update: Unfortunately, we did not consider one major design flaw, and that is ANY node may configure itself to be a master node, which means a disaster for the public distribution applications. Because the master node has full and unconditional access to a share/collection and may do with it anything imaginable, and it will be unconditionally delivered to all the other nodes on a share. Any actions on the master node, such as file deletions, modifications, additions and so on will be automatically replicated on all the other nodes, including all the master nodes.

This may damage or totally destroy the entire share/folder, either as a result of misunderstanding of how the program works, or for deliberate reasons to attack the share.

Basically, the problem is that the master node may simply replace the files in a share with any kind of garbage and it will be propagated to all the other nodes, including the other master nodes, which is a disaster. Or, if some user on some master node decides to delete some files because they are not useful to him, then these files will also be deleted on all the other nodes, and he may not even realize the consequences of his actions, possibly thinking that it only affects his own node and his own collection.

Furthermore, in the current design, which utilizes the Lampert clock for the archival file version, they can make their version anything they want by merely renaming the files to any version number they want, in case the program also considers the file version to make the decision on whether to perform the file update or not.

In case the file update time is used to make a decision on updates, it may be set at the attacking node to anything they wish.

As a result, all the nodes will be updated with their changes, no matter what all other nodes, including the master nodes, think or do about it.

Update: As of version of Syncthing (0.9.18), there is a concept of an "Introducer" node, which might look like something that could be used for automatic joining the shares in general public application. Unfortunately, joining the share isn't really automatic, even with the introducer nodes. Effectively, this concept does not help much in terms of joining the share. The new node still has to be manually approved by all the other nodes that wish to share a particular folder/share with it. In order for the sync process to start between the nodes, both nodes have to enable the node(s) with which they are willing or are interested to exchange the files.

So, for now, the entire issue of using Syncthing/Pulse for general public applications is basically totally "no go" situation. Full stop.

The good news about it is that it is an open source product, which is exactly what is needed. Unfortunately, it is written in Go language, that was created by Google and has some built-in mechanisms for "global, transcontinental, massively parallel" blah, blah, blah and so on. But what has to be kept in mind when you deal with Google, no matter what issue you are dealing with, is that Google = Evil. Even the CIA and NSA pale in comparison to Google, which, at this particular junction, is probably the most powerful and influential force on this planet, and in many respects.

"Google-Berg: Global Elite Transforms Itself For Technocratic Revolution"

What we discovered was groundbreaking and represents one of if not the most important development in Bilderberg’s 59 year history.

Put simply, Bilderberg is merging with Google under the stewardship of Google CEO Eric Schmidt, a regular Bilderberg attendee. Google’s annual Zeitgeist conference, which has been based at the Grove since 2007, immediately precedes the Bilderberg Group conference by a matter of days.

[...]

Bilderberg is indeed being recast as ‘Google-Berg’ – partly because of efforts on behalf of activists to tear away the veil of Bilderberg’s much cherished secrecy, and partly as a means of re-branding authoritarian, undemocratic secret gatherings of elites as trendy, liberal, feel-good philanthropic-style forums like Google Zeitgeist and TED.

In reality, behind the scenes Google is using such forums as proving grounds on which to form the consensus that shapes the globe. We were told directly that the organizers behind the so-called “Arab Spring,” which began in Tunisia and Egypt, which as we have documented [2] is in fact a series of contrived western-backed color revolutions masquerading as organic uprisings, were recruited by Google and subsequently attended the Zeitgeist conference at the Grove.

[...]

Google is clearly positioning itself to become a force more powerful than governments in controlling and monitoring people’s behavior across the globe through all manner of different means, from cars that drive themselves (and are constantly tracked by a centralized Google database), to Google Glass which is akin to having a Google microchip in your forehead, to Google’s deep involvement in manipulating mass movements through social media as they did in Egypt and Tunisia.

"Google-Berg: Global Elite Transforms Itself For Technocratic Revolution"
http://www.prisonplanet.com/google-berg-global-elite-transforms-itself-for-technocratic-revolution.html/print/

Long story, but if you study it, you will see it in quite a colorful picture, even though not everything is quite as apparent, as it might look. And the evil starts with the language syntax. Everything is turned upside down. So, reading that code is like reading something standing on your head.

Side note: Microsoft also favors this technique of mind control by confusing things. With each new release, everything you are used to is changed, to the point that you might not even be able to find things you are used to in the previous version. In some cases, even the menus are gone and replaced by all sorts of colorful and yet meaningless pictures with some zombification images. Basically, these are the techniques of mind control and very few of those things usually help anything to the better, at least from the standpoint of a user, who is used to the way it was done before.

Now you have to start "learning" again, and that takes days, weeks if not months in some cases, as you have to completely change the way you are used to work with this thing and those things that you could do in a wink of an eye, now may take much longer, at least initially, until you "learn". This stuff is really subtle and works on your subconscious mind and perception nearly being unnoticed, at least by your conscious mind, and that's the idea. But on the level of subconscious, you feel this discomfort for months, until you finally "learn" the "new ways". And, as soon as you do, BOOM - the "new" version of the Universe is released. And so you are always "behind" and in a state of discomfort. Why? For what? What does it really "buy you"?

Well, it does "help" them in terms of "competition". Because any developers that might be also working on some products that "compete" with yours, are literally thrown back at least 3 to 6 months, until they catch up with you and feel comfortable with totally new ways of doing things. So, everybody else is forever and inherently behind MS, and by months, which is all it takes "to be ahead" in the software business.

Proposal to developers to implement the public shares functionality

This post was made on syncthing/pulse forums.

alberto101 (alberto 101) — 2014-10-18T09:11:42Z — #1

I'd like to ask if any developers, actual or potential, would be interested in looking at the ways to implement the public shares/folders/collection functionality. Yes, there are some issues to discuss, but there may be no need to attack this thing head-on and do some radical redesign, unless it looks like the end-benefits worth an effort.

Also, it seems to me that the scalability and efficiency may be improved, and I would not be surprised if it could be quite a significant improvement.

From my rough estimate, just by quickly looking around the code, it could be that syncthing/pulse will start having the performance degradation and load issues at about 50 nodes on line, and it might even choke at a 100. Sure, this needs a much more detailed analysis, but still, I doubt it will be able to perform even at medium scale installations.

In particular, keeping the permanent TCP connections seems to be one of the issues that needs further look into it.

The same thing with necessity to download the entire index. What does it buy you and why is it needed?

Thirdly, the issue of a need to know too much of information about the nodes with which you are either in sync or both of you have nothing to contribute to each other for variety of reasons.

(Edit: this basically means that you do not have to keep the permanently open TCP connections to ALL the nodes. Because with nearly all the nodes you are going to be in sync, probably more than 99% of the time. Therefore, you do not need to have the permanently open TCP connection to all the nodes with which you are in sync.)

That could be a start.

Otherwise, how could you even compare syncthing/pulse to BTSync? For example, one of their reps specifically addressed this issue by saying: well, syncthing isn't even in the same league, because it can't do what BTSync can do. Secondly, BTSync, at least in versions before the 1.4.nn disaster update, had a much better dynamics reporting GUI-wise. Because they have the Transfer tab which shows you exactly which files are being transferred and to/from where and transfer speed. And that feature I find one of the most desirable. Otherwise, you do not really see the picture of dynamics, especially during active periods where you could be transferring things to/from several nodes simultaneously.

And there are other features and abilities of BTSync that are not currently present or supported by syncthing, and we can discuss the specifics if anyone is interested. I know BTSync quite well from the user standpoint and have been working with it pretty actively for at least half a year and did some analysis of various things, limitations and issues.

So...

Hope it helps.

(Originally posted on the following thread:)

Forum thread: How is Pulse different to BitTorrent Sync?
https://pulse-forum.ind.ie/t/how-is-pulse-different-to-bittorrent-sync/1083/15?username_filters=alberto101

But then was moved by the main developer to this one (in Dev forums):

Implementing public shares (Dev)
https://pulse-forum.ind.ie/t/implementing-public-shares/1186

How to make Syncthing/Pulse BTSync comparable by implementing the public share/folder functionality

This post was made on syncthing/pulse forums.

Well, what about the following scheme?

1) There is an additional concept of a share hash. Share hash may be constructed from the share name and represents the combination of share ID and access mode (r/w or r/o).

(Edit: as described later in this doc, it is better to use the purely randomly generated hash instead hashing the share name because share name in public applications is not necessarily unique and so you may get a duplicate key with some other shares/collections "out there").

2) The share hash is constructed in a way similar to BTSync and contains one character for access mode (A - r/w, B - r/o) followed by 32 chars of a share ID hash in base32 encoding, that is 26 alpha chars in capitals [A-Z] plus 6 digits [2-7].

3) When share is created for the first time, in the Add Share/Folder/Collection there is a "Public share" checkbox. If that box is checked, that share becomes public, (and, may be, the node that created it automatically gains the status of an Introducer node).

(Edit: as described later in this doc, it could be more beneficial to use the button instead of checkbox. Once that button is clicked, you are taken to the new dialog box where you deal with the public share parameters specifically. Because most of them are not applicable to the private shares identified by their node ID or node name).

4) When some other node adds that share hash then, after it discovers at least one other node with that share, it also receives the information that this share is public, which, in terms of config parameters also automatically marks this share as public, sets its mode to r/w or r/o automatically and sets all the other config parameters to default values.

5) The public state of a share is permanent and that share can not be privatized in the future. For that, the new share needs to be created following the usual syncthing procedure. Because, once it became public there is no longer any guarantee of its private status in the future.

6) Any node that attempts to connect to or discover some other node is automatically granted the corresponding access and can start downloading the data without any user interaction. (Again, the share ID includes the [cryptographically signed] access mode).

What kind of problems or issues do you see with this scheme so far?

(originally posted on the following thread:)

Forum thread: How is Pulse different to BitTorrent Sync?
https://pulse-forum.ind.ie/t/how-is-pulse-different-to-bittorrent-sync/1083/15?username_filters=alberto101

But then was moved by the main developer to this one (in Dev forums):

Implementing public shares (Dev)
https://pulse-forum.ind.ie/t/implementing-public-shares/1186/2

Public share/folder functionality
GUI design

AudriusButkevicius - developer said:

It's no more than a days worth of work if anyone is up for it :)

Yes. But the benefits?

And the benefits are that you would get LOTS of users abandoning the BTSync for various reasons, which quite a few already have expressed the desire to do, regardless, right this very moment, and quite a few of them are going back to the 1.3.nn version because of this new web browser based UI, which is simply horrible to me.

For example, security and encryption arguments by BTSync are nothing more than words. Because it can not be verified and what kinds of backdoors they have in their code only God knows, well, and the NSA and other agencies, of course. :)

Secondly, they are closed source and you are an open source. Nobody can fix tons of their bugs, even if they could. And everybody can fix your bugs if they can.

Open source accumulates knowledge and creativity, and the closed source self-destructs eventually, because in the modern world it is not that easy to keep up with "the bleeding edge". Simple as that.

The problem with the browser based UI is that browsers are not designed for it, and HTML and CSS are not dynamic in nature. So, you need either Javascript or PHP, and that is where the problems start, but not END, by ANY means.

My estimate is that it should take a couple of days to get the first version rolling to begin testing the major aspects of it.

The main issue I see is node discovery based on share ID hash, even though the present discovery mechanism should work, except you need a translation from the share ID to node ID, which might be generated by another or existing hashing/node generation algorithm.

AudriusButkevicius - developer said:

The more problematic thing is making it fit in with the UI.

Yes. That is what I was concerned with mostly. But it looks like it should not represent much of a problem.

1) Scratch that share/folder name hashing to generate the hash ID/access. That is not such a great idea because you might easily get the duplicate share names, and that would generate the duplicate share IDs, which MUST be unique.

2) Instead, do the same thing as BTSync, and that is, you MAY enter the share name, but it is nothing more than a label, something that might make sense to the users. But the MAIN ID should be generated by simply pushing the Generate button next to the share ID edit field. By pushing this button, you regenerate a new and unique share ID/access key,. That will guarantee that share/folder IDs will be globally unique.

Yes, that is still, at least theoretically, is not guaranteed to be globally unique. But if you detect that you are a duplicate, you simply use an additional signature, which is a plain ordinary GUID. And if THAT combination is not unique enough, then you have another option: jump from the Golden Gate bridge in San Francisco.

Actually, there are two buttons, one - for the r/w access and the other one for the r/o access. Both have the corresponding edit box next to them where the results of generated key are displayed.

(Edit: well, that is not quite correct. There is only ONE button. The r/o key is a rehash of the r/w key).

3) Add the ability for node discovery via "Predefined Host". Predefined host is any host name/IP:port that knows about this share. It is not even necessary for it to have the physical data for the share in case you want to use that fixed IP:port host as some "global tracker", be it on a private network, even though it might, if so desired

a) But the ability to behave purely as a tracker is highly desirable. Because you can use it for MANY shares and all sorts of nodes. It becomes your own universal tracker. Once some share info is added to it, from then on, ANY node globally may use it just like a plain ordinary torrent tracker.

b) The "Predefined hosts" is actually a list, not just one particular host. You can add as many of those hosts as you wish. The more, the better and the higher chance of node discovery. And the beauty of it is that any host with a fixed host name or IP and a well known port will behave like a predefined host and a tracker at the same time.

And it can be changed, edited, removed or you name it any time you want. As long as all other actual or potential participants know it.

4) If you add a checkbox in share/folder edit dialog that says: "Use as a tracker only" when you create that share, that means that this node does not actually contain the data for the share. It is merely a tracker that knows about ALL the nodes on this share.

a) But the tracker functionality is on per share/folder basis and not per node. There is no such a notion as global per node tracker. It simply does not make sense logically, even though it is possible to use some nodes as global suppliers of any public information they know about. But this is the "next step", if it makes sense at all.

5) Furthermore, ANY node should be able to behave as a general purpose tracker in respect to the shares it knows about. As soon as the 1st node that actually has at least some data for this particular share is on line, any other node contacting it ever since, regardless of how it learned about it, will get the list of ALL the known nodes for this share, including the contacting nodes themselves. They simply become the members of the pool, regardless of whether they do have any data, or are empty at the moment. But there are some details to consider here.

a) Any node can behave like a tracker for as long as you discover it by any means available, such as DHT, which is a pretty good and efficient general purpose mechanism of node discovery and mapping the share ID hash to the list of nodes that have that share.

The DHT code is in public domain and there is even the github repository for it if I recall. There are versions at least in C++ and Java, and that is for sure.

DHT is a "must have" mechanism as far as I can see. There is no better and more universal and more efficient global key to value mapping discovery mechanism than DHT, unless you have a reliable and guaranteed "Predefined host" known to all and is always on line.

(Edit: you may also supply at least one predefined host when you distribute a key and this concept may be extended and further automated by, for example, using a torrent that contains the file which has a key and a list of predefined hosts available at the time of torrent creation. That could actually be a configuration file, written in JSON or any other standard format. So, you can execute the Load Configuration and that will load the parameters for the share automatically without any need for further configuration by hand.)

Alternative solution for GUI design

6) There is an alternative solution to the GUI design issue, which will reduce the amount of extra checkboxes, edit fields and lists in the main share/folder edit dialog.

You simply provide the "Public Share" button. When that button is clicked, it opens up a dialog that is specifically designed to deal with all the necessary information for the public shares. That way, you do not overload the perception of plain ordinary users, who are interested exclusively in private mode sharing, with unrelated information. So, there is nothing much to change even from the GUI aspect of it.

(Post moved by main developer to:)

Implementing public shares (Dev)
https://pulse-forum.ind.ie/t/implementing-public-shares/1186/7

Disastrous consequences for public shares

Well, persistent denials by the main developer to even consider the issues of public applications of this program with an excuse that "it was not designed for that" are not entirely groundless, even though they are in principle, if one looks into it on a deeper level.

The problem number one, and probably the most significant one of all, as far as public applications go, is the fact that the very concept of a share/folder/collection ID is entirely absent. Each share is identified by the share NAME, which, first of all, can not be guaranteed to be unique in public applications. Yes, it may work just fine for small, private uses where one controls the share names, but in public applications you can not be guaranteed that your name will not collide with others "out there".

Secondly, since there is no separate share ID for the master (r/w) and r/o nodes, ANY node may configure itself to be a master node, which will eventually and inevitably lead to devastations to the entire share in public applications, which we observe nearly all the time with BTSync.

If you have a separate ID for the r/o nodes, then they can not possibly damage the share. They can only GET the information from the master nodes, or other r/o nodes, but they can not update them in any way, shape or form. And you can also grant the master status to other nodes, if you are certain that their intent and understanding of the program workings are on the par with what it takes.

So, since you join the share by merely specifying its name, then it becomes impossible to distinguish between the master nodes and r/o nodes. The net effect is that ANY user may configure itself as a master node and simply fill the entire share with garbage or destroy any files merely by replacing them with any kind of garbage, which will be immediately propagated to all the other nodes and they will be effectively destroyed. Here's an "explanation" of one of developers:

(Note: we will comment the most significant points in-place in italics.)

audriusbutkevicius (Audrius Butkevicius) — 2014-10-28T10:34:15-04:00 — #4

It's your choice, nobody is stopping you from shooting yourself in the foot when you own a gun. If you have a tool, understand how it works before using it.

I don't think nobody should prevent anyone from anything. It's a peer to peer thing, and every peer makes its own decisions on how it wants to see the data.

This implies that "a peer to peer thing" is purely public situation where everyone is welcome to configure their nodes as they please and there are no agreed upon rules among them. But syncthing is not designed for that. It is designed for the applications where either all the nodes are controlled by the same party, or, in case of friends or some group members, all agree on a particular node configuration. Meaning that if the main supplier node is A, it is configured as a master node, but nodes B and C are to be configured either as r/o nodes and, therefore, can not update the master node with their changes, or, if any of them is set to r/w status (at some point), then they will be able to update the master node, and, therefore, do anything they want with this share and it will be propagated to all the other nodes on a share. Which means that they can damage the original copies of the files, intentionally or not, does not really matter.

It doesn't matter how the folder was shared with you, what matters is how you share the folder with others.

This is incorrect. Everything matters in current design. Because if you were previously an r/o node, but later on decided that you want to share this folder differently and assign yourself an r/w status without other nodes knowing and/or expecting it, then you will be able to modify, add, delete the originals, at least as they were intended to be. So, the originating node might think that he "controls" the situation and is the provider, but the other nodes effectively give themselves the equivalent status. That is the crux of the matter with current design. Because if parties do not know anything about each other beyond the fact that they know the others are participating as well, then, effectively, NO ONE knows how this share will behave and how it will be updated and by whom. In other words, the whole share turns into a mess.

You have access to other devices, you can check how you've set them up... I don't see a need for one client to report everybody's different configuration.

By the end of the day, if I am sharing some files with my friend, why would I ever care how he's got it setup?

Yes, you WOULD care how he configured his node. Otherwise, you have no idea what happens even to your own files because he may do anything he wants with them, not even necessarily realizing what he is doing in essence.

There is also a privacy argument here, why should you know that I have the repo set as master? Perhaps I don't trust you, and I don't want you to nuke all my files?

Sorry, but this is nothing more than a concoction. Synching is not designed for this use case. If you do not know what other nodes may do in terms of configuring themselves as r/w nodes, DO NOT SHARE ANYTHING with them. Because it is an invitation to eventual disaster. As it stands right now, syncthing is designed for totally controlled environment where everyone knows his status and every node knows what other nodes may or may not do. It is not designed with the "privacy" provisions in mind. Those things imply the public and non-cooperating mode of operation, which means anyone can do anything they please, and it is "OK" with others and does not affect them in unpredictable ways. But syncthing will not work with non-cooperating situations.

It introduces a lot of complexity to the UI, and the only benefit I see it gives, is for the lazy not having to go around into multiple devices checking if they set it up correctly.

This is just a concoction with net value rapidly approaching zero.

But lazy people who usually come up with genius ideas, hence feel free to use the cli to craft up something which gives you an overview of your cluster.

This is just another concoction. Because CLI does not give you any particular "magic powers" and it can not configure nodes in a way different than it is done via GUI. All it does is to allow you to work with the engine without GUI in a command line interface. And that is ALL it does.

Master folder information on other devices (Support)
https://pulse-forum.ind.ie/t/master-folder-information-on-other-devices/1267/4

First of all, the arguments are profoundly incorrect. EVERY ONE of them. Basically, these are not the arguments, but the concoctions, trying to defend the conceptually incorrect design that has a fundamental flaw and a built-in potential for damaging or even destroying the shares.

For example, you would like to give the sync access to the latest version of your site so that everyone can have the freshest version of it, even if it is inaccessible or even blocked. Now, as it stands right now, this can not be done. Just because your site is no longer yours. Because anyone in the world can edit it, delete or add to it anything they want or modify its contents in any way they please, merely by configuring their own node as a master node!!! WHAT?

You can not even guarantee that you yourself can update it in a reliable and predictable way, unless you create the 2nd copy of it and give it some other name and do not disclose that name to anyone to prevent the damages to the site. So, in that case, you can assure you can update the site, but the public version of it is still vulnerable to all sorts of chaos, be it out of misunderstanding or an outright attack on it.

This entire line of "reasoning", which is nothing more than denials of the REAL issue, as real as it gets, simply translates into a dire warning to all who might think of using Syncthing for public applications without even suspecting that their share may be eventually converted into a garbage dump if someone decides to attack it.

This particular developer is simply not being honest and is trying to defend an essentially bankrupt and clearly incorrect position and a critical design flaw. Unfortunately, these are the kind of developers that will be eventually left on the project after all the "undesirables", who vocalize their opposition to some design flaws, will be eliminated by the main dictator, as funny as it is.

Side note about dictators and Intelligence

Because the dictators are essentially sick people and they carry around their monster size egos like some treasure and demand the submission from anyone they deal with. ALL the dictators are dishonest, oppressive and violent, by definition. Because they refuse to recognize the mistakes they make and crave to maintain their "god"-like image and status, idiotic as it gets, simply because it inhibits their own growth and growth of Intelligence as such.

And so, eventually and inevitably, who you see around the dictators is the obedient ass lickers. Because dictators won't tolerate anyone else around them, and especially those, who might challenge their "greatness" and their pathetic egos forever craving for approval, even from the "herd". This is the law, and not some theory or fiction, and if you don't see it today, who knows, you might be able to see it one day, provided that you are not closed and are not rock-like and are willing to change, grow and learn, from others and everything.

Secondly, no one in this world, or any other, is "greater" than any other. They are all just individuated expressions of the Infinite Multidimensional Intelligence, ALL Pervading and Ever Unfolding and Expanding.

The very idea is just a lie, and a grand one at that. And this is also blindness, probably the worst there is, to imagine that Intelligence, ALL pervading and Ever Expanding, could be "better" in once case and "worse" in some other. This is simply an absurd and the deepest level misunderstanding of what is Life and Creativity, forever unfolding, in ALL possible directions and dimensions.

Because in public applications you have no control over what other people may and will do to it. Sooner or later, some violent person is bound to join this share and will decide to fill it up with garbage, merely as a joke, for example, or as a result of clearly evil motivations.

The "moral" of this story is simple and so is the solution:

Implement the concept of a share/folder/collection keys, and the separate ones for the master nodes (r/w) and the r/o nodes, and distribute the master keys ONLY to the people you can trust like you can trust yourself. All others are considered to be the leachers and are not interested in contributing anything to the collection, or may not even be allowed to do so because any arbitrary changes may damage the integrity of author's intent and a purpose. Else, no matter what you do, this thing will never work. In principle. Because anyone can destroy the collection.

Yes, one can argue: well, as soon as I notice the share is being damaged I can disconnect that particular user from MY node and restore the collection, if I am smart enough to keep the original copy stashed in some other place. Except you do not even know which particular user destroyed the collection. Secondly, the collection is still destroyed on ALL the nodes, and so every node has to go through the same pain as you did and disable that node. But how does he restore the collection? He might not even have the archive flag turned on, so the entire collection may not even exist on his disk except in damaged version.

That means that he has to resync everything again, and so is everyone else, except that he can not control the collection even though he might have the master node status set on his node. Because the source of the collection is on some other (originating) node and only that particular node may restore the original version of the collection, and even then, all the nodes have to resync again, from the start. Can you imagine what happens to large collections having humongous number of files and total size of terabytes, as some people report their collections?

Average Joe and public distribution of information

Following is the arguments of one of the Syncthing/Pules developers regarding the need for public shares/folders/collections. Basically, his argument is simple: Average Joe does not need this because he only wants to sync his pictures and "does not care" about anything else, just as his mind was programmed...

Basically, the whole argumentation looks like a "twilight zone" talk.

Well, initially I did not feel like replying to this post. But that would mean disrespect. So, just briefly:

AudriusButkevicius, post:5, topic:1016

I think the added complexity of this doesn't make it worthwhile.

Note: interestingly enough, the same developer, later on, after step by step instructions to implement this functionality were presented by the author of this writing, has said "It's no more than a days worth of work if anyone is up for it".

Public share/folder functionality

Implementing public shares
https://pulse-forum.ind.ie/t/implementing-public-shares/1186/6

Depends on how you define "worthwhile".

AudriusButkevicius, post:5, topic:1016

You don't think you can achieve this granularity of control in BTSync, nor I think any other tool available.

What does BTSync have to do with this? Secondly, do you consider BTSync as some kind of a standard by which all are to measure?

AudriusButkevicius, post:5, topic:1016

This special case that you have is very corporate,

Sorry, I do not see it this way, by ANY means. Me, personally, have nothing to do with any kind of corporate anything, and I'd love to see this feature, and have seen plenty of users asking for this sort of thing. Implementation wise, this is probably one of the simplest things to implement and could be done in a couple of hours I suspect. Unfortunately, I do not know Go and it will take a few days to study the code.

Secondly, I would not want to fight with main developers trying to prove something. I have no interest in that.

Thirdly, I am as busy, as it gets and for me to get involved on development level would mean I have to dedicate every minute of my life to the project, and for weeks and months. Yes, it would look and like a tank as a result, if we do, and I have not doubts about that because I have developed a methodology of development in general (that allows one to proceed pretty rapidly). But this is a totally different subject.

AudriusButkevicius, post:5, topic:1016

and I think that average Joe who wants to sync his pictures from his phone to his laptop, and perhaps later share them with his wife doesn't need this case.

Fine with me. There is such a thing as Default behavior. I could also be considered an "average Joe" with quite a few programs I use, especially during the complex installs and configuration. But I just accept whatever defaults the program provides. I do not have to even understand every checkbox and its esoteric meaning.

AudriusButkevicius, post:5, topic:1016

It seems that you want to make Syncthing a swiss army knife, which does everything in every possible way,

Well, that certainly would be great, but I do not have expectations set so high and there are only few most important things I am concerned with as far as sync goes. And I know precisely what I want to see, and the list is not that long. But that does not mean that others agree with it, and I have no problems with it. Everyone has a right to see the world and Life in the way they choose and experiment with things THEY like, not me. I am not anyone's judge as to their choices.

But it does not mean I have to be like others, does it? I also have the same freedom as anybody else, if you don't mind.

AudriusButkevicius, post:5, topic:1016

though I already feel that Syncthing is complex enough to repel average Joe from using it, hence adding more knobs and buttons to specify some very specific set of cases is not what I think should be done.

Well, again, "default behavior" is an operative world here, and no, you don't need to add tons of checkboxes. Just one would be sufficient for now and its default behavior could be configurable via configs, just like any other default which you can change if you are not "an average Joe". And if you are, just skip it. Shouldn't be a problem.

Life does not start with average Joe, not does it end, nor the planetary systems turn around him, nor creativity of any kind becomes possible thanks to him.

AudriusButkevicius, post:5, topic:1016

I think for this very special use case, and you can already achieve what you want by switching off introduction, and do your thing manually, or scripting stuff up via CLI...

Introduction is currently per node global, unless I am missing something here.

AudriusButkevicius, post:5, topic:1016

This way, you are happy, and average Joe is happy not having to understand what 10 additional buttons do.

Where do you see 10 buttons? Could you enumerate and give a functional description of more than one at this point?

AudriusButkevicius, post:5, topic:1016

But then again, I am not the project owner, hence this is my personal opinion which does not influence anything.

Let me ask you one thing: do you consider yourself to be "an average Joe"?

I did not even suspect Syncthinig was written for the "average Joe". I think it is pretty good as is considering that it is a relatively new product. I am using it for a few months and it works reasonably well so far.

The question, from the creative standpoint, would be: where do you take this thing from now on? In which direction do you develop it? What are the most desirable things to be done, that are, at the same time, interesting to developers to work on? (Did you ever bother to ask such questions?)

Am I hearing that development stops from now on because the needs of an "average Joe" are fully satisfied?

Finally, at this point I have a complete architecture that, as far as I can see at this point, fully describes and covers all possible cases and in very simple and clear fashion. It would be clear even to an "average Joe" if he bothers to even spend a minute or two to understand it. And it is many times over more sophisticated than just one or two checkboxes functionality. And it is very simple to implement.

In a couple of words: you need a list of clusters and you can announce each cluster in any way you please without any logical conflicts or contradiction of behavior or logical complexity. If you are interested to really investigate this thing, I'd be willing to give you some time and my energy, but only on MY turf. I would not like to interfere here with things that are not of interest to developers.

So, just ask, if you care, and I will give you a link where we can discuss it in a very relaxed and productive manner, without any interference from any one and with full freedom to express whatever opinion one might have.

That is ALL I can do for you in case you care.

Otherwise, I am done with this subject here. If anyone cares to clarify something, fine with me. But ONLY in a CONSTRUCTIVE and creative matter. I am not interested in insults, post mutilations and reverting to violence of any kind. I just do not like that energy. By the way, when I got back to my box, the first thing I saw is your post. The second thing I saw is I lost my main disk. It was just gone. It was a situation where your hairs may raise.

In case you did not know, there is such a thing as energy projections, and they could be very destructive and could have profound effects on others, but eventually on the attacking entity. Essentially, they just hurt themselves more than anything else. I hope you can comprehend what I am saying here.

(Edit: this was said in relation to one of his highly negatively charged, insulting and angry posts directed at this author.)

Good luck and enjoy. Life is a Grand Ride for me, and for anyone I care to deal with.

Introducer description and functionality
https://pulse-forum.ind.ie/t/introducer-description-and-functionality/1016/6

How is Pulse different to BitTorrent Sync?

This post was made on syncthing/pulse forums.

alberto101 (alberto 101) — 2014-10-16T09:03:06Z — #18

hobarrera:

Here's a couple more points:

* I belive the BT Sync protocol is propietary too, not just the implementation.

Well, let them be as proprietary as they wish in their attempts to control the sync market. But you don't HAVE to be compatible to them. In fact, it is THEIR "problem" to try to be compatible with you, if your higher level logic is well designed and YOUR version is more stable and reliable and more consistent and assures the database consistency across ALL the nodes regardless of anything.

Because, at least currently, their protocol, if they have such a notion to begin with, is NOT consistent and does NOT assure the data consistency across the nodes. And this is a fact, which I have seen personally in numerous cases. In fact, I can provide a number of cases that could be easily verified independently. I can even claim that I have never seen anything so inconsistent as their "protocol", if there is any. From what I was able to gather so far, they are simply using the UTP protocol and all its mechanisms. I have my reasons to doubt they even have such a concept as a higher level protocol and logic. It all seems to be wired together in ad-hoc manner.

So, by the sheer fact that you are an open source and can provide as much of protection against snooping, security and so on, YOU become the standard by which others are to measure. Because your source is open and can be verified, and, which could be even more important, ANYONE can improve on it, extend it and you name it, if they have what it takes to improve on a tank, and if this is acceptable by other competent developers, it will be wired into the next version or so. Why not?

The "bottom line" is you do not HAVE to be compatible with BTSync, or anyone else for that matter, any way you look at it. If your system works and is reliable why should you even bother about BTSync? Sure, if they ever decide to go open source, which they won't, at least in the foreseeable future, and they have proposals on how to improve things, then it is a different matter. And even then, you are not forced to follow up on their proposals if you find they do not actually "buy" you anything of real value.

I'd be willing to discuss this issue with anyone having an argument.

hobarrera:

Pulse has local network announcement (eg: nodes in the same LAN can communicate offline). BTS needs to be online or have access to a tracker to find nodes.

Well, these things are minor issues as far as I can see, and so are the issues with network protocols, such as TCP or UDP, or even ICMP, need be.

In fact, you can utilize the synthetic approach and use some protocol for a specific operation only, and not system-wide, that gives you the most benefits and provides better performance.

For example, you may decide to do UDP broadcasts or send packets containing a slice of information on the current state while contacting other nodes you know about at this point.

One of the important issues is node discovery. For that, you are welcome to use quite efficient DHT network approach, which allows for distributed storage of node discovery information across all the nodes of the network. So, this way, you do not carry ALL the DHT traffic on every node and do not need to store the entire DHT tables on your box. You just carry the traffic and a part of the DHT tables of the nodes "closest" to a particular hash you are requesting.

And so is the issue with trackers. Technically, you do not have to provide some centralized, prewired tracker. You MAY, but it is not a predefined requirement. The trackers need to be configurable on per installation basis. Sure, you may provide a default and "guaranteed to work" tracker with a release. But even that one may stop working for whatever reason, and could even permanently go off line in the future.

I also like the approach of any node behaving like a tracker and upon ping would tell the requesting node about all the nodes it knows about for a given share. There could be even two different pings - global - per node, and per share only. But this needs more thought, especially in the private network application, where you know for fact that your data for some share(s) is strictly private and confidential and you may not even let the other arbitrary nodes to query some information about those particular shares.

I also like the idea of keeping the node history. Meaning, once you see some node on line, it goes into your history file. So, even if that node goes off line for any length of time, you can still have its information. So, in cases of static IP/domain:port situations, you can periodically query the nodes you saw before to see if they respond. This is a fallback strategy to node discovery.

Basically, I am of the opinion that node discovery needs to be fully decentralized, or at least give you a configurable option to do so. Nodes and networks may have their own trackers or they may share some of them and should not, at least in most cases, rely on some fixed central tracker.

But when you see that the TCP connection may be more beneficial for some specific operation(s), then why not use the TCP? The TCP connection does not HAVE to be permanent to any node, if you find for any reason that you would not like to keep a huge number of connections simultaneously. But I think on modern hardware, you can easily utilize at least 50 simultaneous connections without winking an eye. The additional TCP handshake and connection overhead is not that significant once you start a transfer session. It is only inefficient in cases when you have burps of information, such as various queries that do not necessarily cause any further communication with that node.

Furthermore, BTSync can fall back on alternative protocol. This could become important for example if the company/organization network blocks some protocols for whatever reason. I recall some such discussions on BTSync forums.

Basically, you don't have to get stuck with specific protocol, and even there, for the most part it is just an efficiency issue and not something fundamentally important.

So, all these broadcasts, handshakes and all sorts of bells and whistles can be implemented in variety of ways. But fundamental things, like higher level update logic engine can not and remain the same, regardless of anything. Because they are designed like a tank to handle the craziest conditions imaginable and guarantee the data consistency across all the nodes.

Hope this helps.

How is Pulse different to BitTorrent Sync? (Support)
https://pulse-forum.ind.ie/t/how-is-pulse-different-to-bittorrent-sync/1083/18

Choice of programming language and GUI design

This is a little technical, so, people without technical background and programming experience may skip this chapter or its parts even though a quick discussion on program user interface may be helpful in understanding some general issues.

Anyways, the main argument of main Syncthing developer to use Go is that it is highly portable language, which is great, because it allows for support of wide variety of platforms. But its GUI aspects are basically browser based and are quite primitive compared to natively designed GUI.

And GUI design and its power and flexibility could be easily underestimated. Basically, a native GUI design, meaning it is compiled to run as a binary executable code vs. web based design differ drastically in potential power. In native design you can have the features of sophisticated manipulation of massive amounts of data. For example, you have a huge list of files that need to be displayed in GUI in some box or a list. With native design, manipulating such a list is relatively easy and natural. Take for example the multiple item selection and then deletion or dragging and dropping aspects. Web browsers are not designed for these things.

In many cases, you have some dynamic language underneath, such as Javascript or PHP. But with all due respect, those are not as powerful as native code, neither in terms of performance, nor in terms of power and flexibility. If you observe, any time you go see some dynamic web page the response to user actions is sluggish in most cases. You push some button and you have to wait, and not necessarily because of the network speed.

About Javascript and privacy:

If Javascript is enabled for some site, then your privacy simply does not exist, as you can be nearly perfectly identified no matter what your IP, login, password or anything else. Yes, you may claim that since you are dealing with your own nodes then having a Javascript enabled on them represents no risks. Sure, everything looks good on paper. But reality is not just paper. When it hits you, it is no longer a theory.

Because Javascript exposes so many things about your comp that in combination they are equivalent to a fingerprint, as unique as it gets. Among those things are the screen size, installed fonts, IP/domain, plugins installed, version numbers and so on.

It does not matter which login account you use and it does not matter where are you coming from, your box is identified in nearly perfect way, with odds better than 1 to a million.

Another aspect of GUI is packing as much useful information on the screen or a dialog box as possible and being able to drag and drop things even to some other applications. With web based GUI such things are practically impossible.

Basically, we could go into GUI aspects deeper, but no matter how you cut it, browser based GUI is simply not something natural to the browser, and for quite a few reasons.

Therefore, the native GUI, as provided by the native mode or inherently portable languages, such as Java, is far more superior, far more flexible and allows one to pack much more of significant information into the smallest possible screen and window size.

One of the most important characteristics and properties of GUI is to pack as much of most significant and useful information per square inch of the screen, as possible, which isn't quite the case with browser based GUI, even though one might argue that with more extensive and more precise use of the CSS styles you should be able to pack things to the same degree as with native GUI design. But this is mostly a wishful thinking.

The better choice, in our opinion, would be using Java language, which is also about as portable as it gets. It is INHERENTLY portable, because it uses the JVM (Java Virtual Machine). One other plus for Java is that it uses the familiar C/C++ syntax and could be easily read and understood by most programmers.

So, why turn everything upside down? What does it REALLY "buy you", in principle? Why do I have to stand upside down just to be able to read this thing? In my personal opinion, a well designed language and a well designed program should read like a morning paper with a cup of coffee and not like the grand unifying equation of creation of the Universe. Why create the stress and strain? Is it REALLY necessary? For what?

Java is a well designed language and you don't have to stand upside down to read it. So, if there was some automated conversion tool that could convert at least the bulk of the code from Go to Java, that would give Syncthing a significant boost and create plenty of interest for other developers to join the project. Several people expressed pretty much the same opinion as far as Go language goes. But this is too technical to discuss here.

Interface to the 3rd party GUI subsystem and extending the functionality

What is highly desirable at this point as far as synthing/pulse goes is providing the full functionality access to the core of the program via interface to external 3rd party GUI subsystem.

Basically, the way it is designed right now, the GUI, even though it looks quite clean and functional, still does not show all the dynamics of transfers in the ways the users like to see it. One of examples of that might be BTSync. In versions before 1.4.nn GUI had a Transfer tab which shows you exactly which files are being transfered currently and to/from which nodes and their current transfer speed. So, just switching to this screen in a single mouse click you see exactly what is happening.

Also, a browser based GUI subsystems are not really powerful or effective. Because web browsers are not designed to behave like a well designed native GUI subsystem. Such things, for example, as drag and drop, dialog box resizing, multiple item selection in some list box and some other things are not quite possible in a browser.

Another unpleasant aspect of a browser based GUI is that it does not allow you to size and move the dialog boxes and pack as much useful information in each of them as, for example, you can see in the O/S file browsers. Moreover, it might contain some visual elements of the browser window, such as button and command bars and so on and those things occupy the extra space on the screen and are neither necessary, nor desirable in the context of some GUI screen.

Each pixel of the screen needs to be used effectively in order to pack as much of useful information per square inch of the screen, as possible. This is particularly applicable to all sorts of mobile devices with limited size screens.

This means that if you provide a fully functional GUI interface to the program core, you can hang on it your own GUI subsystem, going as far as developing a native mode version that could or might be O/S dependent.

For example, you can design the sophisticated and high performance GUI subsystem in Java, which will provide you the support for most platforms out there since Java runs on the top of their own virtual machine (JVM). And it is relatively easy to write the Java code for the GUI and it can be done in a matter of days.

So, if you have such an interface and it supports the async mode of operation via event notification messages, then your GUI becomes responsive and it does not consume much of the CPU resources. Plus it does anything the way you want it, and not merely provide you the basic functionality of a browser based design.

High performance program core interface to the external programs or program controlling mechanisms.

Also, providing a well designed, language independent, interface to the core of the program will allow to control it and via 3rd party mechanisms. This could be done for all sorts of reasons, starting from adding the new functionality, not available as is, performing a much more informative logging facility which allows to see much more details as to dynamic aspects of operation, such as a progress of file transfers, and for other reasons.

This interface needs to be able to support the asynchronous operations and information exchange between the core of the program and the external controlling programs or functionality extensions.

Ego politics

The ugly part of the "good, bad and the ugly"

Well, the following section might be suitable to those who like the excitement of a battle and have courage to look at the raw reality of it, without any "nice" embellishments of the phony world.

Basically, what you will find in this section is how all sorts of arrogance, perversions and denials of some of the key issues with Syncthing/Pulse have been accumulated until it finally literally blew up in utter insanity of "suspending" the messenger from the forums for 10 YEARS, if you can even imagine such a perversion and the utmost blood-boiling hate.

And suspending for what specifically, one might ask? Well, if you have guts, then you might be able to figure it out and make up your mind by reading this section, even though it is not for the faint-hearted. Basically, "suspending" for things that have absolutely nothing to do with either Syncthing, or sync software as such, or even with things that have anything to do even remotely with software and technology as such.

WHAT? How could that be? - Well, who knows, ANYTHING is possible in this world some say. See if you can figure it out if you have any interest in intense impressions.

Syncthing problems and issues

Eventually we were banned ("suspended") on Syncthing forums, and for the most bizarre reasons, having nothing to do with either Syncthing/pulse project, or with being "off topic", or doing anything "bad" or "illegal" whatsoever, and for 10 YEARS, if you can imagine such an insanity. Here's some explanation of the that ban and its possible implications from the standpoint of this takeover of Syncthing project by the ind'ie operation, which seem to lead to Amazon:

Mutilate, delete, deny, kill and ban all those who disagree with your self-image of grandness!!!

It is nauseating to even talk about that ban, even though it is profoundly significant event in itself, even for the entire project, and not to us personally. But it has to be done, so that the general public and other developers have a chance to take a hard look at what is happening to the project and think well before they decide to continue to contribute their energy and efforts to it. To us, this foolish ban means nothing more than this project will no longer be supported by the forces beyond visible and no further ideas are likely to be forthcoming to take it to the next level, even though we will try to outline the most significant ideas here, in this document.

At this junction, it seems that probably the best way to progress with this project in truly open source spirit is to fork it at the point of license conversion and to form a new team to keep it truly public and operating on genuinely democratic principles and open in the deepest sense of the word. Because there is a need for such a project in the modern public information distribution sense, even though some other projects are beginning to show the signs of significant progress.

Furthermore, it is at least conceivable that this project takeover is based on some motives other than purely commercial ones. It could be that the reason for this conversion is an attempt to prevent and/or interfere with public distribution of information in the context of the global information warfare currently in progress as we speak.

One other aspect of it is surveillance and one of the goals of this takeover might be to eventually install the back doors and surveillance mechanisms into the program, at least in the binary distributable versions, where some parts of the project may become closed to the general public and publicly distributable sources would only include those parts of the program that do not have to deal with the interests of various agencies, such as NSA, for example. For example, as Snowden pointed out, the information gathering operations by the NSA are massive and all pervasive.

Technically, the key information about any node or share/collection may be easily forwarded to the 3rd party, which, in turn, may pass it even to the global network routers that may begin to block and/or interfere with P2P communications via methods as simple as merely dropping the packets.

Sure, we do not know for fact what is really going on behind the curtains regarding this takeover and it does not look like any of it or the real motives behind it is even intended to be disclosed to the public and even to other developers, and so we might never know what is going on behind the curtains.

Because that ban indicates that the main developer is essentially a totalitarian dictator with hyper-inflated ego, who only understands the principles of domination, which eventually and inevitably leads to violence and elimination of the "undesirables". Because in HIS twisted mind, he is the only one that "counts". It is like because of HIM the Sun is rising in the morning, if you can even conceive of such a thing.

That means that this individual will not tolerate anything that does not align with his own ideas and his self-image of grandness, and will go as far as to mutilate, delete, deny, kill and ban. ALL the dictators do. In fact, from our very first message on the Syncthing forums regarding some ideas to implement the public share functionality his response was an ugly form of intellectual insult and an attack on the "messenger". This was quite a strange response in the context of purely technical issues that could be dealt with by simply stating ones own position, point by point, instead of resorting to the level of outright insults.

But who controls the Syncthing/Pulse project now?

That means that this project might be in jeopardy as it stands right now. In particular, this "deal" of converting Syncthing to "Pulse" essentially means that the whole project has been taken over by the commercial interests.

This is what WHOIS records show for this ind.ie operation, Noteworthy is the Amazon.com.

Dates 	Created on 2007-04-02 - Expires on 2015-04-02 	
  
IP Address 	54.230.68.76 - 14 other sites hosted on this server 	
  
IP Location 	United States - Virginia - Ashburn - Amazon.com Inc.
ASN 	United States AS16509 AMAZON-02 - Amazon.com, Inc.,US (registered May 04, 2000)
Whois History 	114 records have been archived since 2006-10-03 	
  
Whois Server 	whois.domainregistry.ie
Website
Website Title 	  Kieran Masterton 	
  
Response Code 	200
SEO Score 	86% 	
Terms 	211 (Unique: 142, Linked: 150)
Images 	7 (Alt tags missing: 6)
Links 	102   (Internal: 74, Outbound: 27)
Whois Record ( last updated on 2014-10-28 )
domain:       ind.ie
descr:        Indietech Domains Limited
descr:        Body Corporate (Ltd,PLC,Company)
descr:        Corporate Name
admin-c:      ADD376-IEDR
tech-c:       AAM456-IEDR
registration: 02-April-2007
renewal:      02-April-2015
holder-type:  Billable
wipo-status:  N
ren-status:   Active
in-zone:      1
nserver:      ns-526.awsdns-01.net  
nserver:      ns-1987.awsdns-56.co.uk  
nserver:      ns-258.awsdns-32.com  
nserver:      ns-1234.awsdns-26.org  
source:       IEDR

person:       Aral Balkan
nic-hdl:      ADD376-IEDR
source:       IEDR

person:       Blacknight.ie Hostmaster
nic-hdl:      AAM456-IEDR
source:       IEDR

Note: well, we just saw one message from ind'ie guy on syncthing/pulse forums which says:

aral (Aral Balkan) — 2014-10-10T04:47:32Z — #10

simplypeachy:

nslookup api.ind.ie
…Amazon.com, Inc. AMAZO-USEAST-11…

Hi Simply Peachy,

Thank you for sharing your concern.

The app at api.ind.ie is a Node.js API that is used by the static ind.ie web site for any dynamic features we need. Right now, it doesn’t do much (it was used mainly for signups for the Indie Summit). It is hosted on Modulus.io, who host their infrastructure on Amazon.com. We also host certain services on DigitalOcean. And the web site itself is static and currently hosted on Amazon S3.

Introducing Pulse, and ind.ie
https://pulse-forum.ind.ie/t/introducing-pulse-and-ind-ie/1074/10

Well, may be, may be... But why not just use some VPS solution for $20/mo. and install anything you want on it? Does $20 represent such an issue for the "sponsors" that promise all sorts of "support" to this project, that they are compelled to use the companies that are not known as the hosting or VPS providers?

What is the benefit of using the Amazon or even coming close to them? What business they are in? - Hosting? Or buy/sell anything under the sun?

For one thing, accessing the syncthing forums now shows us the spinning wheel in a browser nearly half the time. What kind of "improvement" is it and of what?

Change of license issue

Now, the first indication of the effects of this takeover of Syncthing project was the change of license from MIT to GPL-2. For what specific reasons was it done? That change has some profound consequences. See the discussion here:

https://pulse-forum.ind.ie/t/license-question/1066
(Support forum: License question thread)

GPL is probably one of the most convoluted and complex licenses one can find and one particular person, who claims to have spent years on dealing with various licenses warns "if anything, stay away from GPL like from a plague" (not a literal quote).

Secondly, if you are a honest man and are not merely an ego maniac or a totalitarian dictator, you don't do things like these without any preliminary discussions with other developers or even with users, bouncing things back and forth. Because things like license conversion have some profound effects on the entire project and all the developers that contributed their time and energy to it. Simply doing it silently and putting every other developer before the fact that the massive code changes related to license conversion AFTER it has been already submitted, could be interpreted as what kind of thing and according to what kind of motivation and intent?

Well, to put it bluntly, it was a spit in everyone's face and a "screw you all" attitude of some dictator-in-chief, or an arrogant fool, at best, that has no respect to anyone else. No honest, half-decent individual would be able to even conceive such a thing, most likely. Because it is UTTER disrespect to all other developers and even the users that have given this project their trust and might have even involved some other people and started doing things that depend on it in the understanding that this is truly an open source and non-commercial project.

One has to be profoundly sick in his very soul to even conceive doing such a thing in the way it was done. Because it is well beyond any reason. Basically, it is a mental department level, just to be blunt.

Furthermore, as it was "explained", this license conversion was done without explicit agreement with other developers, or even a discussion of it, since no one "objected", which is a perversion. Yes, those who are currently actively involved did not vocally object, and it could be because they know that they are dealing with a dictator and in his language and understanding objecting to anything means you are considered from then on as "an enemy of the people" with all the consequences thereof. This is a standard trick of all the dictators.

But those developers that are not currently active on the project or could be temporarily busy at the moment with other things in their lives, did not even have a chance to know or realize what this change of license means for the project and all the work they have done for it, and it does mean quite a few things. Furthermore, all the developers, currently contributing to this project, may not fully comprehend the real motives for this license change.

Because the issues with licensing are some of the most complex there are, and to fully understand all their implications or hidden meanings and/or motives you'd have to be a pretty competent copyright lawyer, and the copyright law is one of the most complex areas of the law according to the competent lawyers.

I am the king, screw everyone else

Then, the main developer begins to behave like a CEO of some sort. He is invited to the meetings with these "sponsors" that promise the project "support" and all sorts of "bright future" to butter his hypersized ego and to make him feel important and grand in the scheme of things. And he does not even ask the other developers whether they all agree that he should go there or not.

Moreover, what is the need to be physically present and travel some place? ANY kind of information, questions or issues could be simply discussed, even in the conference setting, directly on line. So... What was the REAL reason to physically travel anywhere?

Nor does he tell them the purpose of those meetings and what is intended to be discussed there and asks other developers of their opinion on whether they are even interested in such discussions.

Nor does he report the results of it when he comes back, and so no one knows what was actually discussed there and what was agreed upon behind the curtains.

Essentially, he behaves like a "king" of some kind and does not seem to even consider that the opinions of others matter in the scheme of things. Because the only one that counts is he himself.

Yes, if this was his own private project and no other developers have given it some of their energy and life in the understanding that they are doing something for the common good, then it is another matter. "My toy, I play with it in any way I please".

But what he did instead is to open up a project as an open source, hung an MIT license on it, which would be acceptable to anyone, more or less. Then people stated joining and contributing to it. And then - boom. It turns out they are just "nobodies" and mean nothing. At best, they can fart or even piss against the wind, if they don't like something, and that is the extent of their significance "in the scheme of things". WHAT?

What this all means is that developers might have some naive expectations and understanding that they are still working on an open source project, whereas they are working on/for something entirely different from now on, and the sorriest thing about it is that they do not really know what is it.

He is probably given a grand reception, like a king of some kind, invited to some meetings with the "important" people, and it is whispered into his ear how "important" the particular individuals are. He is probably placed in a very nice hotel and could be even driven on a limo, just to butter his ego and to buy his corrupt soul of an ego maniac.

"Bright future" for the project - hell

During the discussions of this "conversion" on Syncthing forums it was quite something to see all the praises and promises of the "bright future" offered by the new "sponsors". It looked like some kind of Herbalife or Scientology promotion rap, full of the same kind of hype and the same sleazy ego buttressing templates and techniques.

Furthermore, these "sponsors" and their agent, giving a rap about the "bright future" for syncthing and investments of resources into this project, did not really disclose the REASONS why are they so interested in this syncthing thing. When we looked at their site the last time around, they had some beep, beep, beep type of projects and were offering to build the "communities" that can exchange files, pictures and other things. But, according to the main developer, the program is not designed for that.

If any user can configure his node as r/w (master), then the whole "community" will pretty quickly turn into a madhouse, "ruled" by chaos, as various people begin to add/delete/modify whatever they think belongs to them and their own collection, but, actually, mirrors what everybody else has or does, and everybody can mess with the stuff of others. It is like: you look at the screen the next time and notice that all your pictures or even videos are gone, and now you have tons of other crap, that magically appeared, which you don't even want to see. So, what do you do then but to delete it all, which will delete it all on all other nodes. And some other people might start adding it all back because they happen to like that particular zombie stuff. How long will it take before all of you will simply go mad, not even realizing what is going on?

Well, some say: "do not try to make something grand out of something that may turn out to be mere stupidity". But some also say: "better be safe, than sorry". Yes, to look for some sanity in a madhouse could be not something beneficial to one's own health and well-being.

But we choose to play this game to its fullest potential, until it exhausts itself like a balloon. All you need to do with a balloon "to see its nature" is to pinch it with a small needle. So, join the trip if you wish. Else, go find something "better" to do with your life and time.

Who are these ind'ie clowns promising a "paradise" and what does Amazon have to do with it?

And who are these ind'ie clowns? Where did they come from? Where do they get the money to "sponsor" this thing and what kind of financial benefit do they expect to gain as a result of taking over this syncthing? What was the purpose for that 4 day trip of main developer to visit these clowns and who paid the expenses for it? If they are mere nobodies, then who is going to throw away thousands of dollars on such trips? And what was the need to see the main developer in person? All the issues can be discussed in all sorts of ways on line. About the only reasons there is a need to meet in person is either to sign some papers of impress the ego maniac in order to buy his soul cheaper. What other alternatives are there?

The whole thing looks like a puppet theater and nothing more than that. But the REAL motives are not disclosed or not even discussed in a way that makes any sense. About ALL you hear is grand level hype and promises of some "bright future".

What seems to be strange here is that it is not clear for what exact purposes the Amazon, if it is them who stands behind all this, is interested in this project. Amazon is in the business of public distribution of products and public distribution requires the public functionality in Syncthing, if that is what they intend to use it for. But, even after the meeting with the ind'ie "sponsors", the main developer still refuses to even consider the public share and have bluntly stated "I do not need it and I am not going to implement it". Like it is his private toy and not an open source project to be used by all sorts of people with all sorts of needs and applications.

So, what would the Amazon need this thing for if it does not implement the public share functionality and, moreover, the main developer flatly denies to even consider such functionality? Do they intend to use it exclusively for the internal corporate network and do not even consider it as a tool for public distribution of information?

So, there are plenty of questions and warning signs of all kinds related to this "conversion" and you need to investigate the issues for yourself and in the in-depth manner and come to your own conclusions. Because we can only guess here with the information that is publicly available at this junction (as of October 29, 2014).

Now... As a result of this utterly foolish, utterly perverted and evil-intended ban, which does not help anything whatsoever as far as this project goes, we can not contribute anything to the project via forums. Nor we can express our opinion or view on some matters and the issues. But, nevertheless, the issues still remain, just as Truth itself. So, we will cover some of it here for everyone to have a chance to look at the whole thing from a different perspective, and especially those developers, that might consider joining this project, to be forewarned of some pretty nasty effects of some of design flaws and product limitations, shortcomings or some general project issues.

Side note: Otherwise, we have no interest in this project in the current setting and alignments. Our primary interest is global information distribution in public setting, primarily to disclose true information about the evil, the so called NWO scheme of taking over the world and converting it into the Luciferian global empire of evil by the "banking mafia", Rothschilds, Kuhns and a few other biggest bankers in the world that took over the FRS back in 1913 and, since then, totally took over the USA and finally devastated it as a sovereign state and converted the constitution into a toilet paper.

Every day, more and more sites on the net are being blocked under false pretenses in gross violation of the Basic Human Rights Convention. A result, some of the most important and critical information disclosing the very roots of evil becomes inaccessible and that information is vitally needed to prevent the takeover of the entire planet and destruction of over 90% of global population, as planned in this "NWO" grand plan.

That is why open and automatic access to public shares/collections is a MUST at this junction. Because, even if the sites are blocked, without even a court order or a proper legal proceedings as required by the Human Rights Convention, then these sites are still available via sync process, which is gaining momentum worldwide as we speak.

Let the powers of evil and their servants know this:

You will not succeed in destroying the Truth. This is simply impossible, even on the purely technical grounds, and under no circumstances whatsoever it will be allowed by the powers beyond visible, watching and assisting this planet in all the ways possible, conceivable or not. Now has come the summary time to pay the bills and everyone shall pay in full measure, regardless of anything, with no "discounts" of any kind this time around. Because it is a summary moment. The time has come to end this all pervading evil on this planet and to restore its pristine environment and its true treasures, which will be made available to ALL, and not only to some "elite" or the "banking mafia".

The ego-centric system of parasitism and exploitation had utterly exhausted its potentials and its inherent destructive nature can no longer be allowed to continue or propagate.

Time to start thinking?

Now, considering the utter stubbornness of the developer "in chief" as to implementation of a concept of a public share, which is a must in our opinion, and in combination with some issues as to the very nature of this project takeover, it might be not such a bad idea for the developers interested in the progress of this project in genuinely open source spirit to start considering the alternative paths to further development, since from now on, whatever they contribute may not bring the results they hoped for when they joined the project.

And "the bottom line" is that either this project grows as truly an open source and non-commercialized and not controlled by some individuals with their hyper-inflated egos, dominating over it, or some other parties, or it will fade away into the lands of obscurity, where it essentially belongs from now on.

In our opinion, either the program supports the public information sharing functionality, or it has no future, at least from the standpoint of information distribution to the general public. Simple as that.

Because the information distribution needs of the modern world are orders of magnitude more significant than any individual needs to sync one's laptop to the mobile phone and his desktop and a server, or a chip in his shoe. That is all nothing more than "spinning the wheels getting nowhere real fast", or, in other words, the "karmic convulsions" of the unconscious and unaware.

Because the world is going through the most profound changes and every moment now counts more than centuries in the past, and everyone will have a chance to see it in quite graphical terms. It is just a matter of time, maturity and willingness to truly investigate the issues.

Witch hunting:
Mutilate, delete, deny, kill and ban all those who disagree with your self-image of grandness!!!

And here is the kind of things these cunning cowards aka dictators do:

You can't log in until Oct 16, 2024. The reason you were suspended: pattern of negativity, author of website of hate speech

Ban for 10 YEARS? For doing what kind of thing specifically?

Disagree with dictator? - Guilty of "hate speech"!!!

Let us start looking at it starting with probably the most significant "reason", that might have some far reaching consequence: "author of website of hate speech".

Well, WHICH "hate speech" site is he talking about? HOW did he learn of such a site if we never published any site links to ANY site that even remotely has to do with any kind of "hate speech"?

And if you look at the very definition of "hate speech", it is quite something indeed. First of all, there is a notion of a "protected group" and "hate speech" is saying something that will create problems of all kinds to those "protected groups". So, some "groups" are EXPLICITLY protected via this perverted trick. But others are not. Why is that? Do we live in a world of "democracy"? Why certain "groups" are "protected" better than others? What is democracy then? And if you look at the list of those "protected groups", it might blow your mind. You will find sexual perverts among them, various Nazi kind conspiracies, hiding behind the backs of plain ordinary people based, for example, on nationality and so on.

But if you are a mere "mortal", then you are not offered the same "protection". Only the "elite" of some sort is protected by it. Then the question arises: but what is fascism in that case? Isn't the separation on "special" and not so "special" classes of people is in fact the very root of fascism and Nazism in particular? Why some groups are protected and others are not?

Basically, this "hate speech" trick is pretty much the same trick as to classify someone as "extremist" or even a "terrorist". There is a system of zombification of human consciousness utilizing the so called "slide response" technique developed by the CIA. The essence of it is really simple: you just repeat the same lies ad nausea and label them with some negatively charged label. The more you repeat, the better. Adolph Hitler was told this trick by his psychologist.

So, the time comes, that when some of such labels are merely mentioned, the mind simply "slides" to the next subject, agreeing with the zombie program in ones mind. For example, you see a sentence starting with the words "terrorist" or "extremist". What are you going to do but skip it, subconsciously accepting and agreeing with whatever garbage will follow that label? Or are you going to waste your life reading about some "bad" guys, like "extremists" and "terrorists", again and again and again? How many times are you going to read the same crap that tells you the same thing, and that is that "extremism" is "bad for you"? Or, are they going to tell you something different in that sentence or even an article?

The same thing is with this "hate speech" concoction. Once someone says: "an AUTHOR of a hate speech site", who is going to object or even bother to find out what does he mean and how did he come to such a conclusion and based on what arguments or facts?

Because "hate speech" = "evil". PERIOD. Everyone "knows" it, don't they? WHO is going to even consider the implications? Interestingly enough, these tricks are invented and propagated by the most evil ones there are, the criminals against the mankind, who, like George Bush, Sr. said: "should be chased down the street and hung on the nearest light pole".

And how would this pathetic dictator know who is the author of the entire site? There are no authorship claims on any of our sites, at least for any works that do not explicitly identify the author. With all other works that do not mention the author specifically, the authorship can not be determined, unless it is claimed. And there are no such claims on any of our sites.

And why did he use precisely the "AUTHOR" label? Well, because in THAT case there can be no doubt that this is HIM who is "evil". Otherwise, if you say "the one who runs the site" or "serving the site", then it isn't as convincing, and many people may simply laugh at such pathetic accusations. But the "AUTHOR"? Well, it is "serious business" now, even though he did not explicitly state "the author of hate speech", published on the "hate speech" site.

Well, thanks for THAT much. We really "appreciate" your style and foresight. Otherwise you could be sued, couldn't you? But even saying the "author of a hate speech site" is by itself deserving to be sued for. Because it is derogatory in the most profound sense of the word, at the very least. In its very essence, this is the very nature of defamation and slander.

Now... Yes, understood, if he would state that some site has some false information then, who knows, may be he does have some point. But then the question arises: what does ANY site in the world have to do with the Syncthing/Pulse forums? Is this a court decision? Was it determined as such by any competent specialists in a proper legal proceeding, as required by the Universal Declaration of Basic Human Rights in articles 10, 11, 18, 19 and 30?

And what is "hate speech" in essence? What are the EXACT examples of such a speech on any sites we have?

Basically, to even conceive of carrying the "hate speech" on your site is nothing more than lunacy. Because mere hate does not resolve any issues, does not solve any problems and does not clarify anything. Hate is just that, hate.

First of all, the sites we run are some of the most prestigious sites in the world, just to be blunt. And they are visited on a regular basis by thousands of ALL sorts of people every single day, including the organizations, educational institutions, local and federal governments, military and even the intelligence agencies, as strange as it might look, not even mentioning the world's foremost researchers and some of the most well known writers and bloggers of the world fame caliber. And the kicker is that they are visited on a regular basis even by the employees of the major search engines worldwide, and for those guys to visit some site, that site must be "way up there", in the sky, as far as quality of information goes.

Basically, our sites are considered by many to be the REFERENCE grade sites, and they contain the works of the outstanding and world famous authors, researchers, historians, scientists and so on, and there are so many tens of thousands of links and all sorts of praises to our sites that it would be nearly impossible to even begin to enumerate them all, even though we could, if we had nothing better to do with our time but to brag about how "great" we are. What does it matter to others or to the Truth of it?

Nearly everyone thinks he is "great", this way or that, doesn't really change anything of substance. Telling someone "I am great" would, most likely, be taken as an offense and a reason for some kind of fight, just to "prove" that MY ego is "better" or "greater" then yours, even though any ego stinks in pretty much the same way and merely offends others. What is so new or "special" about your "greatness" and what does it worth or "buy" you beyond pleasing your pathetic egos, forever craving for attention as a result of the inferiority complex, hidden in the depths of your being, well below the surface of your mask of some "persona grande"?

Secondly, these sites stand at the very top positions of the search engines worldwide, and to get into the top slots on Google, for example, you have to be quite a site. People are willing to pay some serious money to get into the top slots, and we did not pay a penny to anyone. Because Google search engine has over 500 THOUSANDS rules and variables that analyze your pages unlike you could even dream about, including the AI technology to actually analyze your content for meaning and significance, even though the very idea of an automated content analysis isn't really worth much to begin with.

Thirdly, these sites are some of the oldest sites on the web. Not exactly the very oldest, but old enough to be considered the stable and reliable sites to be used as a reference. And we have never received ANY kinds of complaints from anyone regarding anything, with one exception of some cunning dictator coming to tell us that we should not refer to one of our sites from our Osho site.

Well, we did cover his bitching, and on such a level of detail and precision that it became a book, and that book is regularly read in numbers and it became one of the best books on those specific subjects, and it is being read on a regular basis even after 5 years since its original publication.

Since our public answer to his totalitarian bitching, we have not heard from him ever since in the last 5 years. Because he was disgraced to the degree that he probably lost his "client base" in his horseshit peddling business of a "group leader" in the "spiritual growth" business, zombifying the minds of his victims with the ideas that he KNOWS something they do not know, which is not even pathetic, but simply laughable. What a fool? To waste ones own life on trying "to make a buck" on people's confusion as to their own worth, dignity and creative abilities and potential.

And in fact, we are grateful to him for his bitching in totalitarian style. Because it gave us a chance to cover not only the concoctions he brought up, but the entire "spiritual business", and to go into it on a much deeper level and expose his entire utterly dishonest manipulation of the facts in order to achieve his rotten agenda of a plain, ordinary conman with totalitarian tendencies, a parasite essentially, sucking the life energy of blind people and leading them astray, and not towards some kind of "enlightenment", as he claims. As it was said: "Blind, leading the blind, shalt fall into a ditch". Simple and clear as a bell.

Actually, this ban is pretty similar to that guy's accusations and fabrications. The same kind of people, doing the same kind of things.

So, which "hate speech" site this pathetic dictator is talking about?

But the "thousand dollar question" is: but what does ANY site in the world have to do with the Syncthing project, especially if it does not even mention it? Yes, sure, if he would claim this "hate speech" argument on his FORUMS, then it is another matter. But then he would have to provide the specific evidence of "hate speech" and explain the mechanism and principles that allowed him to come to such a conclusion. Otherwise, how could such an ugly accusation stand and what does it mean in reality and in relationship to this project?

The "pattern of negativity" trick

As to the "pattern of negativity" argument, yes, indeed, that blanket and a highly negative notion is definitely a nice trick in itself. To control and dominate others, putting up the mask of "positivity", which is as false, as a three dollar bill. It makes one SHRINK in guilt! Works like a champ. But not in our department. We shred this kind of stuff to dust, to fine powder, every bit of it.

But there is a "but" here, at least with this particular dictator, aka "developer in-chief". The thing is that if we post nearly every single one of his own responses to purely technical ideas, proposals or pointing out the shortcomings, that were not meant to do any harm to the project in any way, shape or form, and in fact, just the other way around, were meant as a help and a contribution to it, then he should be simply "chased down the street and hung on the nearest light pole", just like George Bush, Sr. said:

"If people knew what we have done to America, we would be chased down the street and hung on the nearest light pole".

Nearly every single response of this dictator to purely technical proposals was utter denials and the most poisonous intellectual insults of despicable grade, camouflaged in some candy wrapper of "grandness" and "know it all" attitude. And the most significant and critical issues were simply ignored or denied in bulk without any valid argument or consideration of its meaning or significance, which in itself is an insult to a senior developer. Because it implies that the arguments are so dumb and incompetent, that they do not deserve even a response! Here is just one example:

Virtues and blessing of "revolutionary" Go language

calmh (Jakob Borg) — 2014-06-01T14:49:50-04:00 — #5

alberto101 said:

Actually, Java is probably a better choice, especially from the standpoint of code clarity, synchronization and portability issues.

So, all the critical code is already there in open source version.

So, build something awesome on top of that then?

Comment: WHY NOT? Or you are compelled to carry around your image of "greatness" of the re-inventor of the wheel?

Why do you need to waste your creative energy on things that have already been done and are available in the public domain? What is so "revolutionary" in your work that couldn't have been done with already existing piles of public source code?

Look, sorry for the curt answer here, but I get kind of weary of the "I haven't actually looked at how syncthing works or what it does, but here's what I think you should do instead" kind of posts.

First of all, what is noteworthy here is that he did not use a SINGLE "bad" word in his highly offensive response. Everything looks perfectly "reasonable", at least on the surface. But deep down, it is such a poisonous hate, full of such profoundly negative connotation, that one could hardly come up with something more disgusting than this arrogant, self-boasting concoction of utterly unsubstantiated crap, and of the LOWEST grade at that.

First of all, there isn't and can not possibly be even a shred of anything "revolutionary" in sync as technology, syncthing or BTSync or anything else. Nearly the whole thing is intuitively obvious, and there is simply no need for any miraculous mechanisms for it to work. Any more or less experienced developer has seen the permutations of what it does many times over in his experience. And he knew at the moment of that post that he is dealing with an experience developer.

Why not reinvent the wheel?

What he is saying here is "if you are so smart then write your own program". Well, fair enough. But the thing is that there is no need to reinvent the wheel and start writing something from scratch when it was already written and is available.

How much of your life are you willing to waste spinning wheels for nothing, just to achieve pretty much the same result, as has been already achieved in some open source project? For what? The base is ALREADY there. All you need it to make sure it all reconciles and has powerful and flexible enough architecture and is stable, has a predictable behavior and assures the consistency of data as far as sync operation goes, and has a fair level of performance. That is ALL you need with this particular project.

Yes, this is the key to understand all this: "all the critical code is already there in open source version." What are you wasting your creative energy on, even though, certainly, you are free to spend it on ANYTHING you like? Reinventing the perfectly working wheel?

And what did you reinvent that makes it even half as good as what BTSync has or does? Actually, BTSync has one major and the most significant problem: it is not an open source. All the mountains of bugs and problems could have been fixed a long time ago, if it was an open source project, and the program would probably be much more powerful than it is right now, if you could have the external development resources of the open source community contributing to the project.

But they want to dominate the "indie promotion distribution market", just as this Pulse thing is trying to do as of now. Strangely enough, they seem to be after exactly the same market as BTSync. It was pretty interesting to see. Just do a search on http://ind.ie and you'll get the idea, unless they have removed some information which we have seen on their sites.

Now, could HIS own response be interpreted as some kind of a "positive" or constructive opinion or a feedback on quite a reasonable opinion of the language?

Portability argument

The thing about Go is that one of the MAIN arguments of this particular developer-in-chief, when asked why did he choose Go, was that it is so portable that you don't even have a "single dependency".

But that argument does not work with Java. Because Java is probably the most portable thing there is. Simply because it operates on the top of JVM (Java Virtual Machine) and it runs on so many platforms that they claim that it currently runs on 3 BILLION devices, down to your fancy super duper bread toaster, not even talking about your "smart" refrigerator.

Secondly, does one have a "RIGHT" to like one language over another? Or, does he think that unless you write in Go, you are "outdated" or "out of fashion"? But does not he know that fashions come and go, like a wind and rain? Is it "OK" for me to like the vanilla ice cream and not to like coca cola?

Dictators know better... Always!

And what kind of an ugly concoction is "I haven't actually looked at how syncthing works or what it does, but here's what I think you should do instead" Yes, "shuda wuda" is certainly one of the favorite words of dictators of all kinds and this one is not an exception.

But this is nothing more than a fabrication, and an ugly and highly insulting one at that. Like there is some miraculous mechanism in syncthing that does things no developer has seen in his life.

Basically, we could present the overview of the entire sync process, regardless of whether it is done in Syncthing or anything else, without even looking at the source code. Because all the basic mechanisms and subsystems will eventually resort to doing some very specific things, that are logically implied by the mechanisms and higher level logic design. There isn't anything "special" in Syncthing, unless the author simply went "nuts" making things more complicated than necessary.

Sure, any program may utilize its own tricks to do the same things that other programs do, only slightly differently. But that does not invalidate the entire logical operations of the high level logic.

And yes, there are some quite tricky issues specific to sync operation as such. Most of it has to do with node update notifications or requests, maintenance of the modified file lists and ways to verify the correctness and originality of a file version, ways to inquire or inform about file updates, block locking in case multiple nodes have the same block that some node needs updating, node discovery, various hashing and encryption algorithms and methods and things like that.

Yes, there are plenty of things to be done. No arguments about that. But NONE of it is something so esoteric that a fair level developer will not be able to conceive or comprehend it even without looking at the source code, unless one deals with some specific and tricky issue that could be done in different ways to either increase performance, efficiency, reliability the soundness of the very model and so on.

Moreover, the question arises: how could such a great developer as our developer-in-chief even conceive of designing the sync program with vital design flaws that allow any user to obliterate the whole share/collection, not even having a bad intent, but merely not realizing the consequences of some actions on the files he intends to perform, just as we see in Syncthing? It seems to be so basic, that it would be interesting to know what pushed him to allow for such a fundamental oversight as to crush the entire share? Such things should not have been allowed under ANY circumstances, no matter what is the intent or a purpose. How could it be allowed for the user to damage the share without even realizing what he is in fact doing?

We did propose quite a number of things to improve the program, its functionality, use, logging and dynamics reporting abilities and performance. But nearly all of it was simply flatly denied, and in the most insulting and arrogant manner, with one exception - a suggestion to comment the source code, which has already started to happen, at least with the latest updates to the sources. So, we did achieve at least some progress with this thing.

And that thing alone, when and if is fully completed, will make this project much more manageable and will help any other developers to understand the program workings and things to watch out for, and probably unlike anything else. So, for new developers that decide to join the project this would be of great help in clearing up some misunderstanding of the program operation and internals.

So, instead of being grateful, this arrogant, self-boasting fool starts barking at us, denying nearly every single point, idea or proposal we have brought up. For what? What could it possibly achieve at the end? To show to the whole world that he is "smarter" than us? OK, fine. But will it help the PROJECT? In WHICH way?

Does he at least have the slightest clue about who is he barking at with all his futile and childish "arguments", if you can even call them such? Because one can hardly find a single real argument in his wholesale insults addressing the issues in ANY of his replies. Nearly all of them essentially say: there is no such a problem, screw you, or these issues are so insignificant that there isn't even a thing to even look at or consider as something real, something that needs to be addressed.

One other fool from his team even went as far as to ask for an example of "use cases" for the public sync applications! Question arises: does he actually have a functioning brain? How could one even conceive such a question? It does not take more than a few seconds to come up with a pretty fat list of public applications. The whole net is public applications for the most part. Interestingly enough, he asked this question even after we posted a number of the most detailed articles addressing all sorts of issues and aspects of the whole thing.

Java could not be possibly a better choice because it is heresy!

And where did it say you "SHOULD" do this or that? The post merely said:

"Actually, Java is probably a better choice", and the reasons were provided: "from the standpoint of code clarity, synchronization and portability issues". Is it "OK" to have ones own opinion? Are these arguments valid, by any chance, or is it all merely a bluff? Can you PROVE it, Mr. Smart?

Does he at least know Java to any degree of competence to make a comparative judgment? At least we did look at the Go main documents that explain what it is and cover some of these issues and its design philosophy. Well, like anything else, there are some points that seem significant, at least from the preliminary look. Yes, certainly, we are not the experts on Go, by ANY means, and do not have the slightest desire to become such. Simply because we have an urge to puke when we are looking at that code. It is simply nauseating, so upside down and so childish some things look to us.

Go language revolution - turn everything upside down!

What these language designers seem to do is to begin to "revolutionize" that, which is already accepted and has a tremendous base of developers behind it. And if you look at some of those "revolutionary" language design ideas, about the first thing that jumps at you is that, yes, there IS something in it, but why should I scramble my entire knowledge base, which works fine and has been validated with time, to follow yet another way of doing the same essential thing?

What is the need to turn everything upside down and create all sorts of notions that fundamentally do not "revolutionize" anything of substance?

But hey, understood, to YOU it may look like some kind of Bible to be worshiped or make you feel like a "superman" of some kind, who is capable of comprehending a new "breakthrough" in languages that will rid the world of all evil, and that is how it looks like.

But if you look at it on a deeper than skin-deep level, then what you might find is that, yes, some things do make sense. Except the same exact results could be achieved in other languages by merely writing a wrapper or a library that does exactly the same thing, if it "buys" you anything to begin with. Why does one need to waste energy on learning something that does not change anything in any fundamental way?

There isn't a shred of anything radically new or "revolutionary" in all this Go trip. The same thing happened with Lisp, Forth and even Python or Ruby at some point when they just came out. And in fact, any of the above languages have many more "revolutionary" features or concepts than this entire Go trip, at least as far as doing things in utterly different ways. Just Forth alone, or even Lisp is quite a trip in itself. And even Python was considered at some point as something that will make the life of a programmer much easier. But, for some strange reasons, none of those languages were able to gain the wide popularity and support, even though some are using them even to this day. You won't find more than a few full-fledged applications written in them. But what did it change fundamentally?

My language - RULES! And yours - sucks!!!

The same thing happened with Java a while back. It was such a hype that at some point nearly all the bookstore shelves were filled with Java this and Java that books, like you can not exist without Java. Where have all those books gone now? Fair level Java programmers began to behave like some kind of "gods" or "super-programmers", and would talk to others like to "infidels" and "inferior" beings, exactly as we see it here with this Go trip. The same exact crap, only the labels have changed.

And his post was not merely "the curt answer". It was an ugly insult, WELL "below the belt". One can hardly concoct anything uglier and more insulting under the circumstances. And it was the language of a bully, an arrogant dictator, by whom everyone is to measure, at least in HIS twisted mind. He was basically just plowing you out of the way or running over you like a tank. Did he, by ANY chance, expect that one would bow down to him, like before a king of some sort, and ask for his blessing?

Yes, we hold our opinion: Go is profoundly ugly as a language. There are several major violations of accepted and highly effective lexical principles, at the very least. The ideas of dynamic variable typing and scoping have been beaten to death for generations, and, one way or the other, you will pay for it. Because the data types are inherent, down to the level of a CPU.

You can't store a floating point number into an integer register and vice versa. Nor you can store a string into an integer register or vice versa. Sooner or later, at some level or another, there is an inherent need to do the data type conversions, and could be many more times than is necessary. All the dynamic typing, scoping and instantiation of variables does is to introduce the processing overhead and brings in the performance degradation into the process, at the very least.

But... No matter what they have invented with data types, they still have them. Why, one wonders? If they are as "automatic" as they seem to claim, then just forget about the data types and pump any kind of garbage through your smart typeless pipes or "channels" for processing and let that "smart" programming language detect it in real time. Why not? Why even keep the concept of data types on a syntax level?

Yes, the idea is great, just like the ideas of some paradise or utopia. But this idea has not gained any acceptance for some strange reason. Why? If it is so "great", why would vast majority of programmers "suffer" unnecessarily dealing with data type issues when they could have an instant orgasm with dynamic typing and scoping? Are they merely stupid? Or backwards? Or do not really comprehend the benefits of some genuine "progress" in the programming languages?

And, interestingly enough, in Java they did address these very issues and did achieve some improvements, at least compared to C/C++ idiom. But they were smart enough not to go so radical. As a result, it all simply makes sense, and without much fuss about it or some radical "revolution" about data types as such. They were able to eliminate quite some nasty issues, such as, for example, the whole notion of pointers and pointer arithmetic, and you can easily see the benefits of it in significant simplification of the source code and increased clarity as to what is what and behaves in which ways under certain circumstances. In Java, you won't find any of this pointer funk or even mentioning of references with their most bizarre and utterly unintuitive behavior and syntax.

One of the very few "improvements" of Go is that they got away from the concept of generics of Java or templates in C++, or inheritance and some other "object oriented" hype, which is a great idea. Because all those ideas are mostly designed for those programmers that want to LOOK "smart" and do not deliver on the alleged "benefits" they promise. And that is precisely why plain, ordinary, "vanilla" C has at least 10 times more stable following than any other language out there.

But I'd rather do it in Python

Interestingly enough, we have seen somewhere on forums the message from one guy who said something like: well, I'd rather do it in Python. And that is really an insult to such a "great" language, if even a Python looks more appealing to some people. What is it worth then?

Otherwise, even starting with Java, they'd get away with variable types. Java is probably one of the cleanest and best designed languages out there, and for so many reasons, that it would take a pretty fat paragraph to even enumerate it all. And, interestingly enough, they did not radically redesign the language notation and syntax and tried to keep the language as close the the familiar idioms, as possible. So, reading Java isn't much more difficult than reading C++, for example.

Any programmer can understand the most of it without any effort. And they did get away from the concept of pointers and pointer arithmetic and quite a few other pretty obscure and "esoteric" aspects of C++. In fact, Go seems to have borrowed some ideas from Java in respect to the syntax and notation, except they decided to "revolutionize" the language notation, which is nothing more than bluff, creating nothing more but pain on the neck to millions of professional programmers, unless you spend months or even years getting used to living upside down.

In Go, everything is upside down compared to what the experienced developers are used to. In that respect, reading Go to them is like reading Chinese or standing upside down. For WHAT? What does it buy you, but merely confusing things even for the people that might have spent their entire lives as developers? Why do you need to invalidate the whole world and human experience that has been gradually growing and developing and was tested by time? Does it radically change something? Does it make the language more efficient? Increases code clarity? Makes your code more robust? What does it actually achieve at the end?

There is quite a bit to be said about the language syntax, except we'd have to write a few paragraphs to clarify it.

Screw accepted language notations

Now, the "punchline" in Go syntax is that they refused to use the very concept of a look-ahead token on a lexical level, and, therefore, they can not properly parse an opening curly brace in a block of code, unless it continues on the same line as the conditional statement. This is simply bizarre. First of all, it implies that half of the programmers in the world that use the left curly brace at the beginning of a line and not at the end of the line of the statement itself, will feel nausea when looking at such a code. The reason people like the left brace at the beginning of the next line is that it allows them to very quickly match the braces visually, by merely scanning the code block at the same horizontal position, and, possibly, find some otherwise hidden errors in case of braces misplacements.

But with the brace at the end of the line of the statement itself, it can not be easily noticed. Your eyes would have to jump all over the screen if you wish to have a rough idea of your braces matching correctly while looking at some block of code. And the end of statement could be on virtually any horizontal position on the screen. And this is done on a subconscious level as a "background job". So, you do not actually waste any additional energy on scanning the entire screen like a madman or a monkey. And mismatched or incorrectly placed braces may and are likely to cause the most profound logical misbehavior of your code.

The argument for it is that you know about it by the sheer fact of code indentation on the next line, is utterly invalid. Because code may be indented, but the brace is missing on the previous line and you may think that you are in a new block of code, while you are in fact are in the same old level. How do you make sure it is not the case but by explicitly looking for the left brace in addition to looking for the indentation level, doing twice as much more work than is necessary?

This "breakthrough" alone is enough to turn a half of the world's programmers away from this language and they are not likely to touch it "with a six foot pole". Simply because they have no interest to go insane, scanning their entire screen all day long.

Monkey business: Let your head spin when you look at the source code

Another "breakthrough" in Go is that they got away with a concept of an explicit end of statement like it is done in C, Java and a bunch of other languages. The end of statement is implicit, by the sheer fact that you have started a new line, which is simply bizarre. Strangely enough, they say that the end of statement semicolon is still there, but it is "automatically inserted" for the compiler!!! WHAT? Why?

Well, because, first of all, you create more work for your parser if you don't have an explicit end of statement token. Because it is not inconceivable that you may hit the wrong key in your source editor and both statements will end up on the same line, which you might not even noticed, and, in combination with the nearby comment, this might create a nightmare for your parser.

Because it won't be able to correctly report the exact location of the error since there is no explicit end of statement token, such as semicolon in many languages, at least as a programmer supplied it. So, where is the exact point of error, even in purely logical sense of the word, when you have two statements on a single line? Is it the first statement, which the compiler may not even be able to identify in some cases? Or is it in the 2nd statement, which, and likely, the compiler will not even be able to parse correctly? Because it did not even finish processing the fist statement, as a result of some other things present on the same line.

Then why do they use dots at the end of a sentence in the human languages? Well, to explicitly identify the end of a sentence, which is the same exact thing as the end of a statement in computer languages. Just remove the dots from a human language and try to read it. How long will it take before you have a grand headache?

Question arises: WHAT did this "breakthrough" in languages "BUY" you in terms of language properties, power or anything else whatsoever? Did it increase the efficiency and/or performance? Did it make the work of your parser EASIER? What did it achieve as a net result? What did it "save" you? A single key stroke explicitly identifying the end of the statement, so that, for one thing, the compiler could report the exact place of an error correctly?

What are these "great" and "breakthrough" language designers feed you but a nightmare for every single subcomponent of a compiler and to load your brain with an additional and utterly unnecessary work and make you go "mad" running around the screen with your eyes while trying to match some things or to see where exactly each statement ends? Why does your head need to spin just by looking at some code? Why does it have to be even more complicated than generations ago, while the language designers are trying hard to convince you that it is just the other way around?

Yes, in Go, everything is upside down, starting from notation, lexical analysis, syntax and even semantics. You don't have to have a PhD in CS to see that. Their "explanation" of exchanging the variable/argument names with data types is really quite some piece of horseshit, and of the royal grade. And sure, they do have some point, especially for the "great" programmers that have a tendency to write the most complicated looking code, just to show their "grandness" to everyone else.

It is like: if I can write some code that nobody will be able to understand, but myself, then those, who would have to read this pile and won't be able to understand what it means or does, are the ones to be fired from a job before me, in case there are job cuts, but not me. Well, at least according to the upside down logic. Otherwise, the whole project will collapse if there is no one left to understand my great code. This trip is called "job security", at least in the Silicon Valley.

Screw libraries!

One other thing, which is as bizarre, as it gets, is that it looks like you can not even build the libraries in Go. Just now, we have noticed a message from one guy proposing to separate the Syncthing engine itself from the rest of the code, such as GUI or the controlling logic, for example, which is an excellent idea. Because, if you have an engine library, like libtorrent or libutp, then you can hang on it any controlling mechanism and any native mode GUI you wish and it will work and behave exactly as you desire, which would be a tremendous help to the very project.

In that case, you can implement the most sophisticated higher level controlling logic which will allow Syncthing to behave even in the ways the original version was not designed for. Because most of the problems of Syncthing are purely logical. The engine, or the "worker" part does not change, at least for the most part or in some major ways. The underlying mechanisms are the same, regardless of a specific application and implementation. And THAT is what modular programming allows you to do.

But our great "developer-in-chief" says: sorry, Go is "very ill suited" for that. WHAT? Is this a joke?

calmh (Jakob Borg) — 2014-10-31T02:39:14-04:00 — #2

The current implementation language (Go) is awesome for many things, but unfortunately very ill suited for creating a loadable library. I think a C or Java implementation would be a better base to build on. (No idea about Swift really but I suspect it may have similar issues to Go, seeing as it has garbage collection and thus a runtime of its own.)

Splitting Syncthing into a library and a file sharing tool (Dev)
https://pulse-forum.ind.ie/t/splitting-syncthing-into-a-library-and-a-file-sharing-tool/1277/2

It would be interesting to see someone, who claims his language is real, except you can not build libraries for/with it!

"Lord, SAVE me"!!!

Actually, when you begin to think about it, it does make some sense. Why? - Because Go and the entire concept of "net computing", pursued by Google and their threats to slaughter Microsoft, and all their promises of "trans-continental" and fully network computing makes the very concept of a library utterly unclear. What if you have a browser "plugin" that might require some library?

Because with Go you don't even know what and where you run. How do you know what you run and where in case your GUI is browser based? So, the program itself may be on one continent and the GUI is on the other one. So, on WHICH end would you say you need to use some library or a plugin?

Well, this mental exercise certainly looks quite funny and unreal, but we would not be surprised that at least a part of this argument is applicable to Go, at least to some degree.

"The bottom line" is that the arguments as to Java vs. Go are not invalid, by ANY means. And if we look at more complex issues of the language, such as synchronization, exceptions and a few others, then Go simply looks childish with its blanket ideology and methods. And if you merely want to replace one thing with another that is doing essentially the same thing, then you need to consider the "bang for the buck" issue. Just making a lot of bang isn't necessarily what makes the real tangible difference.

Revolution in synchronization

What they have with "channels", as a synchronization mechanism, can be implemented in pretty much ANY language by simply wrapping up the existing synchronization primitives into a wrapper to make it behave like a channel. There isn't any "miracle" in that mechanism.

A while back, there was one computer scientist in France who argued that the very idea of some miraculous language that will solve all the problems and will be able to even write some programs in an automated fashion, is nothing more than utopia. Because every system is unique and it has to be painstakingly crafted out of subcomponents and plain, ordinary logic. There exists no general purpose "do it all" mechanism, in principle. And they have been battling this argument for generations and invented all sorts of languages since then, but to this very day, they were unable to disprove his argument.

Every language has its flavors and its own ways of doing the same things, and if you look at any of them, then, sure, it can do some things better than others. But what is interesting here is that the most active Usenet group among all the languages happens to be C group, not even C++ or Java or even Python, as popular as it was and is, at least for some things. So, it turns out that plain, ordinary "vanilla" C is orders of magnitude more usable than any "advanced" language out there. And it does not even have threads, synchronization, or any other mechanisms and bells and whistles? Why would that be?

Sure, if you study their mechanisms in more detail then who knows, might be there IS something to it, that is not as obvious as it might look from the first glance at it. Except the issues of synchronization do not necessarily reduce to the issues of serialization. Those are not quite the same animals. No matter what "channel" do you use to serialize things, it does not mean that you do not have the synchronization problems on a higher logical level, or even on the parallel level. It all depends on what you synchronize with what and in respect to what. There isn't a general purpose mechanism that covers all the cases conceivable. Otherwise, it would have been implemented generations ago.

Synchronization is something "natural" to Go

So, to claim that synchronization is something "natural" to Go, and is way more powerful and universal than anything out there, as this developer-in-chief claims, isn't quite the case. Yes, certainly they have come up with some tricks that make certain things easier, hopefully, but so is with nearly every other language.

But synchronization is rather a subsystem or a resource access protocol in itself, and it is highly contextual and needs to be crafted and well thought out as a whole, system-wide. Else, you might be unnecessarily blocked most of the time, if not getting into the blocked states for prolonged periods of time, while you could actually do some other things. When you have multiple threads running, that could be on different levels of execution, and you need to synchronize access to a number of different resources, depending on a context, regardless, then it is doubtful that there exists some magical trick that will cover all the specific cases conceivable, even though, sure, you may be able to simplify some things.

That is why Java, for example, did not even attempt to take some radical steps in language design and introduced only some additional mechanisms while retaining the synchronization primitives, familiar to all, without any fundamental changes.

Exceptions issues and error recovery and reporting

Let us look at the exceptions mechanism in a much more detailed way. Because this is one of the keys to reliable programs and some aspects of it might not be apparent and the benefits of exceptions could be easily underestimated, especially from the standpoint of error recovery, meaningful error reporting and the overall stability of a program.

The authors of Go go as far as to claim that the code with exceptions is actually BIGGER and MORE complicated than without them, which is simply bizarre and utterly invalid. How could the code without conditional logic be bigger than with it? Even if you consider the catch clauses, it does not necessarily mean that all of them need to be handled with any additional code, and if they have to, then all the code in catch clauses is meant to either do some cleanup work for resources release, or to provide some informative error message, which has to be handled, no matter what, exceptions or not. Otherwise, what is that code for?

Exceptions are nothing more than "long jumps" essentially, that simply redirect the flow of execution. Nothing more than that. Except that there might be a stack pointer adjustment that puts you in the correct place in a routine that catches the exception. Exceptions are like a fancy branch instructions for the most part. There isn't any additional complications in your code because of them. NONE!

Claiming that the code with exceptions is bigger or more complicated than without them is essentially claiming that checking for return codes to detect the errors takes an additional work. Yes it does. But does it meant that you do not handle the error conditions? HOW? And in fact, that is exactly what exceptions are for. Because they allow you to avoid checking the insignificant errors in the context of what is happening and bulk it all in the exception block. But you are guaranteed to handle the errors, while with return codes it is much easier to get lazy and not handle some seemingly insignificant error conditions, and that is precisely what breaks the programs in most cases. Lazy or sloppy error checking or handling.

Furthermore, it is precisely because of exceptions the mainframes are known to be orders of magnitudes more stable than even Linux, which is probably the most stable thing the "mortals" have a chance to deal with. I have heard from one guy who was a system programmer for the IBM, that the exception mechanisms in the IBM mainframes are so sophisticated that they are even more complex than the program itself. They are effectively chained and re-handled on the way back to the main working routines. And that is precisely their power in our opinion.

Very often, if not in vast majority of all the cases, the error reporting value is highly underestimated, if not ignored entirely. As a result, you can see it in nearly every program out there. Something did not work, but what you see as an error message is totally uninformative. It does not tell you a single thing of real value that helps to identify the problem. In most cases, they do not or even can not report the correct file names. Because as soon as they encounter any error on a deeper level, they simply dump any crap that they know about on that particular level of execution, which, in vast majority of the cases, can not possibly tell you which exact file, being transferred from which node to which other one, had a problem. About ALL they can tell you is that some block read have failed. But what does it tell you of any value? How can you identify what did it have to do with? In relationship to which file? During which higher level operation? What was the source and destination in the logical operation sense and not on the level of O/S nuts and bolts?

If you do NOT use the exceptions mechanisms, then you'd have to check the return codes from nearly every function call. Because anything can go wrong, down to seemingly simplest operations. That is why nearly a half a well written and reliable code does the tests of nearly all the return codes, and only proceeds if the results are OK. But the thing is that from the standpoint of errors and program operations those intermediate things are not necessarily significant overall.

So, if you do not use the exceptions, the program logic related to constant checking the return codes easily gets so complicated and so confusing that sometimes you can not read that code, at least easily. Because it is full of if/else and unnecessary while or for loops that may be trying to redo the operation, and, possibly, even go to sleep doing that. Because the resources might not be available, at least temporarily, or they may be blocked by some other thread.

If fact, one may easily claim that the code without exceptions, as a rule, is more complicated than necessary, more convoluted and less efficient, simply because there are unending tests all over the place, and every test eats the machine cycles away. Every jump is the most expensive CPU operation there is outside of function calls and associated argument pushes/pops and expensive arithmetic instructions.

But what IS significant is that a particular routine did execute correctly, and if it did not, it is not an absolute necessity to report which exact called function did not work or even take any further action. The caller may implicitly understand that things did not work as advertised and will reschedule it if necessary. For example, if you want to transfer some file in the torrent approach, you have to transfer each block/piece, which has its own hash and its own algorithms to do various sub-things.

But, from the standpoint of overall program operation, you do not necessarily report, or even care to know, that "block number 447 was not able to complete because the connection was lost", or even worse than that, "for unknown reasons". You might not even know to which file that block belongs when you hit that error condition in a low level code, so all your error reporting at that level becomes meaningless.

But about ALL you need to report is that file such and such was not completely transferred from node A to node B because the network or the remote node is down, but it is rescheduled to complete as soon as the net is back. That is about ALL you need to know about that error. Anything else might be nothing more than a meaningless noise that simply saps your energy while delivering no benefits of any kind.

Furthermore, one node may work and perform the transfers happily, while, at the same time, the other node may not work. But if error is not reported descriptively, you might have an impression that the entire net is down!

So, what you do with exceptions is to place some block of code into the exception handled block. But, no matter what this block of code calls or does, it might be utterly irrelevant from the standpoint of the overall operation. So, you may have 10 things called, each of which may result in an error, but the fine-grained analysis is not really necessary, or even helpful to the overall operation, or could be even distracting from more important things. So, you don't need to check on every return code in that block. You just execute any statements in it or any routines it calls, and if ANY of them fail, you hit the exception for the entire block, even though the called routines might have thrown the exception themselves that took you into your exception catch clauses.

But you do not necessarily care which particular routine and at which depth of stack and while what exact activity have caused it. Thus, the exceptions also behave as information gathering gateways. They are like the interceptor nodes through which the traffic flows regardless of what is happening under the hood and in which particular place. So, this way you can test for various logical consequences of all the significant operations.

Since in the code with exceptions you don't need to test the return codes, there is no if/then/else logic, filling up the program sources, at least for the most part. About ALL you have is statements that actually do things. And, if ANYTHING goes wrong somewhere in the block of code, the corresponding exception is hit, which can be handled locally, to log some things or free some resources, and then re-thrown to the higher levels with probably a different and more precise exception, that has a higher level logical meaning, and will allow you to handle the situation in the most appropriate way and log the error in a much more meaningful manner, with all the pertinent details as to the arguments to the routine that caused some error.

Because the routine itself is not likely to know the higher level context of execution, and, therefore, its error message will either miss the really important conditions, arguments, or parameters to the main worker function, or will display quite a meaningless error message that will give you some irrelevant low level details that will not allow you to figure out the source or relative significance of the problem in the clearest and fastest way possible.

To achieve the same results without using the exceptions, your code would probably be twice as big overall. Realistically speaking it is simply impossible. Exceptions are probably the only way to "control" the higher level logic from the low level code. Not exactly "control", but at least to communicate something far more meaningful then simply return some error code and then let it guess what was the exact reason for this error condition.

Furthermore, different exceptions may be caught in different places of the higher level code even if they were hit in the same lower level block of code, which means that you effectively have a number of different paths "back home" - to the higher levels, depending on the exactly appropriate logical meaning of those. This way, you can "staircase" the exit path to higher level, where each successive level logs or displays the more general level information about the problem, and at each level you may log the pertinent conditions or arguments, starting from the most fine grained and up to the most upper level of general operation.

Furthermore, from the same exception block you may hit the higher level catch clauses in different higher level subsystems, such as the file system interface, or networking or any other subsystem, at least for the most part, at least if you run in the same thread. In order for you to do the same thing with return codes the number of source statements would probably triple, and at the end you might simply get confused and would loose the control of the whole thing as far as being able to follow the code conditional logic or trying to figure out what exactly has happened by merely looking at a single error message, which could be insufficient for clear identification of the problem.

Go design summary

And we can go like this for a while, combing this Java vs. Go arguments in the most precise and specific manner, and after we're done with that, there will hardly be anything left of this entire Go vs. Java argumentation trip, which is peddled by Google like something that will help to "save" the world form all the evil.

Go, essentially, is nothing more than a salad of languages, such as Pascal, Python and so on and some ideas implemented in all sorts of external libraries, such as synchronization grade issues. And it took probably the WORST of those languages, strange as it may sound. Except in Python they have gone even further in madness and removed even braces around the code blocks and replaced them with mere white space indentation on the next line(s), following the conditional statements.

And yes, they also removed the end of statement tokens. That is probably where Go had borrowed this sickness from. And the kicker is, if in one case you have a line beginning with 4 blank chars and the next line is intended to be on the same indentation level, but starts with the tab character, which is also "legal", then depending on the tab expansion setting in your favorite editor, these 2 lines may look like they belong to the different indentation levels, whereas they are not. And these are not just some "wild" cases that never happen. In fact, they happen all the time, by merely copy/pasting some code from a totally different program, or even a browser window and then thinking that it all looks right, whereas you are now executing a totally different block of code than what you thought you are.

And we stand by these arguments, whether he likes it or not. It does not matter, even an iota, and it does not change anything, at least to the better.

Therefore, the very argument of portability, at least as compared to Java, is utterly invalid.

Sure, you can fall in love with Go. But it does not mean that your blood needs to boil when someone mentions Java, or even Python for that matter, and tells you that he would prefer in was done in Java.

Yes, certainly, it is easy to fall in love with some new language because it is fascinating, at least for a while. But eventually, when the freshness and all the hype around it rubs off, you might not believe how naive were your expectations of that "bright future of mankind" and a "miraculous" solution this new language claims to offer.

Finally, "There are no friends on likes and tastes" they say.

But would you be willing to KILL someone or even BAN them for 10 years, which is probably the sickest joke in the poorest of tastes one can imagine?

Interestingly enough, that ban was put in place after we posted a single message about Java port of Syncthing, where we said that we support that development with both hands, only if it was of any help to that work. Well, was THAT the reason for this "suspension" for 10 years, classifying something as "hate speech"?

:)

Will Syncthing (Pulse) remain open source?

The following post was made on the Synching forums in regard to Syncthing being rebranded to Pulse in relation to new sponsor, which, how we could say it without offending anyone, is "taking over" the former Syncthing project and gives all sorts of promises of the "bright future" to the project including the resource and other forms of "support" to the project.

How is Pulse different to BitTorrent Sync?

calmh said:

There are various operational differences, but the main thing is that syncthing is open source and the protocol is documented and secure. If you care who can see your data, you probably don't want to use BitTorrent Sync. If you don't care because the files are anyway public, then use whatever.

I'd like to ask a couple of questions:

1) Will Pulse remain an open source and Syncthing will simply go away forever as a product name?

2) How can users/developers know that Pulse will remain an open source and/or some parts of the source code might not be closed for whatever reason? In other words, is it GUARANTEED to stay the open source, and for that, you'd have to have some signed agreement which explicitly states so, I'd guess?

Will ALL the parts of the source tree be available?

3) Somehow I see some indications that this conversion has far more reaching effects than it looks on paper. For example, the big players, like Amazon, for example, may decide that they would like to have a lion's share of the indie "bundle distribution" market currently monopolized by BTSync. The whole indie thing looks exactly like what BTSync is after - the independent artists promotion bundle distribution market.

4) Anything said or any guarantees, if any, are not legally binding and may change in the future in any way conceivable unless a clearly written contract is signed by both parties. Furthermore, the claim that "we can all go back (to syncthing) any time we want if we won't like this deal any longer" does not seem to carry that much weight. There are ways, and many of those, to make you do what you are told or "mildly suggested" with a whisper into you ear, and with a smile.

5) The claim that "no money has passed hands" also does not look that convincing. You don't have to actually pass any money to buy someone's soul. You can just pass the company shares according to agreed percentage of the total pie, or even promises of "bright future" for this thing, or simply use the tricks of ego gratification and notions of "grandness", "fame" and "visibility".

Finally, what can I say but what some wise man has said:

"It is easy to buy the souls susceptible to ego gratification techniques".

So, if ind'ie that took over syncthing and changed its name to Pulse has any commercial interest(s) behind it, it is not clear why wouldn't they want to close the source at some point? What would be their commercial "advantage" with an open source solution? Why would they want to have a control over this product if it is already available in open source form and they are welcome to it, unless the license explicitly prohibits it?

And there could be even reasons beyond their control, such as various agencies, providing the "grants" or funding to them, exerting some pressure on them to make a back door for surveillance purposes for example.

Update 1:

As to claim "if you care who can see your data, you probably don't want to use BitTorrent Sync", one needs to mention that this is not the full picture of comparison. It is missing one of the most vital components for public distribution applications. Because BTSync allows for automatic joining of the shares, which, in case of syncthing, could be interpreted as "public shares", when and if it is fully implemented.

So, as it currently stands, syncthing is not well suitable for the public distribution of information where people could start downloading as soon as they get the key to the share/sync folder.

Because syncthing requires manual approval by other nodes and those nodes may not even see your requests to join on their screen or new r/o node showing up on their screen or the share edit dialog, or they may be doing some other things or not even present at the computer at the moment the new node wants to join the share, unless I am missing something here. Because even with this new idea of an introducer nodes, the process of accepting the connections to new nodes is not automatic, but manual and requires user intervention.

Forum thread: How is Pulse different to BitTorrent Sync? (Support)
https://pulse-forum.ind.ie/t/how-is-pulse-different-to-bittorrent-sync/1083/10

Messages to Syncthing main developer about optimized architecture proposal

The following message was posted on Syncthing (now Pulse) forums in response to the post of the main developer, which was his followup on the 18 points of optimized architecture proposal as described here:

Update logic engine design overview

calmh Jakob Borg Developer said:

alberto101 said:

Otherwise, it might look like a "job security" strategy. "I am the only one who really knows how this thing works and, therefore, I am irreplaceable and can not be fired otherwise the whole product flaps".

Actually, I was surprised how popular is this trick among top developers in the Silicon Valley for example.

Well, no, not really. There's no job security angle here. The code is out there; the more contributors, the happier I am!

Well, I certainly hope so. Except "job security" is a much broader notion than it might look on paper. Only time will tell what is real and what is illusion. :)

calmh Jakob Borg Developer said:

You probably haven't seen it yet, but v0.10.2 is going to be the most *amazing* release ever (hello Apple!) in terms of contributors. We've added five contributors to the list since v0.10.1 so far. That's people who may or may not be new to the project, but who have their first contribution added just now. I think it's absolutely awesome how people are picking it up, finding things to fix, and *fixing them*.

Well, that sounds interesting. It certainly does not hurt to have more contributors. But how did they appear in numbers all of a sudden? Does it have anything to do with the change of brand name? I have a strange feeling about it, but it is "good news" for you and the project. Except I would not like to discuss it, at least right now.

So, if that is the case, then I have a proposal:

Since you are quickly becoming someone like a director of development, or could be even the VIP kind of guy, then some additional things could be done...

For example, you do not seem to be under pressure now to "carry the world on your shoulders". So, why not optimize the resources?

So, the next few days you might spend on documenting the code really well, so it begins to look like reading the morning paper. That thing alone would help all other devs tremendously. Because then they'd be able to understand the code, and even could be some subtleties, much faster and much better.

Then you suggest they start studying the code that you have already commented to a sufficient degree.

Meanwhile, you clear your name, in terms of the notion of "scientific truth" by going through those 18 points of mine and addressing the issues. There is no hurry as to when and to what degree of detail it will be done.

Because right now, it looks like your great reputation has been tarnished a bit. Because you kept throwing around all sorts of blanket and quite wild formulations without actually addressing any issued therein, some of which looked pretty close to the point of intellectual insult.

Secondly, you'd provide the contribution to the document, I am working on, with your insights, which could, and probably would, be helpful to quite a lot of people that started to get interested in this work, which I know for fact.

Thirdly, it might help you to get some courage back and take a little step back from the aggressive position and clarify some things for yourself, especially in regard to such things as a need to download the entire index and looking at what it actually helps. Who knows what may come out of it?

Because by merely downloading the index, it still does not assure the data integrity. Even by the time you actually start downloading the actual files from that node, those files might be either different, or there might appear a bunch of new files and even subfolders with a bunch of new files. Which means, that even if you downloaded the entire index, it might become invalid, at least partially, almost immediately. So, what was the point of that exercise then?

So, if you do this, then you do not look like you are just trying to spit into my face and make a fool out of me, who is "running his mouth" without knowing what is he talking about. Not that it matters much to me because I have no need to carry the image of "greatness" around. But you would fulfill one of the major requirements for discovery of That Which Is, which is a definition of Truth, if you care about such things.

Otherwise, it might look not in your favor and might mean that you are simply frightened of the consequences of such a "radical" change in the point of view in the entire project's architecture and design. Yes, I see some things that might cause some serious reconsideration of some things, which, at the end, may turn out a major effort.

But... The good news is that now you no longer have to look like a "coder slave" and can afford some luxury of a more distant view and supervisory abilities.

Plus, you will be able to PROVE that WE are so wrong with those 18 points that there is just nothing in it worth even discussing. Pick the ones you "hate" the most and go at it and get as "critical" as you wish. The more, the better. But the arguments have to be precise and sufficiently complete to comprehend what you mean and why. Those things do not hurt us. Because we have no need to carry the big egos around. They are meaningless to us "in the scheme of things".

You will also be able to prove to me, the others and to yourself, which is probably the most critical aspect, that "lots of words" I produce are merely "full of hot air" and do not signify anything of substance, and are in fact, "tales told by an idiot, full of fury, signifying nothing in the end".

Then you could really start looking at a good place in the world to put up a monument to yourself. Because to "beat" the guy like me and those who stand behind me would be "something else" indeed and would deserve an applause, no matter how you cut it. And also, all those who might be following these things in various places would be able to rejoice in their heart with the idea of "the justice has been done, finally".

Makes sense?

So, you either scrap those 18 points as a hole and provide the "killer" argument, or you invalidate any of them, which you find the most "ridiculous" or unrealistic or are "full of holes". That would help me, personally, and other people that are watching this show, and who knows whom else? The world is big! And once things are born they become alive forever, some say.

calmh Jakob Borg Developer said:

Because god knows there's enough stuff to fix, and all of us who work on it do so on our free time, without compensation, taking time away from family and friends etc.

Well, I already gave you quite a compliment and encouragement in the very first post of mine, if I recall correctly.

And I can repeat: your work is certainly appreciated by many, including those who stand behind me, and that is probably why your project and you were chosen "to take this thing to the next level". You know what we are looking for, don't you?

calmh Jakob Borg Developer said:

Edit: Yes, you have a point - the current code doesn't read like the morning newspaper. If you look at the last few months of commits though, you'll see a conscious effort to both clean up dark corners and document *why* changes are made, in comments.

Well, actually, the last two times I have looked at your code, especially the last time, precisely this aspect just jumped into my eyes. I wanted to rub them, asking: "Do I believe what is see? This guy actually started adding some comments, and some of them are pretty good, and some of them are even in line?", (which is great idea to warn about tricky places and issues).

What to do if it takes so many words to make that happen? But it DID start to happen and the value of it is hard to even comprehend. Because with well commented code you open up the gates of Heaven to your project. From then on, the development could start progressing in such a rapid manner, that it is hard to even believe it could. And so the bug fixes might start pouring on your head like a spring rain.

Who knows?

calmh Jakob Borg Developer said:

I also think some credit should be given to the fact that we've created something that fills a niche and mostly works

There is no doubt in my mind this is actually the case. Except I do not like the very notion of some "niche", which is like a hole to me. Who are you trying to hide in those "niches?". Not from YOURSELF, by ANY chance? Why niche, when you can have that, which I can not even begin to describe to you?

I can tell you this, and I hope I am not too "pushing" here:

Once you implement the concept of a public share/folders, no matter how "difficult" it might look on paper, you'd be doing such a great contribution to the whole of mankind, that I dare not to even start imagining it here. That much of responsibility I am willing to take for making such "grand" statements. Primarily because I see nearly "exactly" what will happen to this project and in which ways it will contribute to some vital things in this world, and these are not just "big words".

calmh Jakob Borg Developer said:

(although with bugs etc). So we're not ISO 9001 standardized in the process, but at least we've *made* something.

Yes, indeed.

calmh Jakob Borg Developer said:

You have to realize that there are very few people working on this project and we have a finite amount of time to dedicate to it. To be blunt about it, arguing about stuff here takes time away from actually taking care of the almost one hundred to-do items we have in the bug tracker at any given time. So at some point, doing stuff is worth more than talking about it.

Well, may be it is time for you to "step back" a little and take a second look at the bigger picture?

Things are rolling now. You are getting new people coming. So, you no longer have to carry "the weight of the world on your shoulders". It seems to me that your contribution in terms of well commenting the code would be many times more effective than the actual coding work you might be able to do in that time.

You can just take a position of "senior consultant" and propose some ideas to other developers and make sure that people do not start fixing things or bringing some new ideas that might break the other things because they do not quite comprehend the consequences.

And if you hate the very idea of a public share/folder/collection, just because it is "too much of a pain on the neck", then why not ask one of the new devs to consider those 18 points and see if they can find a way to make syncthing work with fully automatic joining of the public shares? Who knows what might come out of it? What if someone really gets interested?

Another great thing I see is: what if you spend a few days on writing a design document where you cover the basic ideas about the architecture, its "philosophy" and get into some details on why, what and how?

Combining this with code comments, could mean that the MAJOR work is done for you, personally. And now you can afford to "lay back an relax a bit", unless, of course, you are a restless kind of guy and might even begin to start having the heart problems if someone pulls you by the ear from the screen and coding, which you seem to enjoy quite a bit.

calmh Jakob Borg Developer said:

Killer zombie mutant/hybrid: Deeds not words

Killer zombie mutant/hybrid: "Deeds not words!"
KILL 'em!!!

Well, I am not going to comment on this picture, but if you think that "deeds" could be done with a gun pointing upwards by the hybrid "superman" kind, who has a brain control box on his chest and a brain controlling transmitters on his head band, then sorry, "I am from a different joke".

No, to me this is a shame and disgust that a human being could be reduced to the level of an obedient biorobot, if not an outright killer zombie.

These kinds of images of violence, war and destruction are used extensively by the NWO machine to zombify the human minds and reduce humans to the level of a mechanical machine, killing machine in this particular case. Just ask yourself a question: "what is the need to promote the ideas of death and destructions anywhere you look"? Is it something NATURAL and necessary for Life as such?

Hope this helps.

Are there any "use cases" for public applications and a public itself?

This post was made on syncthing/pulse forums.

jpjp

Any chance you could write a couple of use cases?

Edit: Basically, it is nothing more than an ugly insult of one's intelligence. Because he implies that if I ask this question, being a "developer", that means that you are proposing some lunacy that no one in the whole world needs. So, would you give some examples of using sync for public applications?

Well, HOW MANY do you want to see in order to satisfy your perverted mind?

But what he really implies, without saying a word of insult, is that a proposal to implement the public sync functionality is nothing more than a mere idiocy, and, therefore, the author is either lunatic or an outright idiot, deserving nothing more than to be laughed at. Because who needs it, if even to "developers" it does not seem self-evident?

Nice touch, eh? Yes, indeed, you need to be an utter pervert, to even be able to invent such an ugly insult in such a seemingly innocent manner and in a single sentence, that, on the surface, looks like a "fair inquiry", which it isn't, by ANY means. Otherwise, he is not a developer, but a gorilla, not to insult the gorillas, a plain, ordinary ass licker of the main dictator-in-chief.

Now, at this point, we do have a pretty strong feeling that this particular "development team" will not be able to contribute anything genuinely significant to the project as far as probably the most critical issues and features go. Most likely, they will be doing nothing more than polishing this thing, and even there, it does not seem that they have any interest to even consider all sorts of problems, issues and shortcomings of this thing, many of which had been chewed for them to such a level of detail, that one must be either and outright idiot or utterly dishonest in order NOT to understand what it means or implies.

What is most likely to happen to this thing if even the most detailed, step by step analysis of the issues and proposals "merely fly like a piece of cardboard over the Paris"?

In case you are talking to me:
Use cases of what? Of moving perfectly "on topic posts" to some hole?

Or "proving" the need for a concept of a public share?

I am sorry, but I do not understand the very concept of a "use case", at least if not defined used by whom and for what purpose.

In case you are talking about public shares, the "use case" would be the distribution of any kind of information globally to any users who are interested in it, just like any torrent.

So, the question would reduce:

What is a "use case" for torrents?

The only difference is that torrents are static and sync is dynamic, meaning that if you are a supplier of information and that information is changed, updated or extended, the sync approach would guarantee the delivery of the "latest and greatest" version, while torrents could not do that. Because torrents, once created, can not be modified, updated and extended.

Implementing public shares
https://pulse-forum.ind.ie/t/implementing-public-shares/1186/9

In fact, you might even formulate it like this:

"Use cases for what? - For general public? For libraries? For newspapers? For books? For TV and media? For education as such? And even for zombification of a human mind and biorobotization of the human beings?

Because ALL of those are examples of public applications and sync is probably the best way of delivery of ANY kind of information, and the information is DYNAMIC in its very nature. So, how could one be so dumb as to even comprehend asking such a disgusting and highly derogatory question?

There are dozens of torrent clients regularly present on line. The torrent traffic is probably the major part of the traffic on the entire net.

But try to prove to such perverts that it all makes sense and there are indeed "use cases" for it.

And so, if THAT is what constitutes their "development team", then it seems pretty obvious, even by implication on its face value, that most of the best of developers, that participated on this project were simply eventually banned and you can't see even a trace of them on these "forums". Because it seems that ANYTHING, and quite literally, that challenges the image of "greatness" of the dictator-in-chief, has been either mutilated, moved to some dump, nobody reads, or simply deleted under all sorts of pretenses, such as "not on topic".

Because if you have to "prove" even such obvious things as public applications as such, then what would happen with more creative proposals, suggestions or some critical analysis of the issues of performance, functionality and futures, if any and all of it is flatly denied at the very root?

Then the questions arise: but what IS "on topic" with this project? What is its FUTURE? In which way, shape of form can you develop it any further? What kind of things you might consider "on topic" with this project? What kinds of things and features would you implement next?

Take, for example, one single case. In version 0.10.3 or so one of the things added was an ugly biorobotic face in variations of shapes made out of little square boxes filled with color, next to the node name in GUI. What a breakthrough in development! What was the development effort and time wasted on? Yes, WASTED, LITERALLY! To make one feel like he lives in the world of robots and the ugly pakimons or Simpsons "rule"? Just to remind him on subconscious level that he is not different than a biorobot, a functioning machine? How can one even look at such idiotic pictures without getting utterly disgusted?

And THESE kinds of ugliest things are literally jammed into ones subconscious mind, forcing him to "get used" to the fact that he is not much different than a mere mechanical robot, only a biologic version of it. Because every time you look as Syncthing GUI you see this disgust, even next to your own node name!!! What does ones subconscious mind is to make of it? Is one expected to feel "good" with this kind of "breakthrough" in development? Well, but what happened to Windows 8, but utter disgust with it worldwide and a significant drop in sales, and PRECISELY because of this very ugly model of reducing the world to mere ugly pictures with zombie-like images all over your screen?

And then, starts the usual thing: violence

And then, the last message on that thread was from the main developer:

calmh Jakob Borg Developer said:

I moved 7 posts to a new topic: Implementing public shares

How is Pulse different to BitTorrent Sync?
https://pulse-forum.ind.ie/t/how-is-pulse-different-to-bittorrent-sync/1083/29

Mind you, this individual has moved several perfectly "on topic" posts to some hole that does not even show up in the forums list. And some of those posts he moved were by developers, not counting the author of the proposal, who is the most senior level programmer and a systems architect.

That raises a question: how many other ideas, proposal or attempts to discuss the program problems and shortcomings were simply destroyed and how many developers have just disappeared from this project, dominated by the ego maniac with hypertrophied sense of self-significance in the scheme of things?

So, instead of being grateful for the contributions and proposals of a solution to probably the main and most fundamental limitation of the program, these people would sacrifice even Truth itself, just to protect their pathetic, rotten, arrogant and obnoxious egos, full of crap and hot air.

There is no point to quote the other of his ugly and highly insulting posts, full of hot air claims and denials of the most critical issues of the program. But this is something noteworthy in itself.

Well, what can one say but this is the reflection of the mind of the same man who posted the ugly picture glorifying violence under slogan: "Deeds, not words".

Deeds, not words

Basically, the same procedure as on BTSync forums: deny, insult, pervert, humiliate, followed by mutilate, kill, and finally, the most pleasant thing: ban!

About the only question remains about this is:

How many other posts have disappeared from that thread?

And probably the most significant question would be: what do you think is the future of this project and what may possibly come out of it?

Final message on Syncthing/Pulse forums about public shares and general public applications

OK, we have done our part in what we consider to be one of the most significant issues related to public shares.

If there are any issues or problems you find in what has been proposed, we would certainly allocate some time for it. But, since we do not do coding nowadays, it has to be implemented and tested by other developers.

Yes, if and when the code will be well documented, who knows what might happen.

One of the most critical issues of concern at this junction is the issue of "control". In open source approach any notion of "control" becomes problematic as the process is inherently undemocratic in its very nature. Yes, there is a need to come to some solution that is beneficial to the project and does not cause the instability or impact the consistency of data across the shares, or performance and so on.

But since programmers are not exactly some "street crowd" and are creative and are logicians, then exerting "control" over them, like with some "crowd" could impact the natural growth of the project.

It is our opinion that the need of functionality of the public shares is an issue of a critical grade. So, considering that the set of issues does not appear to represent some overwhelming technical challenge and could be implemented in a few days, at least in the preliminary version, ready to be tested, we hope that this issue will be considered with more attention than it has been afforded to this day.

So, we are done with this thread, unless some specific technical issues come up.

P.S. What does it mean "Dev" in a black box at the top of the page? This thread does not show up in the Support category and so it is not clear how can it be visible to the general public visiting these forums? If this happens to be some "private" place for the devs, this would be highly undesirable in our opinion, at least if there is no access to it from the top level hierarchy of the forums.

Otherwise, it may become an exercise in futility, or some "private club" for the "elite", which is a bad idea in this case. All the messages posted by this author were written for general public and not for some "elite" or "undesirables".

Good luck now.

How to insult without saying a word of insult

The following message is the response of the main Syncthing (Pulse) developer on the above message with point by point analysis of the blanket statement he has made, and not for the first time, by ANY means.

calmh Jakob Borg Developer said:

Sorry Alberto, this is getting too long for me to be able to follow.

[Comment: Well, this statement alone is probably the ugliest form of the intellectual insult conceivable. It implies, first of all, that the message with point by point analysis above is generally nothing more than pure grade garbage, and there is nothing in it to even talk about.

Secondly, it implies that Jakob Borg is such a "great" developer and his time is so valuable in the scheme of world events that he could not be bothered with "clueless", even if they provide a detailed, point by point analysis of his utter bluff about "potential millions of clients" and other equally as bizarre blank statements he had made. Yes, precisely "bizzare", made in utter carelessness and disregard to the very notion of the "scientific truth", if there is such a thing to begin with.]

My point above was that it's tricky in the current implementation. You have made 18 points that I guess argue that it isn't? We'll have to disagree, but again, arguing about it is not the way to get it implemented. Implementing it, is.

Well, in order to have some "point" you need to express it. Just to say "it's tricky" tells you nothing. WHAT SPECIFICALLY is "tricky" and, if that is the case in fact, then why one of the main developers said:

"It's no more than a days worth of work if anyone is up for it :)"

Public share/folder functionality

"We'll have to disagree" on what specifically? Where is your great enlightening insights into reality of how it is? Are you God - All Knowing by ANY chance?

Well, what can we say? We know this position already. But the point of that post and a proposal was to discuss the issues raised by calmh, point by point, and, using them as a starting point, to consider the related issues involved, including the crude model of downloading and even keeping around the entire index from EVERY SINGLE node, and not even talking about the alleged need for the permanent TCP connections. We claim that these things in combination will simply choke this program and will bring it to a grinding halt even at relatively small loads. Are these issues significant enough for yerr royal highness, Mr. Smart?

Yes, certainly, you can refuse even to discuss the issues. But in that case, every point you have brought up, is thereby invalidated. Simple as that.

Else, you'd have to argue your case. Pick any point you like and EXPLAIN the specific details of WHY you think such and such point of yours stands.

Join the herd and attack the messenger

AudriusButkevicius (Audrius Butkevicius) — 2014-10-17T10:38:49Z — #24 said:

@alberto101 I think you should write a book about Syncthing/Pulse.
Every single of your posts is so long it's agonizing to read them, they get sidetracked into something which has nothing todo with what the discussion is happening about.

@calmh says implement a feature if you really want it, and you go about job security on a open source project which doesn't make any money, and general ideology of things.

How is Pulse different to BitTorrent Sync? (Support)
https://pulse-forum.ind.ie/t/how-is-pulse-different-to-bittorrent-sync/1083/24

Yes, SHOULD. That is the only language the dictators and their rear end lickers seem to understand better than anything else.

Well, first of all, regarding the "long articles", this is actually precisely one of the major problems with documentation in general. Because it is written in "one liner" style, that is cryptic, insufficiently described, not covering all the cases and so on.

Well written documentation tells you nearly everything and every single aspect, error condition, various uses and so on. Sure, there is a limit of detail, otherwise you might have to write a pretty fact chapter on nearly every issue out there.

But the idea is that once you read a particular section in a doc, ideally, it should describe anything and everything about some particular case. Otherwise, it is nothing more than waste of time and you have to go find some other information somewhere, waste yet more time and so on.

But you can hardly find a single reference that you can use and expect it to answer at least most of your questions fully, in detail, so the issue becomes clear, even to to not so technically inclined readers.

If you look at most documentation out there, it is simply horrible. Instead of answering questions it merely creates MORE questions.

The same thing happens on forums of all kinds. People write a single sentence in response to some issue, but they do not sufficiently describe things, and their argumentation is usually absent altogether. So, either the issue remains unanswered, or is answered only partly. As a result, there there are followup questions, which lead to yet another one liners that do not necessarily answer it in full.

The "long articles" is precisely the difference between caring and "I don't care"-ing, respecting Life or worshiping and promoting the ideas of death. Because in essence, those "one liner" articles and documentation is essentially sucking the energy of the victims, yes, victims, that are to read those one liners. Because they essentially tell you nothing and clarify nothing, and so they merely sap one's energy.

And so you have to spend days until you finally get an answer to your question or some issue finally gets covered in full. And that is simply a royal waste of time. But if you describe the situation fully, in a single post, then that particular issue does not even need to be discussed further, unless some new aspects are revealed.

Basically, if your TOC is detailed enough and you can find all the issues described in a doc directly in the TOC, then all you have to do is to click that link and go directly to exact explanation of an issue. You really do not need to read things that are of no interest to you at the moment. For what?

Secondly, as far as writing a book, that book is ALREADY written, or, more precisely, is being written as we speak, and you are reading it right here and now, and, btw, this book is considered to be a "Great reference document", even on superuser forums.

BitTorrent Sync - Exclude All Specific Subfolders
http://superuser.com/questions/749957/bittorrent-sync-exclude-all-specific-subfolders

And this is not meant as something to brag about, but just to put things in proper perspective...

Interestingly enough, it is the only article on these kinds of forums that contains a single comment or answer, and that is: "Great reference document: http://preciseinfo.org/BTSync/BTSync_Notes.html#Using_SyncIgnore_file"

And, since that reply, there is not need to discuss it any further. That means that once one looks at that link, there are no more questions left in his mind as to how it works. And that is the best compliment to any documentation that the writer could even hope for, and that is precisely the difference between well written documentation and some cryptic one liners that raise more questions than they actually answer.

There are quite a few people watching it as it develops, strangely enough, at least according to the "smart ones" of your kind. Interestingly enough, it is being watched even by the Google employees, strange as it may sound.

Do you even begin to realize how many people have already read it, and are reading it every single day? Well, probably more than Syncthing/Pulse forums.

"Every single of your posts is so long it's agonizing to read them"?

First of all, it depends on reader as well. If he is just an impatient psychotic, then he won't be able to read anything longer than one sentence. Because the span of attention of most people nowadays is between 3 to 5 seconds.

Sure, if you claim that the posts are just too long and do not cover the issues and do not present any useful or important information and are nothing more than some blabber, then yes, you do have a point and there are plenty of such posts, even if they are not too long.

Question: how do you cover a number of aspects and issues if you do not present your detailed argument that covers the issues from all the relevant points of view? And the issues discussed in every one of those "long" posts are multiple and the consequences are multiple and profound, basically every one of them. Just try to find some irrelevant argumentation in any of those posts? Even if we mention this satanic NWO thing, it could turn out even more relevant than any purely technical arguments. Because the significance of those issue spreads on everything, the very Life as such. So, how can you possibly ignore it?

- Well, then what is the need to read them? Just skip them and go to those posts that make some sense to you. What is the need to make a fuss out of it and in such an augly and insulting matter? Do you, lil rear end licking cockroach, even realize how disgraceful your idiotic and utterly devoid on ANY content post is?

Are you merely trying to make a fool out of yourself? Or show what a big and "significant" prick you are, getting into someone's face with your idiotic insults?

Well, what to do, if you don't seem to be "getting it" even if it is described in the most detailed way, giving you the specific and most precise technical arguments conceivable, and all the reasoning behind them and even their connotations?

The way we write is really simple: once something is written, there should be no need to further discuss or explain anything contained in the writing, simply because everything is described in such a way as to allow even not very competent people to understand any of it with exceptions of some technical details that depend on ones background.

So, once something is written, it then becomes a reference of some kind and once you read something, the picture is clear in your mind to the utmost degree. We have no need to write or even discuss the same things, unless we feel compelled to "drive some point in" since it does not seem to be "getting through".

Side note: Basically, we operate on an energy level. The words that you read are merely translations of the energy clusters which are received by the writer "on this end of the wire". But what is received by him is not the words, but energy. The real communication happens on the energy level. Words are merely the most distorted version, used in the "physical" or "tangible" domain. But they always, and inevitably contain the distortions, no matter how "great" is the receiver, or the "author" of this writing in this case. Therefore, there might be, and as of certain inevitability, distortions of translation of energy into mental structures and constructions, and, finally, into the words that you are reading. That is how it works here in this "department".

So, our suggestion would be: if you still have any issues and are willing and ready to discuss them sincerely and with honesty, then, sure, you are most likely to receive all the necessary "answers", if we can provide any.

But if you decide to "play" with us and/or "challenge" us, this might not prove as the most productive approach, that will benefit you the most, and in some cases, like this one, for example, you might receive the same negative energy, reflected back at you, only significantly MAGNIFIED, and, in some cases of particularly stubborn or extra-egoistic individuals, that negative energy might be not only overwhelming, but may become even become annihilating in its nature. It is like a fire. Except it does not burn your BEING, it only burns the misconceptions, and particularly those that are perpetuated and reinforced with not so noble motivations behind them.

So, "spare your grief" and get creative, once you are dealing with us. Otherwise, just find something more exciting and more interesting to read. There is really no need to waste your creative energy on things that are not interesting enough or challenging enough for you. For what?

And probably the worst strategy for any one of you is to try to attack "the messenger". It might, and is likely, NOT to work the way you have expected, and all the ugly monster-like destructive mental constructions that you have or are trying to send or project out, most likely not even fully realizing what you are doing, are likely to hurt YOU yourself primarily. They do not hurt the messenger to any degree even worth mentioning. Because these constructions are dissolved on his end nearly instantly.

And how do you describe the volumes of issues and design shortcomings which syncthing has at the moment, and seemingly total absence of a good willed intent to even look at probably some of the most important issues with this thing, and that is public shares, performance, stability, predictability of an outcome, consistency and so on?

"@calmh says implement a feature if you really want it"?

How many times is it necessary to repeat that, first of all, currently we are not doing the development work, even if you pay for it, whether you like it or not, makes no difference, and secondly we can not look at Go for more than a few minutes at a time because after that there is a strong urge to vomit?

Secondly, we do not need this feature. YOU do, or more precisely, syncthing needs. Otherwise it has no chance to become something real, at least for the public level applications. Simply because it does not even have such a concept as public applications, and, furthermore, has some fatal grade issues even in some group applications, simply because ANY member of any group can obliterate the entire share/collection/folder if he so desires, and ANY time he so desires.

Basically, the way it is at the moment, it is good only for "play with yourself" applications unless you can trust others in your group as much as you can trust yourself. Tell me, oh master of disaster, how much do you trust even yourself?

Advice message to developers

The following message was posted on Pulse (Syncthing) forums in reply to the main developer.

Basically, this is an advice and a warning message to the developers.

calmh Jakob Borg Developer said:

Sorry Alberto, this is getting too long for me to be able to follow. My point above was that it's tricky in the current implementation.

May be. But it would be helpful to know the issues from the guy who conceived and wrote the whole thing.

calmh Jakob Borg Developer said:

You have made 18 points that I guess argue that it isn't?

The primary reason for those points was that you could pick the one you find significant or relevant and then clarify the issues involved. Otherwise, it looks like blanket statements. I do not see it as "sky is falling". I need to have the precise reasons for things and/or design.

calmh Jakob Borg Developer said:

We'll have to disagree, but again, arguing about it is not the way to get it implemented. Implementing it, is.

Well, YOU can make it easy or you can make it hard.

I, personally, do not work with code that is not sufficiently documented. Because I do not have time to waste hours, if not days, trying to figure out things that could be described in few sentences by those who really understand what and why and explain the overall idea and special cases to watch.

Also, file and/or module naming are not quite something I am used to. I, personally, like everything named in more or less self-describing manner so that just by looking at some file name I have a rough idea of what this is all about. I just went through your code for the 3rd time and spend about half an hour on it, and it did not clarify anything even worth mentioning. In my opinion, every time one looks at the code the more things should become clearer.

The way I do it is that each file has a description header that tells what this file is doing, what is the general idea, what major cases, principles or mechanisms it provides and what it interfaces with.

Each routine has a more or less description header that tells what is it doing in general, who supplies what to it, what are the tricky aspects to be aware of in order not to create some undesirable side-effects and general things to be aware of. This does not have to be a novel size. But just by looking at description I should be able to go directly to precise point I am interested in.

The same thing applies to in-line comments to some important or tricky code. If there is something not that apparent from just looking at the code and may present some tricky cases, it needs to be explained right where it is done. It is common to see things in the code that may look simple but may have quite some consequences if non-obvious side effects not understood.

Furthermore, even if code is written without comments initially, still it should normally take a couple of days for the original developer to comment at least the most important parts of it. Otherwise, it is nearly a sadistic joy by the original developer to torture all other developers by forcing them to suffer reading your spaghetti code.

What is so difficult to explain what you are doing and what is the idea? Why should one spend hours staring at some esoterically looking and, at times, unnecessary complicated code? The point here is that it might take ten times more time to understand some code then it is necessary, if it is poorly documented. And that is a royal waste of time overall. And that time could be spent productively, in joy instead of pain on the neck.

Furthermore, some people deliberately write unnecessarily complicated code, which is a subtle form of sabotage and an attempt to "control the situation" and make one look more "important" and "smarter" than others. It is like "well, if you do not understand my highly 'advanced' code, then you must be dumb, and I am, therefore 'smart'", which is a dumb idea to begin with.

Basically, I like to read code like a morning newspaper and be able to go through the whole source file within minutes. It is not the most pleasant thing in the world to stare at some routine trying to understand what and why, especially if it turns out that this is not what addresses your issues of interest.

[edit: I actually worked for one developer of the world class level. One day he came and announced that he wrote some code to do the analysis on the amount of code comments compared to the total size of the code, and "the kicker" is: he said, the person who has the smallest amount of comments to his code will be fired.]

For the original developer well commenting things is about the easiest thing in the world because only him knows the precise reasons for what and why.

But for developers that do not know your code, this could quickly become a pain on the neck if not the nightmare. Would you like to turn the developers OFF from this project? Or would you rather have as many people getting interested in it so it grows, matures and becomes stronger?

Otherwise, it might look like a "job security" strategy. "I am the only one who really knows how this thing works and, therefore, I am irreplaceable and can not be fired, otherwise the whole product flaps". Actually, I was surprised how popular is this trick among top developers in the Silicon Valley for example.

Hope this helps.

How is Pulse different to BitTorrent Sync?
https://pulse-forum.ind.ie/t/how-is-pulse-different-to-bittorrent-sync/1083/22

Basically, the primary reason for this message was to at least help their project to become more manageable as far as code clarity goes.

In essence, such a flat denial by the top developer of the entire point by point analysis of specific issues raised by him in one of his blanket posts, was beginning to look like an insult of a subtle kind: "well, you OBVIOSLY do not know what you are talking about and so there is no point for me to even bother". This his another subtle insult "well, this thing is getting too long" [for me to bother with]. Implying "I am such a significant guy that I do not have time to talk to the clueless who do not know what they are talking about.

And we have seen this again and again with nearly every "great" developer, at least on sync related projects.

Well, most of these developers are so "great" that it would be appropriate to mention here what Diogenes have said to Alexander "The Great":

"Yes, I understand how great you are. But would you at least move out of the way a little bit because you overshadow the sun for me, and this is a cold morning".

That was his answer to Alexander's query just before one of the biggest battles of his life, when he came to Diogenes and asked him the question:

"Tell me, why everyone in the land says your are such a great wise man"? So, I came to get come wisdom from you before this big battle I am going to.

Yes, and that is the message to all the "great" developers "out there".

(Diogenes, one of the greatest and wisest men that ever walked the face of this planet, lived in an empty barrel on the bank of the river, away from the crowds, their "great ones" and their madness of a herd. And Alexanders of all kind were willing to give him a half of their kingdom if he would agree to teach wisdom.)

Tahoe-LAFS

There is also Tahoe-LAFS that is past beta phase and is a fully functioning product. It allows for creation of distributed storage, which has its own pluses and minuses as far as pure syncing is concerned. It is a bit more complicated than merely a sync program, but it seems to do its job, except we did not verify any of it and so can not make any specific statements about it.

https://code.google.com/p/nilestore/wiki/TahoeLAFSBasics

Hive2Hive

Hive2Hive is an open source library for sync written in Java. This is a good choice of a language because Java is as portable as it gets and runs on a variety of platforms.

Unfortunately, these guys are pretty slow on making a prototype app that can demonstrate what their library can do. Considering it is written in Java, it should take a few hours to create a sample prototype app for the others to see how and if it all works. But for someone who does not know anything about the code it may take days, if not weeks before one can see the results, which is totally unacceptable at this junction.

We wrote them an email at least 6 months ago asking for the same exact thing. But even to this day, Sept 6, 2014, we do not see a working demo app, which is a shame. All we see is another version of some theory, telling you things, but not putting it all together and hooking it all up to some GUI so that one could see how it actually works, instead of wasting days and weeks of his time trying to figure it out just by looking at the code.

What's wrong with those guys?

http://Hive2Hive.org/

github:

https://github.com/Hive2Hive/Hive2Hive

picosync

Picosync was an attempt to create something compatible to BTSync. The author tried to look at the traffic and packets and did a considerable amount of work trying to reconstruct some of the things BTSync does and how it works, and he was able to get some preliminary results.

Unfortunately, that is not quite a correct approach. Just by looking at the traffic may take many times longer to figure out how the program works, and even then some of the weird algorithms or BTSync can not be decoded by simply looking at the traffic. That is practically impossible. From purely technical standpoint, even program de-compilation might have produced more effective results.

For example, there is a pretty good decompiler called REC and it produces a decently looking C code. Sure, decompilation is not a simple thing to do, but still, you could see quite a bit of the inner workings with this decimpiler. Yes, most of decompilers you will be able to find will not even run. They'll crash on you as soon as you start it. But this one doesn't.

Download REC Reverse Engineering Compiler
http://www.backerstreet.com/rec/recdload.htm

As a result, there seems to have been no work on this project since mid 2013. But the author was able to figure out some significant enough aspects of how BTSync works, which deserves an applause.

Actually, considering a pretty weak methodology, the author was able to get quite a few things written in actual code.

Here is his code:

https://github.com/picosync/workingDraft/tree/master/picosync-qt/src

https://github.com/picosync/workingDraft/tree/master/picosync-qt/src https://github.com/picosync/workingDraft

ClearSkies

ClearSkies is not yet a functioning program. But it seems there have been a fair amount of thought and consideration to it, and you can probably find some interesting and valuable ideas in what is available at this point. And there is some code to look at. It is written in C++.

We haven't looked at it to the degree that we can make a comment worth the paper it is written on. So...

Here is the code:

https://github.com/jewel/clearskies

Protocol:

https://github.com/jewel/clearskies/tree/master/protocol

TomP2P

TomP2P is not a complete sync program or specification. It is a DHT implementation for node discovery, which is one of the more important, if not the key aspects or establishing P2P communication. We have just glanced over it and can not make any comments of any value. But one of the most important aspects we have noticed is that it works asynchronously and the author also claims a high performance. These are two of the most important keys to node discover over DHT.

Also, it seems that this DHT implementation is fairly complete and utilizes the distributed nature of the DHT node discovery, which significantly reduces the program load on resources and a memory footprint.

So, for the node discovery, this looks like something worth looking at.

One more good point going for it is that it is written in Java, which means it is as portable as it gets and is written is a well designed language that proved itself with time.

https://github.com/tomp2p/TomP2P

libutp - The uTorrent Transport Protocol library

This is one of the most critical subcomponents of the sync system. For more details see:

Transfer and higher level logic engines

uTP provides provides reliable, ordered delivery while maintaining minimum extra delay. It is implemented on top of UDP to be cross-platform and functional today. As a result, uTP is the primary transport for uTorrent peer-to-peer connections.

uTP is written in C++, but the external interface is strictly C (ANSI C89).

So, here's the code:

https://github.com/andradeandrey/GoUTP/tree/master/libutp

Vuze open source torrent client as a base to build the sync app

We have made some attempts to contact Vuze with a proposal and asking for a feedback on using the Vuze as a code base for the sync app.

Vuze is written in Java, which, for one thing, means a very high degree of portability. Another advantage is that with Java you are no longer forced to use the WebUI approach to GUI, which is not the best way to provide a powerful GUI. Because browsers are not designed for that. Java has a very powerful GUI subsystem, which is orders of magnitude more powerful and flexible than any browser based design.

Here is the messages we have posted on Vuze forums:

Vuze Sync plugin approach query

10-28-2014

I'd like to ask if any developers think it is possible to develop a plugin to do syncing.

Basically, as described in the following document, it seems Vuze/Azureus could be relatively easily adapted to do sync, considering that it has all the necessary underlying mechanism and a pretty powerful and extensible architecture.

If sync is torrent based, then implementing it in Vuze would not seem to be much of problem. Is that right in your opinion?

Or, if you think plugins would not work or would not be as efficient and/or reliable, or assuring data integrity across the nodes could be problematic, then could you please outline the issues as you see them and/or suggest a pointer where we can find some useful information?

Thanks in advance.

Here's description of how a torrent client can be extended to do the sync in that document:

Synthetic synchronized torrent approach

(Note: any valuable feedback and especially detailed analysis of the issues will be included in the Sync Detailed Manual, one of the most authoritative sync documents on line.)

Vuze Forums > Contribution > Open Development > Sync Plugin
http://forum.vuze.com/thread-366-post-995.html#pid995

What is the difference between torrents and sync?

11-06-2014

Basically, the idea with sync is based on the fact that the main difference between sync and torrent approaches is that torrents are static while sync is dynamic. That means that the information in the sync collection/share can be updated without changing the collection key/folder.

With torrents, you can not change, update, modify or extend a collection. The problem with this approach is that the information is usually dynamic in nature. That is why torrents get outdated in most cases, and so one needs to get the new versions of the same basic collection by getting a completely different torrent, while, in most cases, the most information in a torrent does not change.

With sync approach, once you get the key to the share/collection, you no longer have to worry about whether you have the latest information and you are always in sync. Secondly, updates usually come in small increments, changing just a few files, and that could be a significant improvement in terms of bandwidth, update time and so on. Typically, you can update the existing collection in seconds.

But what is most interesting about it is that you can utilize the underlying torrent mechanisms to do the sync. In fact, that is what BTSync has done. The transfer engine, node/peer discovery and the bulk of the code are pretty much the same. You just need to add the higher level logic engine and a few other mechanisms that are not complex in any fundamental ways.

So, it seems to be quite natural to do the sync using the existing torrent base code and merely extend the torrent client functionality. It would be interesting to see any ideas or criticism or suggestions of such an approach.

Vuze Forums > Contribution > Open Development > Sync Plugin
http://forum.vuze.com/thread-366-post-1098.html#pid1098

Messages on BTSync forums about 1.4.72 upgrade disaster mutilated and eventually deleted by the "moderator" and alpha test monkeys

Finally, these are the messages posted to BTSync forums. All, but the 1st one were removed for "violations" of something.

One funny thing about the "thought control police" on BTSync forums is that one of the most violent, obnoxious and arrogant "moderators", about whom several users expressed displeasure as a result of his "moderation techniques", has a title of "alpha-tester". But BTSync is in "beta" stage. So, he seems to be from a wrong joke.

Secondly, why are people like these allowed to edit and delete the posts and even shut down the user accounts on BTSync official forums? Are they at least employees of BT and do they represent the position of BT in all these matters?

Latest Desktop Build 1.4.72

Posted Aug 29, 03:58 AM

I am still trying to recover 1.3.109 related stuff.

Some shares got really screwed up because of new .sync subfolder and so I had to copy things back to the main share's folder and I guess some things I copied changed the mtime on some files.

So, now I have one folder with weird behavior. Since some of my mod-times on some files were of now, then they restart syncing them and create the filename.Conflict files. If I try to touch those files, then depending on how I touch them (-m only) option, they either reupload the same file adding .Conflict to file name, or upload AND download them, again adding the .Conflict to file name. But this is a pretty big share and I'd like to get done with it, rather sooner than later.

So, what is the right way of making sure I don't have to upload the existing files and have all these files with .Conflict? I thought that If the file hash is the same, then file is not actually transferred, but only the file mtime is changed. Does it matter which way the time is off on the file - past or future?

What are the exact instructions to recover that share?

Update 1:

What is interesting about this is that if my target share (r/o) has files with newer date, then BTSync transfers them to the master (r/w) share as well. I did ask this question before:

Why in the world the r/o share transfers files to the r/w share under ANY circumstances? What is the difference between the r/w and r/o shares then if r/o share can update the r/w share?

Does anyone know this?

Latest Desktop Build 1.4.72

Update as of Sept 24, 2014.

Nearly a month has passed since we tried to install the 1.4.72 disaster update. After days spent trying to recover from nearly all the nodes not syncing we can report the following:

All the following messages were removed by GreatMarko without even saying anything or commenting or explaining anything.

Latest Desktop Build 1.4.72

GreatMarko wrote:

I hope someone has an "authoritative" instructions on this.

I'm not sure how we can make this any simpler for you - myself and b0rman have both tried to explain how you can downgrade to 1.3. If our explanations aren't clear or "authoritative" enough for you, I would point you in the direction of this official article from the Sync Help Center, which states: "If you want to downgrade - please uninstall Sync and remove all settings, then install 1.3 and configure all folders from scratch."

First of all, it is b0rman who provided a semi-legible suggestion. Secondly, yes, indeed, these are not technically correct and complete instructions. Because what does it mean "uninstall Sync and remove all settings"?

One of the problems is the database format has changed in 1.4.72, so even if you remove ALL the settings, even if you have any idea what it actually means, all the files and registry entries created after you run the program for the first time will not be removed even if you uninstall the program from the O/S. Because install removes only those things that it created DURING the installation, but not after.

So, if you do not also remove the database files for all the shares so that you would start from the equivalent of the fresh install, during the installation BTSync will crash on you because of the bug that does not properly handle the database exceptions it seems. And if you try to restart it, it will crash in few seconds. It simply won't run from then on, no matter what you do. And if you try to reinstall BTSync again, hoping to recover, it won't help a bit. Unless you completely remove all database related stuff, you won't be able to run the program. That took us a day or two to figure out on our own since these alpha-tester monkeys and "thought control police", called "moderators", or better yet, dominators, simply started deleting the messages posted to BTSync forums expressing the issues as they were coming up.

Then, there are several ways of doing the uninstall and it depends on the internals of the install itself, which may behave differently, depending on a program. The most complete and thorough way to do it is via Windows control panel. But it may have a nasty side effect of removing the registry entries and so when you reinstall the program later, you may loose some of your important setting that are kept in the registry. Furthermore, it may screw things up so badly, that the program will not behave correctly if some of the key settings in the registry are deleted and you may never be able to recover some things, just as we see after being able to finally reinstall the 1.3.109 version, after which every single node that is in fact synced started showing that it is empty.

And, no matter what you do from then on, they will show empty forever. That means that we are seeing something of a virus-like behavior and it has in fact propagated to the nodes themselves, and, no matter what you do from then on, you will never be able to restore the correct sync state unless you have access to those nodes physically, which, in general public situation, where you have no idea who are those users and have no way of contacting them, is simply impossible.

It's been more than a week since we tried to upgrade to 1.4.72 and we still have no idea as to what can be done to fix several shares so that all the nodes show the correct sync completion status. On one of them, ALL the nodes but one show as empty while they are in sync. So, at this point, we do not know which of those nodes do sync and in what state they are. And it is likely the users running those nodes also have no clue of what is going on and some of them may decide to abandon either a share or the whole program, interpreting it as "hey, this thing does not work, screw it". And these are the users on some of the most important shares we have.

Which means that you will see those nodes, and in our case the entire share, as UTTERLY OUT OF SYNC! FOREVER!!! You can just jump from the Golden Gate bridge and even that won't help. Furthermore, those nodes will never show as synced in the Transfer tab (releases up to 1.3.109), if they will ever sync at all!!! How do you like THAT kind of issue and what do you call it but a total disaster?

That is why computers are PRECISE tools and no sloppy factor of any kind is forgiven or allowed. Else, you might spend days and weeks trying to recover everything to the proper operating level. So, in such a nasty situation, where BT support staff admits that going back to previous version is not supported, they need to provide the EXACT, step by step instructions to revert to the previous version instead of merely saying "remove all configs" which is meaningless of a statement. Many people may have not a slightest clue of what it means.

The second way to uninstall the program is to simply remove the program executable by deleting the file. In that case, your registry settings are not affected and you may be able to recover at least some of your previous settings the next time you reinstall it.

The way the BTSync install/update works is that you do not actually extract/install some files out of the installation package. The program, as distributed, is simply copied to the final run directory (C:\Program Files\BTsync on Windows) and, depending on where are you running it from - from the download directory or from the final run directory, it simply behaves differently. The code executed is exactly the same, only some conditions change. That means that by simply removing BTSync from the the directory from which the program runs may and should provide it all the clues it needs if you try to reinstall the program. If it detects that there are no executable in the final run directory, then it should properly perform ALL the actions necessary, including the registry settings and config or database file removal, if necessary, to properly reinitialize the program so that it runs 100% correctly when you run it. This is another one of those weird things about BTSync - the installation is done not in a commonly accepted standard way, which is strange by itself.

Secondly, what does it mean "remove all settings" and what is IMPLIED by "settings"? There are configuration files that are kept in appdata BTSync directory, such as, settings.dat and sync.dat. They do have most of the program settings. Then there are numerous settings in the Window's registry and there are quite a few of those thrown around the registry in totally different places as we have noticed. So, if there is one or more of SPECIFIC setting that need to be removed, then they have to provide you the exact key for those settings and their location in the registry.

And what other "settings" might be there, only the competent people know. So, what should one remove and where in order remove those settings? Is this a COMPETENT answer to the precise and specific technical questions or just a sloppy way of saying "if you have a problem, get lost and 'read the manual'"?

Is this something that constitutes a competent and to the point tech support on a dedicated forum, especially in a situation with fatal consequences where you will be banging your head against the wall for days to finally restore it to a working state?

First of all, the 1.4.NN upgrade should have provided you some message during the installation with warning in big letters that once you start installing the new version, all your configurations and your previous installation will be screwed up and you will not be able to recover it.

Secondly, the installation should have made the backups of your previous installation and save it in a way and in a place that you can easily recover and go back to the previous version. Because the new version is a radical change to quite a few things and it totally breaks the existing installation in irreversible way.

Thirdly, quite a few people may and will not like the very idea of being forced to run IE for all sorts of reasons, and may find this new GUI simply pathetic and totally unacceptable, which is in fact happening right now as more and more people say that they are going to try to go back to the previous version and will no longer hope that this whole thing has any future.

This "new GUI" approach of running under IE appears to be the management decision and they have made the statement that this is the way it is going to be no matter who thinks what. There is no way back and old style proper GUI will no longer be supported in the future and versions prior to 1.4 are not going to have the bugs fixed and no new updates of 1.3.NNN version are likely to appear.

Anyway, let us see more of this whole "customer disservice" trip.


Latest Desktop Build 1.4.72

Free cheese is only in a mouse trap they say.
(in response to someone saying "but BTSync is a free software...").

Secondly, doing the PR number with this "but this is just beta" gets boring after a hundred repetitions. You can not run "beta" company, or "perpetual beta" release, can you?

So, instead of fixing the mountains of bugs they simply blame it on "but it is beta", you know.

And, meanwhile, trying to corner the distribution market and not providing the information to create a compatible product. Because since 1.4.72 I do not see any future for this thing except on a zombie market of biorobot "entertainment". The way it looks right now with this new "gui", you can never make it work so it looks even half close to the real UI of half-decent program.

Following message is a response to GreatMarko's excuse and insult post which said something of a kind "but it would be better not to complain about problems".

Latest Desktop Build 1.4.72

Posted Aug 29, 06:51 AM

And it would be better to realize that beta does not mean "it has to be broken by default". Because beta means a product nearly ready to be released, as good as version 1.0.

Seeing this "but this is just beta" excuse makes me wander: where am I at - a Herbalife or Scientology promotion meeting?

Beta means it WORKS, and not "it is full of bugs".

Providing the in-depth information and warning people means CARING, means the product has future. It is alive.

But pissing in people's faces instead of providing the exact and precise instructions and description of behavior is what?

Telling people "look, I do not know how to make it simpler for you (because you look dumber than a piece of wood)" is what? You don't have to make it "simpler" for me or anyone else. You just need to give the precise and detailed instructions or refer me to the place where it answers things in a concise manner. Saying things like "remove everything" or "remove all configs" means many things, such as btsync config directory, win registry and so on.

When you are asked a question: but how come ALL the nodes show as empty while they are in fact fully synced, then where is the answer?

Saying things "I appreciate your difficulties" is nothing but an insult. How can you even comprehend the difficulties I had to go through struggling with this thing for two days now? And what does it mean "I appreciate"? Do you REALLY? Are you a sadist by any chance?

Latest Desktop Build 1.4.72

Latest Desktop Build 1.4.72

Posted Aug 29, 07:28 AM

Why did you delete the message that has a 1.4.72 related issue?

Just for your information, this message is not lost and it will be posted in the best BTSync manual there exists. And there will be more posted related to 1.4.72, general strategy and related issue.

One more time:

Why after I installed 1.4.72 do 14 out of 15 nodes show that they are empty while in fact they are synced?

And even after I completely wiped out 1.4.72 and deinstalled it from O/S and removed all the configs and databases and everything else from the appdata and then installed 1.3.109 I see all those nodes not syncing any more?

What do I do now to recover?

Latest Desktop Build 1.4.72

The "response" from BTSync was: Both of those messages were deleted and then this:

You have been placed on moderator queue until 25-February 15. This means that all content you submit until then will need to be approved by a moderator before it will be shown.

And so, "to add insult to the injury, they send you these emails:

Warning issued by GreatMarko for Abusive Behaviour in Profile.

Given 1 points.

Stanah,
Your latest post has been removed due to the inappropriate, aggressive, and abusive tone contained therein.

Posts of such nature are not welcome in these friendly and constructive forums.

You have been warned by moderators/administrators on NUMEROUS occasions previously about the nature and tone of some of your posts in the forums, and even had your posting rights "moderated" previously.

Continued posting of content that is inappropriate, aggressive, or threatening WILL result in a permanent ban from these forums.

You have been given ample warnings!!

Thank you for your co-operation.

Warning issued by Harold Feit for Abusive Behaviour in Profile.

Given 1 points.

Content moderated for 180 days.

Even after your previous warnings, you continue to be abusive. Due to this you are now on a 180 day moderator preview.

"Moderated" or abused? Yes, most "moderators" you see "out there" are some of the most immoderate people, craving for power, control and domination. In other words, sick people. And that is precisely why they licked some rear ends at the beginning of their "career" to get their hand on this red "moderator" power trip button labeled "Kill", so that they could insult, ridicule, delete messages and even ban the users.

That is their "joy". And to make sure they have it, they pretend to do a "good job", helping people. But most of it is not authentic. It is nothing but endless self-promotion, so that everyone would get an IMPRESSION that these are some of the most helpful people there are. And to make sure they do have access to "kill", "delete" "deny" and ban button, they keep promoting those on behalf of whom they are doing all this trip, those who own the whole thing. And the harder they lick, more "credit" they get. There comes the point where they look nearly irreplaceable, and that is the idea. It is called "job security" trip. Weird stuff, I tellya! :)

Because it is essentially a result of self-denial. As a result, one denies oneself the very possibility of being creative and, as of necessity, creativity as such. That's "the whole point of it". As a result, they claim there is no such a thing as creativity as such. And the inevitable outcome is that you become destructive. If you are not creative, you are BOUND to become destructive. Because the emptiness has to be filled with something, some substitute, some surrogate, even if it is something unnatural, that void needs to be filled.

And a result of that is that NATURAL joy, that comes as a result of creative acts, is absent and a false notion of "joy" comes to replace it - a perverted "joy" of destruction. Just like in sadists. Thus, once you deny yourself the creativity, you are bound to become a pervert and sadist. If you study these issues in-depth, it will become "clear as a bell". As they say: "there is no ifs and buts about it".

Just ask yourself a question: why would anyone waste his life and spend most of his time on being some "moderator"? For what, when you can enjoy the creative aspects of Life and explore things? What do these "moderators" do but performing the functions of the "thought control police", the most pleasing aspects of which, for the perverts, is the acts of mutilating the messages of others, ridiculing and threatening them, telling them how they "should" think and all sorts of other destructive activity?

And the final "joy" of moderation is kill-ban-deny, shut them down, make them dead, which is nothing but psychosis and about the deepest sickness of ones own being!!! What a waste of Life and creative potential! These sadists do not even realize that the biggest harm they do is to themselves by denying the validity of THEIR own being.

These destructive entities do not even realize that they merely perform the rituals of worshiping death, which is the biggest illusion of all. Because Life IS creativity, no matter how "small" or "insignificant" it might look to others and no matter in what area of human activity it is carried out. It does not mean you are some great artist, poet and so on. You can be a cleaner or a cook and even a grave digger, and still creativity is possible. It is the very nature of Life. Without creativity no Life is possible, even in principle.

And that is "the bottom line" and a core reason for propagation of sickness and violence of all kinds, which you can see nearly any place you look, and it all happens on subconscious level. Most people do not even recognize and realize their inner tendencies. HUGE subject to work with...

The sickest of them all get "awarded" a "moderator" status. And this sad thing is the reality of the "modern world" in the NWO style. Impose "democracy" on them with sword and fire!!! This is the very bone and marrow essence of this ugly totalitarian domination trip of "moderators" and "thought control police" of all kinds nearly every place you look.

These people, if you can call them people, are so sick and so filled with "power", domination and destruction trip that they should have NEVER, under ANY circumstances whatsoever, be allowed even close to not only the "moderation" red button, but to the customer service job of ANY kind in any self-respecting company. Simple as that.

And, for some strange reason, the sickest ones of all are precisely those who are "moderators". Strange, isn't it? And the most interesting thing about it is that on BTSync forums quite a few people told these "moderators" about their rudeness, insulting and even offensive behavior. And even if they give you some "advice", quite often it is either superficial or incomplete or simply said by someone who TRIES to get his face into everything and stuff his ugly favicon into everyones face, as often as they can. This "me, me, me" number. Who are they trying to convince of their validity? Not themselves, by ANY chance?

Which means it is not something nobody knows about. The issue is right up in your face. And they still remain as "moderators". In that case, the question arises: is it done by design, by ANY chance?

"Content moderated for 180 days" means that you can not post to the group. Because your posts go to the "moderator" first, and after he "approves" it, he MAY let that post through. MAY. If he is in a mood and you don't bring up or insist on resolution of the issues that they seem not to know how to resolve or wouldn't even want to bother to look at them. And it seems at least conceivable that they do in fact know those issues and are deliberately trying to hush them down and remove all the "undesirables" from even being able to discuss it.

In that case, it may imply the intentionally built-in destructive behavior of BTSync BY DESIGN, as wild as it may sound. For example, this "unusual" behavior may be used to devastate the "undesirable" information. But to users it may look like a bug or something does not work right kind of thing. It is like a back door to your comp or BTSync share in this case. Because some of those issues have a profoundly destructive effect on the nodes and shares and allow one to literally devastate your shares under certain conditions, which we do not wish to discuss at this time, and for several reasons. But if we smell something rotten or destructive, that information will be released.

We have certainly observed this behavior in specific cases while either experimenting with the program behavior or were trying to assist the other nodes to come to a correct state of sync. And we have seen it on more than one occasion and found it to be predictable to the point of certainty.

Several months ago, we have brought up the issue with timediff problem (some nodes do not sync at all if their clock time is not set as BTSync expects, and it looks like what BTSync "expects" is totally wrong. Also, nodes that have a timediff error begin to behave in totally unexpected ways in respect to the other nodes. But these issues are too technical to discuss here).

They simply flatly refused to even LOOK at the timediff issue and this sadist "GreatMarco" simply stated that this issue was not significant enough because not that many people post to that thread. But when we posted a message saying that that particular thread is read by over 80 people every day, and there are all sorts of reasons why people do not post, even if they have the same issue, they simply started attacking the author, insulting, ridiculing and provoking him in the most disgusting and even offensive ways, and finally deleting his messages and going as far as placing him on this "quarantine" trip, insane as it gets. So, they simply suppressed the whole discussion of one of the nastiest problems with BTSync. For what? What is the result?

But did the problem go away? Well, several months have passed and the problem is still there. Not a thing was done to fix it. So, who is "guilty" of what here? And who is "wrong" about what in this situation? Are these issues REAL, by any chance?

Finally, after a couple of follow up attempts to address the issue, they did the same number and issued a "warning for abusive behavior" and also placed the author under this "moderator" quarantine trip, which is simply nothing but insanity.

For what? For bringing up valid issues and trying to get their attention so they even look at it? So, instead of being thankful for the contribution and identification of one of the nastiest bugs and some of the most vital issues, what do they do? "Attack the messenger"?

Eventually, one of the customer reps admitted that there IS such an issue and promised that it will be taken care of "in the future release". That was about 6 months ago as of this writing, and there have been at least 10 releases since then, and that critical level issue still exists to this date, even on this "latest and greatest" 1.4.72 disaster release.

And here's the kicker in this whole trip of insanity. A while back, the author of this manual has received an email from BT saying something like: we really appreciate your work on this manual and it is a great help to BT. May we, please, use it for our needs? And so they were told: sure, that is exactly why it was created in the first place. But then, after all this insanity started with all their bizarre denials, attacks and ridicule, we have informed them: NO, you can not use this manual in no way, shape or form. Why should we support the butchers?

So... Does it look like a customer service more than a butcher department and utter denials of That Which IS? Mind you, not a single competent and sufficiently precise answer to several different questions that address the issues of a fatal grade was provided. Does this look as something promising? Promising of what and when?

Where to get BTSync?

Note: we do not recommend installing version higher than 1.3.109 because of all sorts of UI problems. You can get v.1.3.109 here:

v. 1.3.109
http://syncapp.bittorrent.com/1.3.109/

The latest version of BTSync (Bittorent Sync) can be downloaded from the official BT (Bittorent) site:

http://www.getsync.com/download

The "official" instructions to start using it are here:

http://www.getsync.com/

How to get this manual via BTSync?

Note: This manual also available via following btsync keys ("secrets" or hash strings):

BUTDXDOOUA6IVT73U224QZWSFQSLEOF43
(Read only version, if you are only interested in reading it, but not contributing to it with your edits).

Note: If you edit your local copy on r/o node, you will no longer receive its updates. So, it is not a good idea. Better to create a copy of it, even in the same directory, and work with that one.

But, if you want to keep receiving it again, then in Folder Preferences -> Advanced tab check the "Restore modified files to original version" checkbox. This should restore the original sync status of all the files in the database for this hash (share) and re-download the latest versions to your r/o node.

AOANJC66VZEM2PAEA4WKG2Z5TFRBLT7PJ
(Interactive version. Full access (read and write with real time update of all users on line), preferably to be used by those who know the issues and not merely guess things).

This manual is updated periodically. So, the best way to see "what's new" is to simply open it in a new tab in the browser and refresh the page to see what's new. The TOC is normally updated with the new topics. But edits of existing topics may be found only by reading the topic of interest and/or refreshing the page.

Please keep in mind that if you downloaded the Full access version of it and decide to edit it and/or add something of your own, you may "step on edits of others" and some changes and/or edits and version may get lost, even though the archive version is available in the archive subfolder. Your own edit may be stepped upon by someone else if your editor, assuming you also keep it open all the time, does not notify you of the file changes on disk.

It is suggested to use the editor that notifies you of the file changes when you switch to it. For Windows, the prime choice editor could very well be the Notepad++, probably the best and free text editor out there.

See: Notepad++ Home
http://notepad-plus-plus.org/

So, in case you get notification of the file changes on disk when you tried to maximize the edit window, where you made some changes, do NOT "reload file from disk". But respond - NO. Then, copy paste your version to a new tab and then do "Reload from disk" option in the File menu. Then you can merge your changes by copy/paste.

This document is interactive for all those who got it via BTSync using the above hash, meaning that everyone can modify it to add their own issues and topics, or correct, clarify the issues and so on. And everyone else will see their modifications within seconds.

Program futures and comparison with torrents

BTSync operates on the same principles as regular torrent programs except it does not create a single torrent file for the entire collection, but creates a separate torrent for each file when it needs to transfer it to others.

Advantages:

Disadvantages:

Existing BTSync collections (English)

All the existing BTSync collections are described in the following page:

Existing BTSync collections (English / Русский)
(Opens in new browser tab)

Useful links

http://www.getsync.com/

BitTorrent User Guide
http://btsync.s3-website-us-east-1.amazonaws.com/BitTorrentSyncUserGuide.pdf

BitTorrent Frequently Asked Questions
http://www.bittorrent.com/help/faq/sync

BitTorrent forums - Sync General Discussion
http://forum.bittorrent.com/forum/56-bittorrent-sync/

Files are not being downloaded or synced

There are several reasons why btsync does not start downloading files after you created a new btsync folder. Generally, following are the major reasons:

It is highly recommended to create a folder (directory) from btsync program when you are creating a new share by clicking on "Add a Sync Folder" and are adding a secret word for this share. Because if btsync has problems creating that folder (file permissions do not allow it for the directory where you are trying to create this folder), then it will display you an error message and you will be able to fix the permissions and then click again. Furthermore, btsync will create the directory with correct file permissions and ownership to allow it to save files in this directory.

If you have manually created the folder where you want to download and entered a secret and selected that folder, but (some) files do not download for some reason, you need to make sure that the folder you have created is readable and writable by btsync and the user account which runs the BTSync program (btsync.exe on Windows Task manager -> Processes).

In Windows explorer, right mouse click on that folder and in Properties -> Security, make sure that your user account has Write permissions for this folder.

The same thing is applicable if you work with Linux. That folder should belong to account btsync and group btsync should have the write permissions for it. It is best to create the folder from the BTSync GUI while initially creating this folder in your btsync collection. Simply copy paste the folder name from Linux, enter secret word and hit Create folder button. If btsync could not create that folder because of permissions, it will tell you so. In that case, you need to change the permissions in the parent folder, change its group to btsync and make it writable by the btsync group. Also, set the SGID bit (file mode 2775 - r/w/x for user, and group).

If you create that folder by hand from Linux terminal or from explorer, then make sure its permissions and ownership are set as above. That folder should belong to btsync group. Otherwise, btsync may not be able to start downloading it because it does not have the write permissions for it.

The detailed instructions for Linux could be found here:

How To Use BitTorrent Sync
https://www.digitalocean.com/community/articles/how-to-use-bittorrent-sync-to-synchronize-directories-in-ubuntu-12-04

If you have any problems, send the email to here: preciseinfo [@] mail [.] ru.

If time is set incorrectly on your comp

Time has to be set correctly on your computer, including the time zone. Otherwise, if it differs from the time set on the seeder computers by more than 10 minutes, then btsync can not sync your computer, and you will be informed about it next to device name of the seeder with the message:

Sync stopped: device time time difference more than 600 seconds

Most likely, your time zone is not set correctly or may be your time is not set correctly. Could be any one of those. So that your local time in combination with time zone set you have chosen, does not correspond to global GMT/UTC time standard.

In order to find out how much is your time off from other nodes in a swarm, you need to turn on the logging. See: Enabling logging.

Then, open your log file in your text editor and search for a string "Merge: timediff with remote peer" (NNNN is more than allowed 600, aborting). It will tell you how much your time is off with each separate node in the swarm, which you are trying to contact (+/-). And so, there would be necessary for you to change your time zone (most likely) or the time by that amount in order to decrease that difference to within 600 seconds.

For example, you find a message in your log that says:

"Merge: timediff with remote peer -3909".

This means that YOUR time is LESS than time of the remote by 1:05:09 [HH:MM:SS]. So, to be able to sync, you need to INCREASE your time by 1 hr. You can ignore the 5 mins 9 secs. Because it is within the allowable range.

Here is the log records for this situation:

[2014-03-20 22:36:23.542] Got id message from peer Mac mini (00657FB386A75D415E5684E3734368751E47D336) 1.2.91
[2014-03-20 22:36:23.542] Merge: timediff with remote peer -3909 is more than allowed 600, aborting

If your computer or system still makes the automatic and periodic requests to synchronize its clock with some external source, and, as soon as you change it manually, it is reset to that time, then you have to disable that automatic update.

So, you need to synchronize your time with one of the sources on the net and to make sure your time zone is set correctly. If, for some strange reason, your "normal" time zone is set in such a way that your UTC time differs by a number of hours, just try to change the time zone until the error message goes away and sync starts. Once you are synced, you can reset it back if you have to. But every time you want to update your sync files, set it back temporarily until sync completes.

I can't connect to other devices - I think it's a Router/Firewall issue?

If BitTorrent Sync is struggling to connect through your Router/Firewall, this post may be of use

Also, if you're having trouble connecting to other devices over a VPN, try increasing the MTU size to at least 1500.

Finally, please make sure that all your devices are running the same version of BitTorrent Sync. For example, 1.0.x versions will not connect to 1.1.x versions, and 1.1.15 won't connect to 1.1.22, due to changes made to the protocol between these versions.

Source:
BitTorrent Sync FAQ *unofficial*
http://forum.bittorrent.com/topic/17782-bittorrent-sync-faq-unofficial/

Sync isn't completing - not all my files are synced / file counts don't match!

There are a number of reasons why this may happen:

  1. Sync won't sync files whilst they are currently open/locked/in use by other applications. Closing any open files in these applications should then allow Sync to complete the file transfer.
  2. Some files may be excluded from syncing (check the exclusion rules in your .SyncIgnore files)
  3. Files may have invalid timestamps. If a file has an invalid timestamp, for example, it appears to have been created in the future!, Sync may ignore the file.
  4. Sync won't include a count/the size of files in .SyncArchive folders, where as your Operating System will - this often explains why Sync reports one file count/size, and your os reports a different set of values.
  5. You modified or renamed some file (applicable to r/o nodes only) [not sure about this one though].

Source:
BitTorrent Sync FAQ *unofficial*
http://forum.bittorrent.com/topic/17782-bittorrent-sync-faq-unofficial/

Restoring the original sync status of some files that won't sync any more

If you made some modifications or deletions of some files and, as a result, it does not seem that some files are updated when they are changed on the master machine, and you are running Linux, you can try to do following:

After this procedure your folder should show as fully synced and it should be exactly in the same state as the master (r/w) node.

BTSync goes over all the files in DB (database), removes the “Don’t propagate” flag and requests the torrent file (list of hashes for file pieces) from other peers. File pieces that are changed are re-downloaded.

[...]

... Files that are not on RW peer won’t get into database. Once they appear on RW peers, RO node will add them to database and IF they are different from RW peer - will mark them as “Changed, do not propagate".

(RomanZ, Sync Support)

Yes, this behaviour is designed. As soon as hash of the file changes (i.e. some actual data is modified) OR file is deleted / moved outside of the folder, the BTSync marks it with "Invalidated" flag in database and never re-syncs back.

This behaviour can be adjusted. In BTSync, in folder properties there is a setting "Restore modified files to original version" which is unchecked by default. You can check it and BTSync will restore files when they are changed / removed.

(RomanZ, Sync Support)

When A File Is Deleted From A Remote Sync New Changes Are Never Resynced
http://forum.bittorrent.com/topic/27938-when-a-file-is-deleted-from-a-remote-sync-new-changes-are-never-resynced/#entry79268

But we have seen the cases where this procedure does not help. In that case, you may try to delete this folder from BTSync (but not by physically removing files or the folder itself from the file system). In this case, the files themselves will not be removed by the program. The only thing will happen is the removal of the database for this share in BTSync.

Then re-add the same folder to BTSync with the same key. After that, the program will perform re-indexing of all files and will restore their original status of update and distribution, and all the deleted, renamed or modified files will be re-downloaded from the other nodes.

Installing BTSync on mobile devices

Installing BTSync on mobile devices, such as tablets or phones is described here: Установка BTSync на мобильных устройствах и планшетах

BitTorrent Sync User Guide
(Opens in a new tab of your browser)

General notes

This document contains some text from the sources on the net that may provide the instructions that mention the older versions of BTSync.

Where to get BTSync?

This is the official download location with the latest version, if you decide to install any version above 1.3.109:

Downloads
http://www.getsync.com/download

Also, even "hotter" version may be available via "Sync General Discussion" forum on Bittorent forums. The thread with the latest version information is usually named "Latest Desktop Build 1.3.93" (for version 1.3.93).

See: Sync General Discussion
http://forum.bittorrent.com/forum/56-bittorrent-sync/

So, in any instructions here that deal with installation, use the above locations and skip any instructions that tell you to download BTSync from elsewhere. There is one exception - Linux tuxpoldo version that uses the ppa. That version, strictly speaking is not "authorized" by BTSync but widely accepted as some kind of a standard. But read BTSync forums for more information.

Setting up Folder Properties

Folder Preferences in Folders -> right mouse click -> Show folder preferences choice -> Properties tab defines the parameters for a folder in terms of how it finds the other nodes, connects to them, what kind of tracker does it use and other things that are pretty important in specialized applications, to manage your security and configure other connection related issue.

Even though the Properties tab has only few parameters, but it presents you with many ways to configure your folders connection and security-wise. Some issues and settings are quite complicated in their effects to describe in few sentences, but we can still try.

Let us go through each individual parameter, as presented by RomanZ of BT Sync Support.


Search LAN

BTSync sends multicast packets to 239.192.0.0 over UDP 3838. Only peers in current subnet receive these packets. Peers have to run BTSync to be discovered.

(RomanZ, Sync Support).

http://forum.bittorrent.com/topic/17736-own-tracker/#entry80281
(own tracker thread)


Search DHT network

Well, this one is a tough cracker. The DHT protocol is pretty complicated and is based on principles that may not be easy to understand to an ordinary user.

Basically, it has to do with the issues of how you find other nodes on some share. There are different ways to do it. One of them is when you know at least one node with static IP and predefined, fixed port number and every node on some share connects to it as they would do to a regular tracker. In effect, that node becomes a private tracker.

For example, when you click on "Use tracker server", it tells BTSync to use predefined and hardwired BT trackers with known IP:port addresses. As of March 22, 2014, there are 3 BT trackers (for redundancy purposes) that will be contacted to to get the list of all peers for a given share (sync folder or a "key").

In that case, BTSync is guaranteed to provide the connections between ALL the nodes that have contacted the BT tracker. When some node contacts the tracker with a peer request, tracker returns a list of ALL peers seen for this share. That means that they do not have to contact or search for each other, but simply contact a centralized tracker list. We'll talk about it more in the "Use tracker server" parameter description.

The other way to discover a list of all the nodes for some share is to use the tracker-less DHT network. Again, the DHT protocol is pretty complex to describe here, but the basic idea is that you send out the UDP broadcast packets (to the whole world), or, more precisely, to virtual DHT network, and every node running the BTSync and has "Search DHT network" parameter enabled, becomes a member of global DHT network.

But not all the members of the DHT network necessarily have a list of peer nodes for a given share. That would be too much load on them and would slow down the network considerably. So, DHT has a concept of the "closest address", and each node that has this "closest address" will respond to the DHT requests. Eventually, the node that has the EXACT peer list for a given share will respond and send out the list of all the nodes that are either peers on some share, or have been participating on it recently (which times out eventually).

In this arrangement, you are not dependent on the CENTRALIZED tracker and, therefore, that tracker can not possibly interfere in your ability to discover all other peers if tracker decides NOT to respond and provide the list of peers, possibly on selective basis, to the specific host, being blocked, for whatever reasons. If tracker does not provide the list to your node, then you won't see the other peers. There are many aspects to this, but the basic idea is that the tracker HAS the ability to control which nodes receive which lists. Whether they will actually start interfering this way or not is irrelevant.

What does it mean the DHT NETWORK?

Question: It is not clear if search DHT is applicable to both, the r/w and r/o nodes.

Answer: Both. DHT means you have to store part of distributed table as well as use other peers to find what you need.

Q: The 2nd question is: what does it mean the DHT NETWORK? Who may become a member of that network? Is it only the BTSync nodes? Then, what exactly happens if this search box is checked? Does it mean that the r/o, or even r/w nodes will broadcast some hashes worldwide and all those, who are interested in this share and have btsync running and have this particular share in their database may respond with their IP/port info in case this checkbox is checked on their node? In other words, is it necessary for some share to be physically present on some node in order for it to respond to DHT requests, or is it enough for it to merely run the btsync?

A: BTSync and uTorrent peers. Checking "Use DHT network" means that you are going to search DHT for the hashes you need as well as store part of DHT and respond other searching peers. As soon as you are part of DHT - you are going to keep some DHT load, regardless of the folders you have. The distribution itself has complex logic, you can take a look at Wiki for details.

(RomanZ, Sync Support).

http://forum.bittorrent.com/topic/17736-own-tracker/#entry80281
(own tracker thread)

Comment: This means that if you enable the DHT, you will store SOME node lists (distribution tables) on your own node. Those lists will be only the "closest" nodes to this share (closest does NOT mean closest physically. It means closest in terms of hash code, or "the name of share" and has nothing to do with physical location of nodes. DHT protocol scrambles the hashes in order to randomize the names and then simply uses numerical comparison to find the nodes that are "closest". This is done for the purpose of distributing DHT traffic more or less equally across the DHT network.).

This is also an optimization of the protocol to make sure all participating members of DHT network do not store the whole world' on their nodes, but store only those lists that at least have a chance to be "close enough" in terms of share hash. But all their stored lists are not necessarily guaranteed to be the EXACT matches for some share. And then it gets even more esoteric then this :).

There is an interesting aspect to peer discovery. Once you know at least one peer for some share, that peer ALSO provides you with the list of ALL the peers it knows about. The net effect is that as soon as you discover the 1st node, the process of discovery explodes and you very quickly find all the other nodes for this share by directly contacting them, without even DHT network involvement, and they will provide you with the lists of peers that THEY know about. So, the process of further node discovery after the initial one, goes really fast and is very efficient. Every single node very quickly discovers every single other node on this share and every single one of them knows all others.

So, when new nodes join the DHT network and it provides them the address of at least one peer, from then on, they can very quickly get the list of all other participating nodes without any further contacts to the DHT servers.

It should be noted that DHT is applicable and works for ANY node, regardless of whether it is r/w or r/o. Every node that has the "Search DHT network" checkbox enabled becomes a participant in the network and it may make the requests to the DHT network and also becomes a DHT server that provides the list of peers to other nodes that also have this option enabled. And it also provides the lists of the "closest" members of DHT network, and not only the list of nodes involved in this particular share. So, there is certain overhead associated with DHT in terms of traffic. About 5 to 10% of your bandwidth may go exclusively on servicing the DHT requests. Again, the DHT protocol is pretty complicated, even for the professionals to understand all the nasty details of how it all works, but this is a basic operational idea about it.

So, "the bottom line" is: if you want to make sure you will discover the maximum number of peers, (in case you have enabled the BT tracker and it decides to interfere and not to provide you with the list of peers), simply enable the "Search DHT network" feature. It won't hurt anything besides a little wasted bandwidth.

It has to be noted though that if you enable DHT, security-wise, you expose yourself more than if you only had your own trackers (see "Use predefined hosts" description). Because you broadcast your requests and send out the responses to the whole world, basically to anyone who is participating in the DHT network. But security related issues are even more complicated if you look at it from different angles. So, we are not going to talk about it here.

But in that case, you will be able to discover other peers only if at least one more node on this share also has his DHT turned on. In other words, if you are the only one in the swarm that has its DHT turned on, then it won't help it to discover any other nodes unless it also has the "Use tracker server" option enabled.

See also: Peer discovery principles


Use predefined hosts

This parameter is basically the same thing as "Use tracker server". Except you can disable the BT tracker for whatever reason, such as security, and, instead, use your own private tracker for this folder. So, as far as tracking, it behaves like one. But there is a difference: trackers do not actually have to have the data for the share, but predefined hosts do.

Basically, predefined hosts are no different than any other node in the swarm and they behave in a semi-tracker way not because of some particular behavior, but by the sheer fact that ANY node shares the list of all nodes it is connected to, even though it does not behave like a tracker in terms of discovery of other nodes. It simply shares the list of nodes it knows about, just like any other node.

So, the difference between the tracker and any other node is that you know its IP/name and port, and so any node that knows about it, can contact it as to a "well known address".

The only thing to remember is that you need at least one other node in this swarm that also enables this option and have at least one "predefined host" on their list. If you are the only one that has a predefined host and other nodes can not "see" you for whatever reason, then your server becomes virtually useless. But, if at least one of them has "Use tracker server" enabled, then the result will be joined in terms of total number of nodes known in the swarm to all its members.

Therefore, even if BT tracker does not distribute the full list of all the nodes in a swarm to some nodes for whatever reason, but still distributes at least some of them to at least some nodes, then your "predefined host" becomes your guarantee that all the other nodes will be discovered by all other nodes.

For creating the secure private networks, every single node in your network has to have "Use tracker server" disabled and "Use predefined hosts" enabled. In that case, all their predefined hosts will also serve as private trackers. Also, you need to turn off "Search DHT network", so your "footprint" on the net is minimized and you are not visible to others on the net (unless, of course, BTSync will transmit some information from your host even without your knowledge. Whether they do it in fact or not is not significant in terms of security. But you can prevent them by number of ways, such as a firewall blocking ALL the BT trackers or via other means, which is a separate subject of discussion).

So, let us see how it works. First of all, this is what BTSync team member has to say about it:

Predefined hosts has no relation to DHT. It is just a host with installed BTSync, presumably having the same share and listening to a particular port.

Can ANY host merely running the btsync, can also behave like a tracker even if it does not have this particular share? In other words, what are the exact condition for some node to become the "predefined host"?

Every host is a little bit tracker: contacting other peers it will let them know the information on how to contact other peers related to the share they are merging right now.

Main target to create predefined hosts was to offer a way to contact some peer which is not contacting tracker (and therefore can't share information about itself).

(RomanZ, Sync Support).

http://forum.bittorrent.com/topic/17736-own-tracker/#entry80281
(own tracker thread)

What it means to me is that a predefined host HAS TO HAVE the data for a share, unlike the DHT discovery process. So, it can not serve as purely a tracker. Pure trackers do not need the actual data for the share. But predefined host does act like a tracker ALSO, in a sense that through it you can discover all other nodes in the swarm that are connected to it.

Secondly, what it means is that every host in a swarm has a list of peers it is connected to and can share that list with any other node in the swarm that are known to it. That is the meaning of "Every host is a little bit tracker".

Chaining effect

Let us assume that we have a private network with 5 nodes that are interested in participating on the same share. How would they find each other?

If NONE of them had a single predefined host, then they would not be able to find each other (assuming they do not use either BT tracker or participate in DHT network).

Let us say that host 1 adds a predefined host with IP/name of the host 2. Now, both of these hosts can connect. But all other hosts still can not find them. Then, host 3 adds either host 1 or host 2 to his predefined host list, no matter which one. Now we have 3 nodes participating. The node 4 can add any one of these 3 to his predefined host list, no matter which one. The same thing with node 5.

This means that it is not necessary to add every single node to ones predefined host list. As long as there is connection from at least one of its predefined hosts to the 3rd one, every single of these hosts become visible to each other. Sure, the most reliable way is to add every single host you know in the swarm with this share to your predefined host list, but it is not an absolute requirement.

It also means that in order for you to create a private swarm, you don't need to enter every single host as predefined host. You may simply use the same single host to serve as connector between all the nodes in the swarm. So, all that every node needs to know is a single predefined host that is ideally the same for all of them. But it does not necessarily have to be the same host for all of them. As long as it can chain to the 3rd host, that is enough.

But it is advisable to have more than one predefined host for redundancy purpose. The more predefined hosts you have on your list, the more reliable your network becomes. But they do not have to be exactly the same hosts on all the nodes.


Use relay server when required

Well, in few words, relay server is basically a proxy server.

Use relay server when required: Use a relay server when it is impossible to directly connect to other devices due to NAT issues.

http://www.qnap.com/en/index.php?sn=9136
(BitTorrent Sync Installation & Setup Tutorial)

Here's what RomanZ of BT Sync Support has to say about it answering various questions:

Q: What is the definition of "required"? In which cases some node becomes "required"? What is the exact behavior of btsync if this checkbox is checked? It seems to imply that the information physically flows through the relay servers.

A: Relay server is required when peers can't establish direct connection. Relay server has a public IP, so any peer can connect to it. Information physically flows thru relay server, when it is used. This is rather slow, though.

Q: What has to be done in order for some host to become the relay server? Does it have to merely run the btsync? Does it have to have the particular share available on it locally or will this search be performed even if that node does not actually have the particular share available on it locally?

A: Host can't be a relay. Relay server transfers ANY data and has no secrets, while peer may transfer data to other peers only if it has relevant secret.

Q: Do the relay servers have to run the btsync program in order to behave like a relay server?

A: No. Relay server runs specific software, as it has no secrets and needs just to proxy all the data between peers.

(RomanZ, Sync Support).

http://forum.bittorrent.com/topic/17736-own-tracker/#entry80281
(own tracker thread)


Use tracker server

Here's what RomanZ of BT Sync Support has to say about it answering various questions:

Q: But how does btsync know the IP:ports of BT trackers if they are not defined as user modifiable parameter? This seems to imply that btsync node "under the hood" communicates with BT network to the known and internally predefined hosts unless BT tracker IP:port addresses are discovered via UDP broadcasts, in which case, how exactly does it work?

A: BTSync has hardcoded DNS names. "r.usyncapp.com" for relay server, "t.usyncapp.com" for tracker server. First resolves to 2 IPs, second - to 3 IPs.

Q: The side effect of it may be that BT, being forever present in all the btsync transfers may be ordered at some point (by NSA, for example) to collect and archive all this information and there is no guarantee that they will make it publicly known.

A: All information sent to tracker is a hashes of some secrets and IPs which can be contacted to obtain it. Hash gives no idea what info is synced, even if you have a huge collection of hashes from same IPs.

Comment: Well, kinda. But there is "but" to this. It seems that hash is a hook to the share and all the participants in the swarm. It is all one needs. Once you have hash, you can then hook into the swarm directly, just like any other BTSync app and look at the information directly. Or you can capture the traffic for a given share and if the encryption is not as good as one might think it is, you have "the rest of the story". Isn't this the case?

Q: This means that using btsync you are always under the "watchful eye" and all your moves and all your content is logged by those who are interested and have some influence on BT (guess who), and we are not going to mention the NSA type of agencies here. :)

A: Tracker server has no your data, as I mentioned above. Relay has no keys and can be easily disabled. BT has no technical way to monitor your data.

Comment: Well, kinda. But why would they want to in the first place? But what we are talking about here, is not BT itself. We are talking about the "evil most profound", and all sorts of agencies and clowns of all kinds that have evil intents in their mind. It is highly doubtful that BT itself would be interested in snooping on your traffic. For what? But what IS technically possible is simply passing the information that passes through BT to the 3rd party.

Q: Finally, what are the exact settings in order to EXCLUDE the BT trackers so that they would not be able to interfere and/or snoop on your traffic? Is it possible at all in light of the "search DHT network" checkbox setting?

A: Disable tracker, relay, DHT. BTSync will work fine in LAN, to force it syncing with someone else you'll need to use pre-defined hosts and do a port forwarding manually if your other peers are behind NAT.

Q: Because, security-wise, if you disable the "use the BT tracker" option, you may get a false sense of security. But, internally, there is absolutely nothing that prevents BTSync from still contacting the trackers, or ANY BT hosts for that matter, for as long as they remain prewired.

A: Why can't? Use firewall. Use custom DNS server. Use hosts file.

Q: ... And thanks for the ideas about how to block the IP using a firewall for example. Except, in general public application I can only block it to my own node, while all other nodes remain wide open.

A: It hardly depends on your firewall. If FW supports Application Control - you can set a rule for BTSync to let it out only to hosts you define in predefined hosts section. Or just block all DNS names / IPs used by BTSync.

http://forum.bittorrent.com/topic/17736-own-tracker/#entry80281
(own tracker thread)

Setting up private secure networks

Well, the questions of security is a huge issue to discuss. It turns out that even if you encrypt your traffic, the encryption algorithm may not be as secure as you might think. In particular, it has been recently revealed that one of the major players in the SSL business, allegedly as a result of their cooperation or funding from the NSA, has released an encryption scheme that can be easily cracked. Because their random generator is in fact not as random as it should be. This, in turn, implies that such an algorithm may be cracked relatively easily by brute force approach. But this is a long subject to discuss.

Nevertheless, here is an interesting quote with plenty of ramifications, which leave little doubt that BT is indeed keeping this issue silent and does not comment on it instead of making a public statement:

Re: Proposal: Open source Bittorrent Sync clone in Go

From: Starfish
Date: 28.01.14 9:08

I experienced something that made me believe Bittorrent is indeed working with the NSA.

A while back I posted a thread on their forum: http://forum.bittorrent.com/topic/25831-are-you-working-with-the-nsa/

There were many threads in the forum, and most received comments from Bittorrent staff. My thread was among the most read, but no answer.

After a some weeks I went back and gave the thread a bump. A while later the thread was deleted; none of my other threads were.

Re: Proposal: Open source Bittorrent Sync clone in Go
https://groups.google.com/forum/?_escaped_fragment_=msg/golang-nuts/7WUj3nASuLo/Uq2268VmVzQJ#!msg/golang-nuts/7WUj3nASuLo/Uq2268VmVzQJ

So, some questions arise naturally.

1) Why BT never commented on that thread?

2) Why have they never made a public statement on this issue?

3) Why did they delete that thread? And this one is probably the most revealing one of all, even though it seems to say nothing. But, sometimes nothing may turn into everything...

If you "put two and two together", the answer becomes simply inevitable...

And about the simplest way for the NSA to snoop on what you transfer is not necessarily by decrypting the traffic, but merely knowing the "secret words". At that point, they can simply look at your data directly, without even wasting the resources on decrypting it, unless the data itself is encrypted with a reliable algorithm. And that is the easiest thing in the world.

This, in turn, simply means that BT itself does not even have bother to look at your traffic. The very fact they know your hashes is enough. And if BT knows them, then how much of a difficulty would it be for the NSA and other agencies to also know it?

Nevertheless, besides the issues of encryption and cracking in general, regardless of how secure BTSync claims to be, there are some things you can do if you are interested in setting up a secure private network that would leave no footprints of itself on the net beyond the fact that it exchanges the information on the net.

If you have a number of hosts on the net with fixed IP and you are interested in exchanging the information only between them, which is not available to general public, then it is possible to do it with BTSync. It has several configuration parameters that allow you to enable or disable some features.

For example, even though BT claims that they do not collect any information on the transfers that are managed through their tracker(s), still, they have enough information and are capable of not propagating the information between the hosts. Whether they would actually do it or not is irrelevant and quite arguable from several angles. But the fact is: if you rely exclusively on BT trackers, then there is no guarantee that you will see any or all hosts that actually try to connect to your share if BT does not propagate some traffic.

This problem can be solved on private networks, especially if you know the IP or domain names of your hosts. But for dynamic DNS access things become more difficult and in some cases impossible.

So, how do you create a private and secure network that is not going to inform and rely on BTSync trackers for transfers?

Well, there is a concept of "predefined hosts" in BTSync in Folders -> Folder Preferences -> Properties. If you check the "Use predefined hosts" checkbox, then you can enter a list of host IPs or host names and ports to be used to communicate between the hosts directly. In that case, you don't need the BT tracker. And you do not need to do the DHT broadcasts to discover other interested hosts. And you don't need the relays to communicate with each other.

This means that you virtually leave no footprint on the net while transferring the information between those hosts.

In this case, you simply enter the names or IPs of all the hosts on your private network and they all will be able to transfer the information between each other without leaving any footprint on the net, and BT tracker won't even know you exist. Because you do not do any DHT broadcasts and all your hashes and shares are known only to your hosts.

As to "Use relay server when required", all the ramifications are not quite clear. Logically, the relay means that all the traffic flows through them, not only the connection information, in which case, if the encryption algorithms in BTSync are not that trustworthy, then, technically, your traffic can be relatively easily decrypted by the NSA super-computer, for example, via brute force method.

See also:

http://forum.bittorrent.com/topic/21048-defensive-open-source/#entry55992

On Windows

So, on Windows, your settings for various checkboxes would be:

So, the tracker would not know about you (but who knows what really happens "under the hood") unless you enable the DHT network, so the tracker will not be able to read your broadcasts. Otherwise, if you rely on tracker, it might decide not to present the lists of connected nodes and so they won't see each other.

On Linux

On Linux, things are configured via configuration files. The default location on Ubuntu is /etc/btsync/debconf-default.conf

Its location is specified on the command line to btsync process:

~/btsync/install/btsync --nodaemon --log file --config /etc/btsync/debconf-default.conf

... I cannot get the server to join the syncing party. In my server configuration I am trying to setup a user configuration rather than the default one. The secret is the same in all instances.

Apparently I can't see anything wrong in your configuration file. But you must make sure, that the firewall of the server does not block the required connections. Since you have no chance to see anything, I would suggest to make a backup copy of your configuration file, remove completely the shared_folders section and activate the web UI in order to get a better overview of what is happening (your shared folder will still exist, since btsync has stored it into its internal database).

After you solved your problem, you can deactivate the web UI and write the shared_folders section back.

[Because if you have the "shared_folders" section defined in your config, you won't see those shares in the web UI.]

Here an example of one of my config files: (lines with leading // means a comment and can be used to comment out certain parameter settings.)

//!/usr/lib/btsync/btsync-daemon --nodaemon --config
// or, if you use the btsync version provided in xxx.gz file, then use this instead:
// ~/location_of_btsync_binary/btsync

// This btsync instance provides sync services for
// all common files in the YeaSoft Server Grid
{
	"device_name" : "yeasoft-gate2 - Server Sync",

// "storage_path" is location of btsync configuration files, logs and databases
// for all currently running shares, default on Ubuntu: /var/lib/btsync
	"storage_path" : "/var/lib/btsync-serversync",

// If "listening_port" is set to 0, then every time you start btsync it will use a random port,
// which you do not want if you want to specify the hosts directly
// and avoid using btsync tracker
	"listening_port" : 22144,
	"check_for_updates" : false,
	"use_upnp" : false,
	"download_limit" : 0,
	"upload_limit" : 0,
	"lan_encrypt_data": true,
	"lan_use_tcp" : true,
	"rate_limit_local_peers" : false,
	"webui" : {
//		"listen" : "10.65.0.16:8889",
//		"login" : "Administrator",
//		"password" : "MyTestPassword"
	}
	,

	"shared_folders" : [ {
		"secret" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
// dir is a top level directory of your data folder
		"dir" : "/mnt/data/tftproot/yeaboot",
		"use_relay_server" : false,
// If "use_tracker" : false and "use_dht" : false, then BT tracker
// won't even see your node (if you are concerned with security issues) 
		"use_tracker" : false,
// DHT allows a discovery of other nodes without btsync tracker
		"use_dht" : false,
		"search_lan" : true,
		"use_sync_trash" : true,
		"known_hosts" : [
//			"10.65.0.16:22144",
			"10.65.4.1:22144",
			"10.65.6.1:22144",
			"10.65.8.1:22144",
			"10.65.16.1:22144",
			"10.65.18.1:22144"
		]
	}
	]
}

As you can see, I made the same test: the web UI is disabled with comments... If you activate it, you have to comment out the shared folder section. In my special case, I disabled also all external helpers/relays and work only with predefined hosts.

(tuxpoldo, Advanced Member)

Debian And Ubuntu Server Unofficial Packages For Bittorrent Sync (thread)
http://forum.bittorrent.com/topic/18974-debian-and-ubuntu-server-unofficial-packages-for-bittorrent-sync/page-22#entry79223

Linux related BTSync useful information and discussions

Here is a thread on BTSync forums that contains a lot of useful information:

Debian And Ubuntu Server Unofficial Packages For Bittorrent Sync (thread)
http://forum.bittorrent.com/topic/18974-debian-and-ubuntu-server-unofficial-packages-for-bittorrent-sync/

Installing BitTorrent Sync on the Raspberry Pi (Linux)

[Note: these instructions are generally applicable to any Linux system.]

NOTE: I am using Raspbian Wheezy, but I suspect these ideas will translate to other operating systems.

[Note: These instructions are not using the latest version of BTSync. As of this writing (March 26, 2014), the latest version is 1.3.67. So, the wget command below is not likely to be correct for the current version.

The original BTSync authorized latest versions are located here:

http://www.getsync.com/download

See this thread to see the list of fixes and features:]

http://forum.bittorrent.com/topic/28597-bittorrent-sync-13/#entry80671

You need to grab the ARM build of BitTorrent Sync:

mkdir ~/.btsync && cd ~/.btsync
wget http://btsync.s3-website-us-east-1.amazonaws.com/btsync_arm.tar.gz
tar -xvf btsync_arm.tar.gz

Unless you have a large SD in your Pi you will probably want to use an external drive for your sync location. I will be using a USB thumb drive. You may need to format your thumb drive as EXT4 if you are having issues. (WARNING! This will erase all data on your drive)

df -h                               # find your drive here, e.g. `sda1`
sudo umount /dev/sda1               # replace sda1 with your drive name !
sudo mkfs.ext4 /dev/sda1 -L BTSync  # replace sda1 with your drive name !

Now all you have to do is launch the btsync application and you will be up and running!

cd ~/.btsync
sudo ./btsync  # can be killed with `sudo killall btsync`

If you see the following output btsync is properly running.

BitTorrent Sync forked to background. pid = 3003

Navigate your browser to Your-Pi-IP-Address:8888/gui to manage the btsync process. To add the thumb drive select "Add Folder" and navigate to /media/BTSync. You will need to generate a secret as this is the first time you are adding the folder.

BTSync web interface

BTSync Add Folfer


Other Devices

Now go download the Android app and/or the desktop app and connect them using the secret you just generated It's that simple! Any change on any device will be synced across all online devices. If you keep your Pi online it will store and push the most up to date content as your other devices go on and off line.


Start at Boot

You may want to set btsync to start when you boot your Raspberry Pi. To do that we will place a script in /etc/init.d/ and then register it with update-rc.d.

sudo nano /etc/init.d/btsync

Paste the following code in the script

#! /bin/sh
# /etc/init.d/btsync
#

# Carry out specific functions when asked to by the system
case "$1" in
start)
    /home/pi/.btsync/btsync
    ;;
stop)
    killall btsync
    ;;
*)
    echo "Usage: /etc/init.d/btsync {start|stop}"
    exit 1
    ;;
esac

exit 0

Then change the permissions, test, and register it to run at boot:

sudo chmod 755 /etc/init.d/btsync
sudo /etc/init.d/btsync start       # test that the script starts
sudo /etc/init.d/btsync stop        # test that the script stops
sudo update-rc.d btsync defaults

Password Protect Web Interface

If you expose your web interface to the outside world (or if you don't trust your friends) you are going to want to password protect it. This can be done with a btsync config file passed to the executable at runtime with the --config flag. First, use btsync to generate a sample config file, modify it to fit your needs, and restart the process.

cd ~/.btsync
./btsync --dump-sample-config > btsync.conf
# browse the sample config file and change what you want
sudo killall btsync
sudo ./btsync --config btsync.conf

HINT: Use jsonlint to validate your config file if btsync complains. Also make sure to modify the /etc/init.d/btsync script to use the config file as well.

Questions? Hit me up on twitter @jackminardi

You can also follow the discussion on HN

Source:

Replace Dropbox with BitTorrent Sync and a Raspberry Pi
http://jack.minardi.org/raspberry_pi/replace-dropbox-with-bittorrent-sync-and-a-raspberry-pi/

Configuration

Depending on how you installed btsync and how you are running it, some things will be different. Typically, you do the fresh install from some package, which performs all the default directory creation and installs the default scripts and some configuration files.

[Later on, you may download the btsync update, which contains executable only version, (such as btsync_i386-1.2.91.tar.gz for version 1.2.91, the latest version as of Mar 7, 2014, which does not include any initial install/configure files and should not be normally used as initial install, unless you are desperate.]

If you installed btsync via some package, like xxx.deb, then it should have created the directory for the btsync databases and configuration files and installed the default /etc/init.d/btsync init script for you, and you can start/stop btsync by /etc/init.d/btsync [start|stop|restart] shell script, which will automatically provide the correct arguments to btsync process when it is started.

See also: BTSync versions: user vs. server

You can also provide your own configuration file on the command line to btsync if you are running it manually and not via init script.

BTSync uses the configuration files that contain user configurable parameters. If you installed btsync from some package, like .deb, your default configuration and/or btsync data files (not the location of your sync folder, but the location of btsync database and configuration files) will be located in the "storage_path" parameter of your default or custom made configuration file.

[from /etc/btsync/debconf-default.conf]
// "storage_path" is location of btsync configuration files, logs and databases
// for all currently running shares, default on Ubuntu: /var/lib/btsync
"storage_path" : "/var/lib/btsync",

To get the location of your currently running configuration file, use ps -ef | grep bts and look at --config parameter to the btsync process (default is /etc/btsync/debconf-default.conf).

The location of the config file depends on how you installed and are running the btsync and whether you have /etc/init.d/btsync init file (Ubuntu), which is used to start/stop btsync after reboot automatically.

See also:
Configuration file contents and meaning

How To Use BitTorrent Sync to Synchronize Directories in Ubuntu 12.04

By Justin Ellingwood

Source:
How To Use BitTorrent Sync to Synchronize Directories in Ubuntu 12.04
https://www.digitalocean.com/community/articles/how-to-use-bittorrent-sync-to-synchronize-directories-in-ubuntu-12-04

Introduction

Syncing folders and files between computers and devices can be done in many different ways. One method for automatically syncing content is BitTorrent Sync. BitTorrent Sync is a method of synchronizing content based on the popular BitTorrent protocol for file sharing.

Unlike traditional BitTorrent, files shared using BitTorrent Sync are encrypted and access is restricted based on a shared secret that is auto-generated. While BitTorrent proper is often used to distribute files in a public way, BitTorrent Sync is often used as a private method to sync and share files between devices due to its added security measures.

In this guide, we will discuss how to install and configure BitTorrent Sync on two Ubuntu 12.04 VPS instances.

Install BitTorrent Sync

To begin, we will need to install BitTorrent Sync on both of our Ubuntu 12.04 instances. If you would like to install BitTorrent Sync on your local computer to allow you to sync with your server, you can find the binary packages here.

BitTorrent Sync is relatively easy to install on Ubuntu 12.04, but it is not included in the default repositories. We can use a PPA (personal package archive) so that we can have access to a maintained BitTorrent Sync repository and manage it with our normal apt tools.

Ubuntu 12.04 includes the PPA tools in a package called python-software-properties, which we can download through apt:

sudo apt-get update
sudo apt-get install python-software-properties

After this is installed, we can add the PPA that contains updated Ubuntu packages:

sudo add-apt-repository ppa:tuxpoldo/btsync

Press "enter" to add the new PPA.

Once the new repository is added, we should update apt to build a package index for the new source, and then install the BitTorrent Sync software:

sudo apt-get update
sudo apt-get install btsync nginx

You will be asked quite a few questions in prompts when you attempt to install. For now, press ENTER through all of the prompts. We will be reconfiguring our services momentarily in a more in-depth manner.

Configure BitTorrent Sync

Now that the software is installed, we're actually going to run the configuration script that prompts us for values a second time. This time, however, we will have access to additional options that we require for our purposes.

To run the script again, this time choosing our settings, type this on each server:

sudo dpkg-reconfigure btsync

The configuration directory for the service is:

/etc/btsync

Do not edit the config file generated by the menu system by hand. You can, however, copy the configuration to use as a template for another configuration if you'd like to adjust details not covered in the menu configuration.

This will run you through even more prompts than during the initial installation. For the most part, we will be going with the default values and you can just press ENTER.

Below, I've outlined the values that you need to configure:

As you can see, for the vast majority of settings, we can accept the defaults. The above choices though are very important. If you mis-configure these, run the command again to correct your selections.

Configure SSL Front-end to the BitTorrent Sync Web Interface

Now, we have BitTorrent Sync set up for the most part. We will set up our sync directories in a bit. But for now, we need to set up our nginx web server with SSL.

You may have noticed that we configured our web interface to only be available on the local loopback interface (127.0.0.1). This would normally mean that we would not have access to this when running BitTorrent Sync on a remote server.

We restricted access like this because, although the BitTorrent Sync traffic itself is encrypted, the traffic to the web interface is transmitted in plain text. This could allow anyone watching traffic between our server and local computer to see any communication sent between our machines.

We are going to set up nginx with SSL to proxy connections through SSL to our BitTorrent web interface. This will allow us to securely administer our BitTorrent Sync instance remotely.

Again, we will need to do all of these steps on both of our hosts.

Generate the SSL Certificate and Key

The first step towards getting this set up is to create a directory to hold our SSL certificate and key. We'll do this under the nginx configuration directory hierarchy:

sudo mkdir /etc/nginx/ssl

Now, we can create our SSL certificate and key in a single motion by issuing this command:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/nginx.key -out /etc/nginx/ssl/nginx.crt

You will be asked to fill out some information for your certificate. Fill out the fields as best as you can. The only one that really matters is this one:

Common Name (e.g. server FQDN or YOUR name) []:

In this field, enter your server's domain name or public IP address.

Configure Nginx to Encrypt Traffic with SSL and Pass to BitTorrent Sync

Now, we can configure our nginx server blocks to use our SSL certificates when communicating with remote clients. It will then the information to our BitTorrent Sync web interface listening on the local interface.

We will leave the default nginx server block file intact in case you need to use this in the future. Since BitTorrent Sync operates on port "8888" by default, we will use this as the front-end SSL port as well.

Create a new server block file by opening a new file with sudo privileges in your editor:

sudo nano /etc/nginx/sites-available/btsync

Inside, we need the to add the following lines:

server {
    listen server_domain_or_IP:8888 ssl;
    server_name server_domain_or_IP;

    access_log /var/log/nginx/access.log;

    ssl_certificate /etc/nginx/ssl/nginx.crt;
    ssl_certificate_key /etc/nginx/ssl/nginx.key;

    location / {
        proxy_pass http://127.0.0.1:8888;
    }
}

Make sure you change the red text to your server's domain name or public IP address. This will tell nginx to bind to the same port that the BitTorrent Sync web interface is using on the local interface. The difference is that nginx will use the public address and require SSL.

It will use the SSL certificate that we created to encrypt the traffic to the client. It will then pass it to the BitTorrent Sync interface. In this way, the traffic between the server and the client will be encrypted, but the BitTorrent Sync interface will operate as if we were accessing it from the server itself.

When you are finished, save and close the file.

Now, we just need to link the file so that it will be enabled:

sudo ln -s /etc/nginx/sites-available/btsync /etc/nginx/sites-enabled/

We can now restart the service to implement our changes:

sudo service nginx restart

Make sure you go through these procedures on each of your two servers.

How To Configure Shared Folders

In order to sync folders with BitTorrent Sync, the btsync user or group needs write access to the folders. There are a few different ways to achieve this.

First, let's create the folder:

sudo mkdir /shared

We need to complete these steps on both of your VPS instances that will be syncing.

Giving the btsync Process Full Ownership

One way to give the btsync user access is to simply give ownership of the folder to the btsync user:

sudo chown btsync:btsync /shared

This will allow the BitTorrent Sync service to correctly serve the contents of this directory, but we are not able to write to this as a normal user. This may be what you want, but it usually is not.

Give Your Normal User Ownership and the btsync Process Group Ownership

If you have only one normal user on the system, you can give that user ownership of the folder and give the btsync group ownership of the folder:

sudo chown your_user:btsync /shared

You would then have to give the group write permissions:

sudo chmod 775 /shared

This will allow the btsync service to access the folder. However, any files created inside the directory will be owned by your user and group.

For instance, if we add a file to this folder called test, it would be completely owned by our user:

cd /shared
touch test
ls -l
-rw-rw-r-- 1 your_user your_user 6 Jan 16 14:36 test

This will cause problems for the syncing, because the btsync process cannot modify the file. We want to give it the same group permissions as the folder it is in so that the process has write access.

We can do this by setting the SGID bit on the directory. This will set the group on all new files created inside of the directory to the group of the directory itself. This will allow proper write access to modify things:

sudo chmod g+s /shared

Now, when we create a file, it will be given the permissions of the directory:

cd /shared
touch test2
ls -l
-rw-rw-r-- 1 your_user your_user 6 Jan 16 14:36 test
-rw-rw-r-- 1 your_user btsync 0 Jan 16 14:41 test2

This goes a long way towards getting the appropriate functionality, but it isn't quite right yet.

Delete the test files we created before continuing:

rm /shared/test*

Add Your User to the btsync Group and Give the Root User Ownership

The method above works somewhat, but files that are transferred with BitTorrent Sync are owned by the btsync user and group. This means that currently, any files synced by the service will not be editable by us.

We can change this by adding our user to the btsync group. This will allow us to modify files that are writeable by the btsync group, which is what we want.

Add any username that you wish to be able to use btsync to the btsync group:

sudo usermod -a -G btsync your_user

This will append the btsync group to your user's group definition. This will allow you to edit files created in the shared folder by the btsync process.

However, the directory is still owned by our user, which is not a good way of going about things if we have multiple users on the system. We should transfer ownership to the root user to avoid regular users changing folder settings. We should also allow group write permissions so that anyone in the btsync group can add content:

sudo chown root:btsync /shared
sudo chmod g+w /shared

You may have to log out and log back in for these changes to take affect.

In the end, the process for creating a shared folder that works well for BitTorrent Sync goes something like this:

sudo mkdir shared_folder
sudo chown root:btsync shared_folder
sudo chmod 2775 shared_folder
sudo usermod -a -G btsync your_user

The first "2" in the chmod command sets the SGID bit in the same way that the "g+s" did previously. This is just a more succinct way of combining these commands.

Accessing the BitTorrent Sync Web Interface

Now that we have a folder that is configured appropriately for BitTorrent Sync sharing, we can access the web interface to add our folder to begin syncing.

Again, we will have to do this on each of the servers that we wish to configure syncing on.

Access the web interface by going to your droplet's IP address, followed by the port you configured during install. By default, this is 8888:

your_ip_or_domain:8888

BitTorrent Sync login

You will have to sign in using the credentials you configured during installation. The default username is admin if you didn't change it.

You will be presented with a rather simple interface to start:

BitTorrent Sync main page

Adding the Shared Folder to the First Droplet

Now that we are in our web interface, we can add our shared folder so that the btsync process can register it.

On your first machine, click on the "Add Folder" button in the upper-right corner. This will bring up a box that allows you to select a directory to share:

BitTorrent Sync add folder

Find the folder that you configured for sharing. In our case, this is the /shared folder. Once you have a folder selected, you should click on the "Generate" button to generate a secret for the folder.

BitTorrent Sync generate secret

The secret that is generated allows you to sync this folder with another instance of BitTorrent Sync. This unique value is basically a password to allow the two services to connect to each other.

Click the "Add" button when you have completed these steps. This will add our folder to the interface and give you some buttons on the side to manage this folder.

BitTorrent Sync buttons

Right now, we are only interested in the "Secret/QR" button. Click this to bring up a box that will allow you to choose how you want to share this folder.

We can grant access to the folder with read/write permissions through "Full access". If we only want to sync one way, like a backup, we can allow only read access. The secrets that are provided for each kind of access differ.

Copy the secret for the type of access you want. We will be using full access in this tutorial.

Adding the Shared Folder and Secret to the Second Droplet

Now that we have our secret from our first VPS, we can add the shared folder that we created on our second VPS and use the secret to sync our files.

First, you must log into the web interface just like you did with the first server:

second_ip_or_domain:8888

Once you are to the interface for the second server, click the "Add Folder" button again.

Add the locally created shared folder.

This time, instead of clicking the "Generate" button, we will paste the secret from the other instance into the "Secret" box:

BitTorrent Sync pasted secret

Click the "Add" button to create the share.

After a moment, in both web interfaces, you should see some new information in the "Connected devices and status" section:

BitTorrent Sync connected devices

This means that our two instances of BitTorrent Sync have found each other! The icon in front means that we have given full access and files will be synced in both directions.

Test Syncing

Now that we have configured syncing, let's test to see if it works.

On one of your servers (it doesn't matter which one if you configured full access), we will add some files to our shared folder.

As a user that has been given access to the btsync group, create some files in the shared directory:

cd /shared
touch file {1..10}

This will create 10 files in the shared directory. We can check that these have been given the appropriate permissions by typing:

ls -l
-rw-rw-r-- 1 your_user btsync 0 Jan 16 16:16 file1
-rw-rw-r-- 1 your_user btsync 0 Jan 16 16:16 file10
-rw-rw-r-- 1 your_user btsync 0 Jan 16 16:16 file2
-rw-rw-r-- 1 your_user btsync 0 Jan 16 16:16 file3
-rw-rw-r-- 1 your_user btsync 0 Jan 16 16:16 file4
. . .

As you can see, the files are owned by your user, but the group owner is btsync. This is exactly what we want.

If we check our other server after a few seconds, we should see our files in our shared directory!

cd /shared
ls -l
-rw-r--r-- 1 btsync btsync 0 Jan 16 16:16 file1
-rw-r--r-- 1 btsync btsync 0 Jan 16 16:16 file10
-rw-r--r-- 1 btsync btsync 0 Jan 16 16:16 file2
-rw-r--r-- 1 btsync btsync 0 Jan 16 16:16 file3
-rw-r--r-- 1 btsync btsync 0 Jan 16 16:16 file4

As you can see, the files are given to the btuser and group. This is because the service can't be sure that the original username exists on the second system.

The last step is to get the btsync daemon to automatically set the file permissions of the files that it syncs to be writeable by the btsync group. This is necessary if you are providing full access in order for your user to edit the files that it has synced.

We can do this in by reconfiguring the btsync daemon. This will open up a lot more options than we were given when we originally went through the configuration. Begin to reconfigure on both of your syncing machines by typing:

sudo dpkg-reconfigure btsync

You will run through the configuration menu, this time with many more questions. For the most part, it should automatically select either your previous selection, or the default choice for any previously unset parameters. One question that you'll have to remember not to skip is the password prompt.

The option that you are looking is the default umask for files synced by the daemon.

We can set the appropriate umask to create files that are writeable by both the owner and the group (which our user is a part of), by typing this:

002

Finish up the configuration and the daemon should restart automatically with the new settings. Once you have completed this task on both servers, you should be able to create a new file on one server, and it will be given correct writeable permissions on the second host:

touch /shared/write_test

On the second host once the file syncs, you'll see somethings like this:

-rw-rw-r-- 1 btsync btsync  0 Jan 30 10:44 write_test

In the web interface, you will not see that your files have been synced, because the files don't contain any actual data. If we add some content to our files, the interface will update to show how many files we have synced:

for item in /shared/file{1..10}; do echo "some content" > $item; done

BitTorrent Sync file size

Conclusion

Now that you have your servers configured, you can easily transfer files between them. You can also configure multiple folders to automatically sync. This can provide some interesting options for dealing with configuration files and such.

The application is fairly flexible in how to sync between multiple computers. You can create one-time secrets to ensure that no one shares access to your directories, you can share only with specific hosts, and sync between your mobile device. BitTorrent Sync provides an archive version control system through the .SyncArchive file in directories and can rate limit to ensure that you have bandwidth available for other applications.

Install BitTorrent Sync on a Headless Ubuntu Server

[Note: this information is applicable if you downloaded an update to btsync, which only contains the btsync executable and does not perform any initial configuration and does not include the default scripts and does not create any directories for you.]

... At time of publishing, the latest version of BitTorrent Sync is 1.0.134.

Before we get going, I’ll assume you’re server is running a current version of Ubuntu, I’m using 12.04.2 LTS (Precise).

You’ll first want to create a directory for BT Sync to run in as the current download only contains the executable btsync and LICENSE.txt

$ mkdir ~/.btsync

Next you’ll want to actually download the files from the BitTorrent site.

For the 64-bit version of BT Sync use this command:

$ cd ~/.btsync && wget -O - "http://syncapp.bittorrent.com/1.0.134/btsync_x64-1.0.134.tar.gz" | tar xzf -

Or use this one if your architecture is i386:

$ cd ~/.btsync && wget -O - "http://syncapp.bittorrent.com/1.0.134/btsync_i386-1.0.134.tar.gz" | tar xzf -

After using that previous command, you should now be able to see the executable file and the license file in that directory you just created.

Run the following to check the version of your executable and see the options:

$ ~/.btsync/btsync --help

You’ll want to ensure that the files you’ve just downloaded have the correct permissions set for them.

Now you’ll want to create a configuration file for the executable to use. BT Sync comes with an option to generate a sample file by using the following:

$ ~/.btsync/btsync --dump-sample-config > ~/.btsync/sync.conf

Once you’ve generated a sample configuration file, you’ll need to open it and edit it. At this stage you may want to do the following to get the a test folder ready for a sync.

But before that you may want to generate a secret for the folder you’re going to want to sync.

BT Sync gives you two options for creating a secret. The default secret for a folder (which every folder you want to sync will need to have) gives a linked device full permissions on that folder resulting in any changes in a linked folder to be replicated across all other linked folder(s).

The alternative to the full permissions secret is a read-only secret, which means that any changes on a linked remote folder(s) will not affect the source folder or any other linked folders.

Use the following to generate a unique secret:

$ ~/.btsync/btsync --generate-secret

That command should output an a 32 character alphanumeric string with uppercase alpha characters.

For example: A1B2C3D4E5F6GHIJKLMNOPQRSTUVWXYZ

You’ll want to make a note of the secret that was created. You’ll need it later.

If you want to generate a read-only secret, you’ll need to pass the secret you generated to the command:

$ ~/.btsync/btsync --get-ro-secret A1B2C3D4E5F6GHIJKLMNOPQRSTUVWXYZ

That will then output another, slightly longer alphanumeric secret which will permit read-only access to the folder.

While you can link to any folder on your system, it helps to keep things neat and tidy by creating a separate folder that holds all the folders you might want to sync.

$ mkdir ~/BTSync && mkdir ~/BTSync/example.com

That creates two folders one in the other. The second mkdir creates a folder called example.com, which could be anything but in this example, it creates a folder that we’ll store some backup files in for our website example.com

Now that you have a folder path and a secret we can go ahead and begin editing the BT Sync configuration file. You may want to keep a backup of the sample configuration file, just in case.

$ cp ~/.btsync/sync.conf ~/.btsync/sync.conf.orig
$ vi ~/.btsync/sync.conf

Configuration file contents and meaning

Once you’ve opened the sample configuration file, you’ll see that it is in JSON format and heavily annotated so you should be able to work your way through it. The key things to remember are the following:

  • device_name: change the this to something that will help you identify the server.
  • listening_port: Define a unique and unused port for the service to listen on.
  • storage_path: This will be the full absolute path to the .btsync folder you created earlier. If you’re not sure, you can go to the folder and type pwd.
  • pid_file: You can leave this as it is, but I like to keep everything in once place so I change it to be within the storage_path and uncomment the line by deleting the two leading forward slashes.
  • check_for_updates: Change to false as its not needed on a server.
  • use_upnp: Change to false as its not needed on a server.
  • webui/listen: Comment out the line “listen”: “0.0.0.0:8888″ to turn off the web user interface as you won’t need it. Use two forward slashes (//) to comment out a line.

    [This is not quite true. Web interface may be used regardless of anything. Here, 8888 means the port through which WebUI will connect to the btsync server to exchange data. 0.0.0.0 mans: use any IP address on which the btsync server is located (applicable to cases where btsync is running on a server with a dynamic IP.]
  • shared_folders: At approximately line 40 in the file, before the shared_folders list, there may be the start multi-line comment (/*) that you’ll have to remove. Don’t forget the corresponding closing comment line (*/) at approximately line 63.
  • shared_folders/secret: Paste the secret you generated earlier here.
  • shared_folders/dir: This needs to be the absolute path of the folder you want to sync.
  • shared_folders/use_relay_server: Change to false.
  • shared_folders/use_tracker: Change to false.
  • shared_folders/search_lan: Change to false.
  • shared_folders/known_hosts: Enter each linked device in the format of IP/host a colon (:) and then the port number. Add multiple devices by separating each string with a comma (,).

    [Here, you can show a list of all the participating known nodes that have static IP. In this case, they will not only participate in syncing, but will also serve as trackers and will provide the list of all connected hosts to this share (secret).]

Once the changes have been made and your happy the document is valid JSON, you can save your changes.

Your configuration file will likely look something like this (NB: I’ve stripped out all the annotations):

{ 
  "device_name": "web server",
  "listening_port" : 8888,
  "storage_path" : "/root/.btsync",
  "pid_file" : "/root/.btsync/btsync.pid",
  "check_for_updates" : false, 
  "use_upnp" : false,
  "download_limit" : 0,                       
  "upload_limit" : 0, 
  "webui" : {
    //"listen" : "0.0.0.0:8888",
    "login" : "admin",
    "password" : "password"
  },
  "shared_folders" :
  [
    {
      "secret" : "A1B2C3D4E5F6GHIJKLMNOPQRSTUVWXYZ",
      "dir" : "/root/BTSync/example.com",
      "use_relay_server" : false,
      "use_tracker" : false,
      "use_dht" : false,
      "search_lan" : false,
      "use_sync_trash" : false,
      "known_hosts" : 
      [
        "101.102.103.104:8888"
      ] 
    }
  ]
}

Now to start BT Sync running, run this:

$ ~/.btsync/btsync --config ~/.btsync/sync.conf

You should then see the following output:

By using this application, you agree to our Privacy Policy and Terms.

http://www.bittorrent.com/legal/privacy

http://www.bittorrent.com/legal/terms-of-use

BitTorrent Sync forked to background. pid = 27732

The pid will likely be different number. That’s what you’ll need to kill the process should you need to modify the configuration file.

If you see an error, this is most likely due to an invalid JSON file or a folder path being incorrect. Check both and try again.

If you need to kill the BT Sync instance you can do that by using the following command.

$ kill -15 27732

You’ll notice the number 27732 is the pid that BT Sync mentioned when I started it. If you forget you can check the .pid file in the ~/.btsync directory:

$ cat ~/.btsync/btsync.pid

Once it’s stopped, make your changes and get it running again:

$ ~/.btsync/btsync --config ~/.btsync/sync.conf

Next, make sure you setup the linked device to start syncing a folder on the server you have just setup.

Once that is ready, try dropping a file in a folder and keep an eye out for the changes.

If you hit any problems getting it up and running, drop a comment below.

Source:
Install BitTorrent Sync on a Headless Ubuntu Server
http://artofsimplicity.co.uk/install-bittorrent-sync-on-a-headless-ubuntu-server/

Note to editors

It looks like we had a couple of edits lost as a result of one node's edit stepping on the other. There were actually 3 edits on Mar 22, 2014 and we do not have time now to investigate what exactly was edited. From the preliminary look, it seems that there were some things that were different. We do have the history, but it would be easier for the author of edit to restore his edit. Our hands are full at the moment.

Unfortunately, sometimes it gets pretty hot and lots of things happen, and it was probably one of those times. So, if the author of those edits could do the diff and see what he wanted to edit and then correct some things, we'll try not to touch this document for now.

May be, if you intend to do the edit not from the master node, but from some other node, say something like "nodename: Now editing BTSync detailed manual" or "nodename: ready to save changes of edit" or whatever you feel like saying in your Device name message in Devices tab.

Try to use Notepad++ editor if you are on Windows, or some other editor that notifies you of changes on disk so that your edits won't get lost. And before you hit "save" in your editor, make sure it won't step on the version of others (in Notepad++, switch from and app to another or minimize and maximize the edit window and take the appropriate action if you see the file was update while you were editing it).

If you find a better way to assure edits are not lost, please say so in "Edit History" section (see below).

One of the things that can be done to inform others that you are doing or intend to do edits, is to add your message to the section Edit History. We need to find the most effective way of working in this environment and everyone's comments or suggestions, and especially the solutions, are obviously valuable.

Major Problems

There are some problems we can see some people experience. The good news is: the solutions are available. The bad news is: they are there and there is nothing we can do about it, at least at this time, primarily because BTSync simply refuses to even recognize them as something real, and even less they are interested in actually fixing them even though we have provided the solutions and suggestions on how to handle it.

But they are not even interested in listening about them for some strange reason. Actually, some of their "reasons" could be understood. Their marketing efforts are directed primarily at distribution of psychosis inducing materials in modern "music" or "entertainment" business, the more appropriate name for which is zombification business, the square purpose of which is to disconnect people from their roots, from their culture and heritage and instead to zombify their minds with all sorts of boom-boom-boom, so they would happily shake their rear ends at some zombie party and would become essentially the happy idiots, easy to control and easy to impose all sorts of things on them and inculcate their minds with all sorts of violence, psychosis and hysteria, precisely in line with the satanic NWO agenda of taking over the world and creating a two class society "elite" and "robots" or slaves. All others are not needed in this scheme.

And most of these people are simply interested in syncing their mobile devices with their home and work computers so they could be zombified wherever they go. And in this setting, they do not deal with the issues of general public grade, because they know all their devices and can change things any time they want. But in general public setting, you simply have no contact with people that might have some problems syncing. That is the basic difference and this is probably the reason for their "attitude". As some say: "you need an attitude adjustment, pal".

So, we will describe those most common problems and provide you with solutions that should fix your situation, at least in most cases.

Major problems new users experiences are the following:

Sync stopped: device time time difference more than 600 seconds

("Sync stopped: device time time difference more than 600 seconds" error message in Devices tab next to some device.)

First of all, when you see this error message in Devices tab, it, basically, means that the time on your computer is not set correctly from the standpoint of other nodes on this share, and, therefore, it is impossible to determine which version of some file is actually newer and is to be downloaded. When some file is changed, there is a need to find out which version is actually newer.

But, it may turn out that in reality time on your computer is set correctly including the time zone, but there exists a different time zone setting in different places in the world depending on which particular source you use to get time and time zone information. Some of them may not be set correctly for your location.

Therefore, do not get scared seeing this error. These are just minor unpleasant things and the algorithms in BTSync need to be radically redesigned in respect to this issue. But you will still be able to resolve this issue and get in sync, even though the solution might not make much sense to you and you might not feel comfortable about it.

The "good news" is that the solution is available. The "bad news" - you might want to puke. But this is just a psychological issue. Do not worry, "we'll get them" and they will indeed fix this issue, sooner or later. We have already created so much noise about it on their forums and provided so much evidence, that we are like a bone in their throat. And there is basically nothing they can do but to finally recognize this problem as THEIRS, instead of blaming the users for it.

Basically, the solution for now is for you to enable logging and then find that node which gives you that error and notice the time difference between you and it and adjust your clock accordingly. Most likely, the difference will be +/- 1 hr. But we have seen cases with +/- 2 hrs.

There is no need to set your time exactly to the second to compensate for the difference between you and the other node. Because the normal acceptable range for error is +/- 10 minutes. You'll just have to increase/decrease your time by +/- 1 or 2 hrs.

This is one of the strangest errors quite common among new users. Basically, it has to do with the case when BTSync needs to decide which version of file is more recent in case some file was modified on one of the nodes and, as a result, there exist several versions of the same file.

This is probably one of the most unpleasant problems with BTSync, especially in general public application where you have no idea about the other nodes on the share and have to way of contacting them and telling them that their computer time is not set correctly.

Furthermore, it is possible that the computer clock and time zones are set correctly, according to your local regulations, but you still see that error. Because in some other place in the world they might not have a correct time zone for your location. And their update service provider does not have the correct time zone tables for your location.

Most often, your clock time will be different from the other nodes by +/- 1 hour. In rare cases it will be different by +/- 2 hrs. It has to do with the DLS time adjustment for different time zones as seen by your local system.

See: If time is set incorrectly on your comp

Incomplete downloads/uploads

This is probably one of the most confusing issues new users may experience. Generally, it has to do with user modifying, deleting or renaming some files on the r/o nodes.

According to present logic (as of 1.3.77) BTSync will neither update (download) or distribute any files that were modified, deleted or renamed. So, if you decide to edit some file on the r/o node directly, you will no longer receive its updates from the other nodes.

See: BTSync behavior table

Therefore, if you want to modify some file and still receive its updates, better make a copy of it and modify that one instead.

Furthermore, those nodes will be shown to the other nodes as though they still have to upload some size to them. And, probably the most unpleasant and utterly unintuitive effect would be that even if you manage to restore those files back to their original state, or restored the deleted or renamed files, they will still not be updated from other nodes, no matter what you do.

If that happens, about the only thing you can do to restore those files to their original sync state is to use the Restore option:

See: Restoring the original sync status of some files that won't sync any more

So, the proper way of handling all the files that you indeed do not want to be updated in the future is to use the .SyncIgnore file located in your main sync directory.

See: Using .SyncIgnore file to exclude some files from sync

Enabling logging

To get more detailed information about the activity of btsync or to identify any problems or observe what is going on "under the hood" as far as file transfers or connection attempts are concerned, you may enable the detailed logging.

On Windows

Logging can be enabled/disabled by right mouse button clicking on the btsync taskbar icon and selecting "Enable Debug logging". The log file is named sync.log and it will be located in C:\Users\your_accont_name\AppData\Roaming\BitTorrent Sync folder, which is a regular text file.

On Linux

Create a regular text file debug.txt in the directory specified by "storage_path" in your configuration file, that should contain the string FF (Default on Ubuntu /var/lib/btsync/debug.txt).

See Configuration for more information.

Then simply restart the btsync. Logs can get pretty big during the periods of high activity, so you probably want to disable it once you find whatever you were interested in. Just rename debug.txt into debug_XXX.txt or something like that and restart btsync again.

BTSync versions: user vs. server

Here is my situation: I've been using the user version...

I'd stayed away from the server stuff because it looked more complicated, but lately I've been thinking it might be useful if the machine that was running as my server for this (the one that generated all the secrets, essentially) was using it, since it *appears* that the server scripts run at a lower level, and I like the idea of being able to specify all those configuration options--especially the file priorities, and controlling how long the archives are kept.

First of all let's clarify one thing: both packages install the same BitTorrent Sync executable. Because of that, there are absolutely no differences between what BitTorrent Sync is able to do. The difference stays in the way, your machine handles BitTorrent Sync:

Let's summarize: for you it makes sense to use btsync instead of btsync-user if:

[...]

Is there a way to configure btsync on a machine that already has btsync-user and already has files that are being synchronised on it?

Yes. But first take a decision if the server package is really the right thing for you. If you say yes, I will explain here how you may migrate your system.

(tuxpoldo, Advanced Member)

Debian And Ubuntu Server Unofficial Packages For Bittorrent Sync (thread)
http://forum.bittorrent.com/topic/18974-debian-and-ubuntu-server-unofficial-packages-for-bittorrent-sync/page-22#entry79372

Setting and changing file ownership and permissions

I'd like to change the user to a different user (not user name for the login, but the user who owns the folders) the user is currently set to btsync.

I'd also like to change the group permission. The group permission is currently set to Group can Read and I'd like to change to Group can Read/Write.

There are settings for both of these: during the first installation, debconf asks for the most important settings needed to install the daemon. If you perform a service reconfiguration, debconf asks for all available settings. If you perform a reconfiguration with

sudo dpkg-reconfigure btsync

you will be asked for the credentials and there you can specify both a user and a group of your choice for the btsync daemon.

Additionally you will be asked also for a UMASK for the daemon. This setting determines with which mask files are created and written. In order to make all files and directories group writable, you have to specify 0002

Comment: this is a very important point. It sets the SGID bit in file attributes. If you want the files to be also writable by you, for such things as FTP file transfer to the r/w node, and your user account is different than default (btsync), then what you can do is to add your user account to the btsync group and also make the file permissions writable by the group (file mode 2775 - note the leading 2)]. Here is the rule:

The set-group identification (setgid) permission is similar to setuid, except that the process's effective group ID (GID) is changed to the group owner of the file, and a user is granted access based on permissions granted to that group.

When setgid permission is applied to a directory, files that were created in this directory belong to the group to which the directory belongs, not the group to which the creating process belongs. Any user who has write and execute permissions in the directory can create a file there. However, the file belongs to the group that owns the directory, not to the user's group ownership.

Special File Permissions (setuid, setgid and Sticky Bit)
http://docs.oracle.com/cd/E19683-01/816-4883/secfile-69/index.html

One question: are you sure, that the server packages are the right ones for your use case? If your Linux machines are used as regular desktop machines with a user that interactively logs in and wants to synchronise directories owned by himself, maybe the packages for desktop usage btsync-user may be more suitable for you.

(tuxpoldo, Advanced Member)

Debian And Ubuntu Server Unofficial Packages For Bittorrent Sync (thread)
http://forum.bittorrent.com/topic/18974-debian-and-ubuntu-server-unofficial-packages-for-bittorrent-sync/page-22#entry77978

Main principles of operation and conceptual framework

Note: this might be too technical for some readers to comprehend some details. So, if this is the case for you, just skip this chapter and go to the next one. It should not hurt your understanding of how it works or what to do if you have some kind of problem.

First of all, the central idea of BTSync, as seen by the author of this manual, is a concept of dynamic torrents. BTSync, "under the hood" is simply using torrents to perform the bulk of most important work as far as node updating and file transfers with the exception that the BTSync torrents are dynamic in nature and are used only temporarily while performing the transfer of specific files, and this is one of the central concepts of BTSync design.

The principle is this: regular torrents, as used by all sorts of bittorrent programs are static and include every file in some collection. Once created, the torrent can not change and it includes every single file of the entire collection. That is why torrents get "stale" and outdated. Because real world information forever changes. Files get updated, edited, extended added or removed and so on. If you look at some torrents that were published several years ago, there is a good chance that many of their files are not longer current or interesting, even though some other files are still current.

With dynamic and temporary torrents that are created on the fly and exist only for the duration of file transfer or update, you no longer have the aging problem. Furthermore, since the entire collection no longer consists of a single huge torrent, but consists of multiple and fresh torrents that always contain the freshest version of the files, the collection can now be extended, modified, or some files may be deleted and so on. As soon as program detects that some file is no longer current and was updated in the file system, it creates a new torrent and sends the updates of this files to the other nodes.

The "beauty" of this approach is that BTSync does not have to reinvent the wheel and can simply utilize the bulk of programming code from the existing torrent applications. There are many excellent open source torrent programs that have all the necessary issues worked out and have all the necessary features and abilities to perform the torrent based transfers, node discovery via torrent tracker servers or DHT protocol and so on.

All this means that you can simply use the standard torrent engine "under the hood" and add some relatively simple higher level logic and algorithms to detect the file changes and schedule the updates.

Processing efficiency optimization

BTSync utilizes the file transfer efficiency improvement for the small files. For small files you still need to generate a separate torrent, which takes time. When transfer starts, you still need to transfer the torrent file and then the data itself, which takes more network cycles and slow things down.

So, what BTSync does in this case is to bulk several smaller files into a single torrent and transfer it all at one go, like a single file. This significantly improves that transfer efficiency. As a result, we have observed some nodes updating at the speeds of over 20 mbit, which is excellent performance. BTSync claims that they could go at the speeds of up to 90 mbit on a LAN.

The concept of a key (so called "secret")

Every collection of information, or a share, as they call it, is identified with unique key (or "secret" as it is called by BTSync, which is incorrect and misleading since it is not a secret of any kind) for this collection.

The only and square purpose of a key is to uniquely identify some collection of information (share). It has nothing to do with any kind of "secrets" and it carries no "secret" meaning underneath, and it is not even encoded because there is nothing to encode, really and nothing to be kept "secret". It is simply a unique hash ID for the collection. Sure, it can be obfuscated further by re-encoding it for some specific purpose, such as to confuse things more for some undesired third parties. But that is beyond the point here.

The key consists of a random collection of ASCII characters and digits. It is generated via standard O/S provided GUID generation algorithms generating the globally unique identifier. This identifier is then processed and hashed in such a way that at the end what you have left of it is a hash string of around 30 characters in length consisting of capital letters and digits that uniquely identify some collection.

It has to be understood that the key is not applicable to any particular node. ANY node that has access to that key has access to that specific collection without any further actions from the user. That is the difference between, for example, the Syncthing approach where a key uniquely identifies the node, not some collection. So, when you get a key, you can contact some node and see all of its published collections it their list of collections. From then on, you may chose to add some of its collections to your node as well and will become a part of that collection, subject to updates and distribution.

When new collection is added, first of all, it gets indexed. The program goes through each file and calculates its hash value similar to a checksum. It then adds this file to its internal database and sets the initial state of various flags for that file, such as propagatable, updatable, physically present in the file system, deleted and no longer active and so on.

Once database is built, the program then contacts the tracker servers. As it stands right now with BTSync, the domain names of several BT trackers and relays are hardwired into the program and can not be changed or configured by users. Actually, more precisely, the program, besides having some trackers and relays hardwired into the program, also contacts the well known BT server and obtains the list of current tracker and relays from it in JSON format. This way, their tracker lists are always current and do not get outdated with time.

Side note:
Monopolizing the "entertainment" industry distribution market

Interestingly enough, it has been observed, at least on one occasion that even if you uncheck all the check boxes in folder properties that have to do with trackers or node discovery, such as "Use tracker server", "Search LAN", "Search DHT network" and do not enable "Use predefined hosts", the nodes STILL can discover each other. Which means that no matter what you do with your settings, you will not be able to avoid going through the BT trackers or relays.

It is suspected that BT has a strong commercial interest in promotion for the entertainment industry and probably has some contracts with the industry that allow them to get the piece of the pie by distributing the promotion "bundles", in particular, via BTSync. In this light, it is interesting to observe that even though BT claims that BTSync will remain "free" forever, they, nevertheless, keep the program source closed. Strange, isn't it? Since they do not make any money on sales of the program, then what could be the reason to keep the source code closed, besides some commercial interest?

Well, it appears that BT is trying to "corner" the entertainment industry distribution market and obtain the monopoly on distribution of various "bundles", which they release on a regular basis. Every few days they release a new "bundle". Question: why would they do that if they did not have a commercial interest in this deal? Well, they simply might have a contract in which they get a set percentage of sales of those products that are promoted and distributed via BTSync. So, by merely presenting such things as the count of transfers and total amount being transferred, it is possible to calculate the size of their share of the pie.

And that is precisely the reason they do not release the source code to BTSync. Because they are probably interested in monopolizing the whole market of distribution via net. If you look at their blogs, things may become "clear as a bell" to you.

Performing file updates for the share

BTSync has two separate mechanisms to detect file modifications, deletions and additions.

First mechanism is to utilize the operating system file system events, such as updates, deletions and creation of files. These events are delivered by the O/S nearly instantly and could be acted upon immediately for updates of all nodes that are currently on line for a given share. But the nodes that are not on line at the time of file system changes are updated according to second mechanism, file scanning.

BTSync performs file system scans on a periodic basis at intervals of 10 minutes as a default. During those scans BTSync recalculates the file hashes of all the files in the share and if it finds that some file hash has been changed, compared to what is stored in the internal database, it immediately places that file in the update queue to be transferred to all the nodes currently on line.

But file updates are performed conditionally and it is done in a different way for r/w and r/o nodes. For r/o nodes, file is scheduled for either download or upload only if that file still has the original distribution flags set appropriately. For example, if some file was deleted, renamed or modified in the file system, that file is not subject either for download from other nodes or distribution to other nodes. It becomes a private copy of this node, and that is precisely the reason we are seeing well over a half of all the nodes on one of our main shares showing as being in the incomplete state, while it is considered complete or "done with" by the node itself.

The master nodes (r/w) have detected some file changes and informed all other nodes on line that it has the new version of some file. But the r/o nodes refuse to download them because according to their view, those files are no longer updatable or propagatable, like they do not exist. That is why the master nodes add the sizes of those files to the total size of the amount remaining to upload to some node and these nodes show as incomplete, while, from the standpoint of those nodes, they ARE "complete", or, more precisely, do not exists from the standpoint of updates.

This is a flawed logic and BTSync will never work in a consistent and predictable manner unless this issue is resolved. This is known as database consistency problem, which means there exist multiple copies of the same data and those copies differ, and, since master nodes do not know and can not possibly know what's in the mind of the nodes that have modified the file, the issue remains unsolvable on logical grounds.

The file transfer phase

Once these relatively simple logic rules are followed and some file was scheduled for update, what remains is actual transfer phase. As mentioned above, the transfer is performed via standard torrent transfer engine.

Each file consists of pieces or blocks. During the block download stage, the block is simply locked for a set period of time and can be updated only from a particular supplier node. If that block was not completely downloaded during the set time interval, it is timed out and rescheduled for repeat attempts.

These mechanisms are well known from regular torrent programs and we are not going to discuss them in this document. There is plenty of information on the net for technically savvy.

BTSync behavior table

Note: this table was initially created as a stub, a concept in an attempt to fully describe the behavior of btsync under all different conditions and modifications of the files by the users on both ends, the master (r/w) and slave (r/o) nodes.

The initial version was created without much understanding by just guessing things and referring to previous observations of some cases. The version you are reading was updated after verifying the behavior in most cases described here.

According to our understanding at the moment, there are a couple of principles that may help to understand the consequences of various user actions and changes to the files in the hierarchy of some share (parent folder/directory).

First principle: Basically, what needs to be kept in mind is that the central idea in the btsync design, as far as we can see, is to keep the files on all nodes in the same consistent state, that is the same as on the master (r/w) nodes. And, if we keep in mind the idea that the r/w nodes "dictate" the conditions and status of the files and r/o nodes have to exactly replicate the contents of the r/w nodes' files, if those files were not modified, deleted or renamed on the r/o nodes, then many things may become intuitively understandable.

The second principle is the rule: "user knows better", which means that if there are any conflicts between the the btsync update logic and the operations on the files performed by the user manually, such as file deletions, modifications, renaming and so on, then the user action takes precedence over the update logic in btsync.

Furthermore, there is a logical difference between the file modifications performed by the user on the r/w and r/o nodes. The behavior of the r/w nodes seems to be entirely consistent with logic and is equivalent to the ways it happens on the O/S level, that is, when you delete some file, it is simply gone with no memory of it left. But on the r/o nodes, the btsync behavior is not so consistent and in some cases may be considered as utterly inconsistent and utterly unintuitive. You will see it in the table below.

As of now, what needs to be kept in mind is that, generally, any modifications, deletions and renaming on the r/o nodes mark the modified (original) files as non-updatable and non-propagatable. Effectively, those files are permanently latched up in a state that essentially indicates that they do not exist. And, probably the worst part of it is that in some cases you can not recover from that state and restore the normal propagation, even if you enable the "Restore modified files to original version" via Folder preferences -> Advanced tab.

Furthermore, even if some file is deleted, it still remains the the r/o node's database and is simply marked as "dead" to btsync (non-updatable and non-propagatable) and any modification of it, even renaming it, won't change the status of the file with the same file name in the future. In other words, renaming the file does not carry the file access attributes to the new file. The attributes remain with the old file name, and they essentially mean "dead file", which seems to be logically incorrect design.

Whatever were the reasons there for this design, it causes quite severe consequences for the future of the file with the same file name and there is no way to change it back to its original propagation status no matter what you do, except of resetting the attributes to the original state for the entire collection (database) on the r/o node via "Restore modified files to original version" via Folder preferences -> Advanced tab on the Linux r/o nodes at least.

The btsync team could shed some light on these issues indeed. Otherwise, we are going to be guessing here as to what really happens "under the hood" and for what reasons. But at least we can try, and, hopefully, this table will become the beginning of detailed clarification of all possible cases, and, even more hopefully, with help of developers or support team, at least with clarifications and reasoning for the present behavior, which has to be done, sooner or later, no matter how you look at it. Otherwise...

BTSync Behavior Table

Operation What takes place
New file added Master side (r/w nodes): File will be added to the database, marked as updatable and propagatable and will propagate and be added to all the nodes, r/w and r/o, marked as updatable and all the modifications to the master version will be propagated to all nodes.

Slave side (r/o nodes): Since or/if this file was added on the r/o node not by btsync, but by some other means, and it is not contained on the master (r/w) nodes, this file will be excluded from propagation to other r/o nodes and will not be added to the database of this node (??? - not sure about this). Because the database contains only files that exist at the moment on the r/w nodes plus files that at some point existed in the r/w nodes' database, but were either deleted or renamed on this r/o node.
File deleted Master side: File will be also deleted on all the nodes, r/o and r/w. If this file is restored in the future, it will appear again and will be propagated (transferred again as any newly created file) to all the nodes. In other words, it will be considered just like a newly created file.

This means that the file deletion on the r/w nodes does not affect the propagation status of the file with the same file name in the future. But on the r/o nodes, local file deletion behaves totally differently and is essentially inconsistent.

Slave side: If some file was deleted on the r/o node, it will be permanently marked as deleted or "excluded from updates and propagation" in the database of this node and this file name will become effectively dead to BTSync, no matter what you do with the file with the same file name in the future. The only way to restore its updatable and propagatable status is to use "Restore modified files to original version", which will restore the normal propagation/update status to ALL the files in the database, and not just this file only.

Furthermore, if file with the same name is created in the future, it will not be updated from other nodes and will not be propagated to other r/o nodes, for some strange reason. What is the reason for this behavior is unclear at the moment. It seems that if some file is "resurrected", then it should again become the subject to the normal propagation and update procedure, just as it happens on r/w nodes. But permanently excluding some file from propagation, regardless of any future action of the user that might restore this file seems to be utterly illogical and inconsistent with normal O/S behavior for similar operations on files. Furthermore, it is inconsistent with behavior of r/w nodes.

Because this file should not be updated or propagated ONLY for as long as file with the same name is not restored, renamed or created again in the future. Otherwise, you have a black hole effect: whatever you do in the future with the file with this file name will have a net result of simply getting sucked in into a nowhere land in terms of updates or propagation, which equivalent to a black hole in logic. This is also entirely inconsistent with normal behavior of the O/S or the simple logic. See file renaming below for all the reasoning behind it.

Yes, there is a logical difficulty here. Because if file was modified by the user, it should stop being updated and propagated by btsync. But when the file is restored after deletion and its status is returned to normal propagation, then what do you do in case of file modification which should NOT propagate it?

But this contradiction seems to be resolvable by introduction of the "force update" command, sent by the r/o node to other nodes when file is RESTORED and not MODIFIED. Therefore, the fact of restoration of the file vs. modification of it can be detected by simply checking its "physically not present in the file system" bit in the database, which is set when btsync initially detects file deletion.

So, if file reappears for some reason, when btsync looks at its record in the database, it knows that the previous state for this file was "does not exist in the file system". Therefore, this is RESTORATION and not MODIFICATION of the file.

Which means that there needs to exist TWO separate bits for the file status in the database. Otherwise, the way it looks right now, btsync is trying to manage all the cases with only ONE bit and that is why it can not detect the difference between modification of the existing file in the file system or RESTORATION of the file that did not physically exist during the last scan.

See: Restoring the original sync status of some files that won't sync any more
File or subfolder renamed Master side: File will be renamed in the database. Then file will also be renamed on all the nodes, r/w and r/o. No actual file transfer occurs. If this file is renamed back to its original name in the future, the results will be the same. It will be simply renamed on all the nodes. Therefore, the file rename operation behaves consistently, as you would expect doing it in the O/S.

But if there are multiple r/w nodes on this share, the behavior will be different if other r/w nodes have already synced with this file. In this case, your newly named file will be sent to them as a new file, but what is even worse, is that they will resend you the file with old file name. So, what you will end up is two files with exactly the same contents, but with different file names (We are just guessing here because this was not verified in action).

In case you renamed a subfolder, what you will get is two copies of exactly the same subfolder contents with two different names, the old one and a new one. This is utterly unintuitive.

Therefore, you need to first add the subfolder with old file name to your .SyncIgnore file and save it, and only after that you can rename the folder. This way your node will not accept the subfolder with old file name, but will still send out the newly named subfolders to other nodes. The same thing with .SyncIgnore is applicable to the case where you have renamed a file.

Slave side: File will be renamed on this r/o node only. There will be no backup made of the old file in the .SyncArchive folder.

Since this file (with new name) most likely does not exist on the master (r/w) nodes, the new file will be considered as a local copy, not subject to propagation or updates from other nodes, and for obvious reasons - it simply does not exist on the other nodes, most likely.

Furthermore, the file having name of the old file will be marked as non-propagatable and non-updatable in the database for this node, and if you decide to recreate the file with the same (old) file name in the future, or rename the file with the new name back to its original file name, it will still be considered as non-updatable and non-propagatable. Strange logic. Would be interesting to know the reasons for it.

Furthermore, if the file with the same file name is modified or created on the master side (r/w) in the future, it will not be propagated to this r/o node from the r/w nodes. This means, that from now on, this file becomes "dead" to btsync for this r/o node and it will not be updated in the future no matter what you do, which is a logical disaster and inconsistency of behavior. Yes, there is a logical conflict in this situation: since the file has been renamed, and it was done on the r/o node, then the file with the old file name should not logically be updated as it does no longer exist in the r/o node, and that is probably why btsync does not delete this file from the database, but instead marks it as "dead" to future updates and propagation.

Yes, the present logic "works" for the situations where you simply do not want to receive the updates for this file, which does make sense. But when the file with the same name is restored again on this r/o node in the future as a result of user action, that file name should logically return to its original propagation status. It is like restoring an accidentally deleted file in the file system from a backup. Can you imagine the case where you can not restore some file once it was deleted no matter what you do? More than that, can you imagine the case that if you restore the file and its access attributes or ownership become something different then they were in the original file? Why?

So, this behavior is utterly inconsistent compared to the equivalent operation on the O/S level, that is: if file is renamed, the old file does not exist any longer and there is no "memory" of it is left, and, therefore, if you rename it back, all its attributes travel with physical file, and not with the file name. So, if some "dead" file is resurrected by the file rename operation, then its update and propagate attributes should be set the same as of physical file. And, if this file was recreated not via renaming back, but by file copy or edit/save operation, then, since we did not merely rename it, it should receive the default file attributes, just like for newly created file.

We do not know the reason for this logic but what seems to be more appropriate to resolve this conflict is to restore the original propagation status for this file name on this particular r/o node if either file with the same name was recreated again on this r/o node, which would imply the file with this name exists again and we do, therefore, want to receive its updates and propagate it. Or, if file is renamed back, that would logically mean restoration of its propagation status back to its original state.

So, more logical behavior would imply the following behavior: if file is renamed on the r/o node, then the file with old file name is marked as non-propagatable and non-updatable on this node, but ONLY if the file with the same name is not recreated again or file would not be renamed back to its original file name and its propagation status is restored to normal.

Since old file name is maintained in the database, then btsync has sufficient information to make a decision as to file "resurrection". This means, that no matter what the contents of the file with the original file name become, its UPDATE (but not the propagation to other r/o nodes) status is restored to normal as soon as it was updated to correspond to the master version. This, in turn, means that this file will be immediately updated from the other nodes to their version of it. Only then its propagation status to other nodes is restored so that they won't get the version of the file that does not exist on the r/w nodes thinking that this is merely a newer version of the file that already exists in their database.

There is a problem with file update time for this file because, depending on how you restored this file, the file update time may become newer then on the master side, which may prevent its propagation from other nodes to this node. But this could probably be resolved by the r/o node explicitly requesting a force update of this file with whatever version and time stamp exists on other nodes.

This all means that btsync has to maintain TWO bits of file status - update (from either master or r/o nodes), and propagate. It looks like currently btsync tries to manage the entire logic with only one bit, which is not going to produce the consistent and predictable behavior no matter what they do.

File updated/modified Master side: The updated copy of the file will be transfered to all the r/w and r/o nodes.

Slave side: File will be considered private from then on and marked as non-updatable and non-propagatable. Any future modifications of this file will not be propagated. Since r/w nodes are not updatable from r/o nodes, future modifications of this file on the r/w nodes will not be propagated to this node.

Even if you delete this file from this r/o node if you want to restore this file to the r/w node's version, it will not be restored neither from the r/w nodes, nor from r/o nodes. This file becomes dead, dead, which is a strange and utterly non-intuitive logic, inconsistent with what you would expect in the same situation in the file system.
File with the same name recreated again after being deleted Master side: This file will be considered as newly created file and will be propagated to all other nodes. In other words, on the r/w nodes, there exists no "memory" of the deleted files. Once they are deleted on the r/w node, they simply disappear with no traces left. This is consistent with O/S behavior.

Slave side: This file will still keep its non-updatable status and will not be propagated to other r/o nodes. The old file name literally becomes dead to btsync. This file name still exists in the database for this r/o node, but its status becomes "dead" - do not touch it from btsync, which is utterly inconsistent logically, including the equivalent operation on the O/S level. The file should be "dead" only for as long as it does not exist physically on this r/o node. But as soon as anything is restored with this file name, regardless of contents, it should again become updatable from other nodes and propagatable, but ONLY after its contents has been restored from the other nodes.
File did not have write permissions at the time of initial indexing and database creation Master side: ???

Slave side: ???
File permissions changed so it is no longer writable by btsync Master side: ???

Slave side: On Linux, since and/or if the btsync process runs as deamon, it runs with superuser privileges, and, therefore, is capable of changing the file ownership and permissions in any way it wants, which is in fact does. Which means, if this file is updated on the r/w node, the updates will still propagate to this r/o node and file ownership will be set to btsync:btsync, which is strange behavior.

On Windows: ???

What is missing in current versions of documentation is the description of the procedure to properly reset the system if something happens that BTSync considers as something different than you might think. For example, for whatever reason, some files stop syncing and won't propagate to the slave machines. But this is not what you expected to happen and is not something you'd like to happen. In this case, how do you regain control of the system and syncing process?

There is also a need to know the name of the database file for some share. Currently, there seems to be no easy way to find out the name of database file for some sync folder. It would be desirable to be able to see the name of the database hash right from the GUI, regardless of whether debug log was on or off.

Current file update and propagation logic problems

It seems that btsync logic for r/o nodes needs improvement, unless we are missing something. Because it does not seem to detect various user operations on the file properly and simplifies all possible cases, such as file modification, renaming, deletion and restoration, by bulking them all together into a notion of "file is not updatable and not propagatable". But all these cases have different semantic meaning.

File modification means that this file does exist in the database and its "physically present in the file system" bit is set. In this case, two separate bits in the database - "updatable" and "propagatable" should be reset. But "physically present in the file system" bit should not be changed.

File deletion means that its "physically present in the file system" bit was set, but no file actually found in the file system. In this case, its "physically present in the file system", "updatable" and "propagatable" bits should be all reset.

If file is restored by the user, its "physically present in the file system" is tested, and if it was reset, then this file has been "resurrected", in which case its "updatable" bit is set, which means that this file will be immediately requested from other nodes and its contents, whatever it was when it was restored, will be overwritten with the version in other nodes with the assumption that it will be exactly the same as on the master nodes. Then, AFTER this file has been received and updated, its "propagatable" bit is set and this file may then be transfered to other nodes. Additionally, "physically present in the file system" bit is set.

When file is renamed, it is equivalent to 2 separate operations: 1) file is deleted, 2) new file was created by the user, which did not exist before. In case 1, the old file is marked as "physically present in the file system", "updatable" and "propagatable" - all false. For the new file (case 2), its "physically present in the file system" bit is set, "updatable" and "propagatable" - both - are reset.

These 3 bits will be able to handle all the cases that we can see at the moment without any problems and any user actions on this file. Furthermore, the net effect will be logical and intuitive to the user, who may restore some deleted file from a backup and would expect it to become a "normal file", subject to all rules, including its propagation and reconciliation of its contents with the version, currently existing in the master nodes, "master rules" rule applies. Because it becomes the same file as it exists on the master side, and the only one who dictates its contents is the master node and the copies of it on the r/o node.

ALL that user has to remember is that if some file is modified or deleted, it will be considered a "private" user file and will not propagate, and if file is renamed, the new file will not be updated nor propagated because it does not reconcile with the master version of it, which does not exist. It is the same behavior as if he added a new file on the r/o node.

Otherwise, everything should behave like a normal propagatable and updatable file. The logic becomes simple and intuitively obvious and everything should work just fine, unless we are missing something here.

Peer discovery principles

BTSync uses several methods of peer discovery.

use_lan_tcp is NOT use for peer discovery; it is a way of speeding up connections to peers that have already been contacted with UDP (like turning off lan_use_encryption).

The relay server is NOT used for peer discovery; it's for working around stupidly obstructive NAT and firewall devices.

Source:
How do the machines find each other via the "Secret" code?
http://forum.bittorrent.com/topic/19434-solved-how-do-the-machines-find-each-other-via-the-secret-code/page-1#entry49550

What does relay server do?

Relay servers help get around firewalled states on one or more nodes.

DHT helps when the tracker servers are down.

Source:
http://forum.bittorrent.com/topic/19434-solved-how-do-the-machines-find-each-other-via-the-secret-code/page-1#entry49538

Note: If you are connected to some node via relay server, then you will see the node's name exactly as it was named by the node itself, but its IP address and port will be the address of the relay server and not the node itself. In this sense, the relay server behaves like a proxy server and it hides the node itself from others.

What does tracker server do?

The BTSync tracker is the same as a Bittorrent tracker, it does almost no work.

For each combination of peer and share it stores:

Once two peers have been introduced to each other the tracker has nothing more to do. If the tracker goes down after the peers have linked up there's no problem. That DNS name resolves to three independent servers which will (probably) be sharing information.

Note: This is, potentially, a security issue. Since these BT servers are effectively hard-coded and the user can not remove all of them for security reasons, there is potentially a possibility that BTSync may transfer the information about all the nodes on some share to the 3rd party, and this is not nice at all.

The problem here is that you do not really know how BT is going to use the information it receives for you, regardless of how much promises they give you that they do not collect anything beyond the most basic things needed to connect the nodes and does not send any of it to the 3rd party. And even if they make a public statement that they do not send any information, including the hashes, to any 3rd party, it is not clear how much you can trust such a statement.

Considering that BTSync is not an open source, how do you know what they may, or in fact ARE doing with the information they get from your node(s)?

But, I expect, (deal killer if not!) there will be a configuration option to allow other trackers later on.

As I said the real magic is the DHT mode where if you know or remember the location of any node in the network you can use it to find the rest of the network.

I've tested the known hosts connection with Windows-7, it's working fine for me. I've tested it with only one side knowing the location of the other and that works too (depending on the firewall).

Remember you have to open UDP ports not TCP ports and usually tell the BOTH clients the public IP addresses of the other so they can work together to poke holes through NAT or connection tracking.

If the public IPs are dynamic you can use services like dyndns.org to get a constant dns name.

If it's still not working you might need to raise a bug report.

Source:
How do the machines find each other via the "Secret" code
http://forum.bittorrent.com/topic/19434-solved-how-do-the-machines-find-each-other-via-the-secret-code/page-1#entry49570

About minimum firewall requirements...

Okay, about minimum firewall requirements...

The relay and the tracker are found using DNS, these are the current settings for relay and tracker.
Note the TTL less than 5 minutes in both cases, this is the warning they need to give to change these.

;; ANSWER SECTION:
r.usyncapp.com. 203 IN A 67.215.231.242
r.usyncapp.com. 203 IN A 67.215.229.106

;; ANSWER SECTION:
t.usyncapp.com. 258 IN A 54.225.100.8
t.usyncapp.com. 258 IN A 54.225.196.38
t.usyncapp.com. 258 IN A 54.225.92.50

Normal operation is using the tracker to find peers and using direct connections between the peers to transfer data. All data is transferred using UDP packets.

Your BTSync has a port configured, say 20001.
The Peer has a port configured, say 20002.
The tracker has port 3000 configured.
The relay has port 3000 configured.

Requirements are:

This assumes your firewall uses the normal 'timeout' method of noticing solicited responses. The problem is that the firewall will not see the request for the first response as it travels via the tracker. It must not do anything "unfortunate" when it sees this "response".

If your firewall is broken in this way then unsolicited packets must be accepted both ways between UDP ports 20001 and 20002.

If your firewall cannot be fixed connections to the relay must be opened for all peers that need to communicate with you.

If you wish to use DHT you must accept unsolicited packets on your port 20001 from any address.

If you configure known peers you can turn off access to the tracker; no packets then need to go to the tracker

(See http://forum.bittorr...dpost&pid=49550 ).

Source:
trouble with rotuer (thread)
http://forum.bittorrent.com/topic/19214-trouble-with-rotuer/#entry50004

BTSync Tricks

Excluding some files and/or folders from syncing

If you want to exclude some files from syncing, so that they are not propagated to other nodes, one of the ways is to use the .SyncIgnore file and enter the names of all files and/or folder that you want to exclude.

The other way, that works on r/o nodes only, is to either modify or delete the files you do not want to either download or propagate. All the files that either modified, or deleted are marked in the database as "non-propagatable" and, effectively, become dead to btsync. This is basically not a recommended way and the sync folder will show the state of the share and the amount remaining to download/upload incorrectly, and it will look like there is still data to be transferred while no transfer will ever occur.

Using .SyncIgnore file to exclude some files from sync

The proper and recommended way to exclude some files or subfolders from syncing is to use the .SyncIgnore file, located in the top folder of the share (hash, key, sync folder). In this case, BTSync will exclude these file from downloading and propagation to other nodes and will properly report the statistics and the amount remaining to download/upload in the Devices tab.

.SyncIgnore is a regular text file (UTF-8 encoding) containing one file or folder per line.

You can use "*" and "?" wildcards in .SyncIgnore files.

Examples:

The effects of .SyncIgnore are more subtle that it might look at the first glance and file specifications are not intuitively obvious and behave slightly differently than they do in the O/S.

In the following rules we will use the directory level notion, which is the same thing as directory depth in the file path. The 1st level is the root folder of the share. The 2nd level is a subfolder of the main folder, and so on.

Rules

What is the meaning and function of .SyncIgnore file?

Question about .SyncIgnore, asked on btsync forum:

Does it mean "do not update this file from the master (r/w) node? Or does it mean "do not propagate this file to other r/o nodes? Or both?

Answer:

Both. BTSync checks ignore list before requesting files from remote peers and never tells other peers that it has such file if it is in ignore list. Please see more in

http://forum.bittorrent.com/topic/17782-bittorrent-sync-faq-unofficial/,

(RomanZ, Sync Support)

This implies that if other peers do not have the same files/folders in their .SyncIgnore file as this particular node, then they will still get them from the other nodes (r/w or r/o) that do not exclude these files in their .SyncIgnore.

It also means that .SyncIgnore has an effect on per node basis. If some node wants to exclude some files and not to download nor propagate them from/to the other nodes, then it adds those files to its .SyncIgnore.

If that node was already synced and already has those files, then after editing the .SyncIgnore, the unwanted files will not be automatically removed from the disk and may be simply deleted from that node manually.

But if you want to ignore the same files on ALL the nodes, then every single one of them should have exactly the same files in their .SyncIgnore file.

"Fighting" r/w nodes and magical appearance of duplicate files/folders

Finally, there is an interesting case of r/w nodes "fighting" with each other (in case you have multiple r/w nodes on some share). If you delete some files or folders on one r/w node, it won't help anything if you have multiple r/w nodes.

Even deleting it locally, will not prevent it from being re-downloaded again in few seconds from other master nodes if they are on line, and even worse, if you are the only r/w node on line and delete some file or subfolder and verify that it all worked fine, you may not realize that it was nothing more than illusion.

The same thing happens if you rename some file or subfolder, with even more amusing effects.

Because it still exists on other master (r/w) nodes. As soon as they come on line, you will get your deleted or old versions of renamed files or folders downloaded from them again.

Remember, all the fully synced r/w nodes contain the aggregate collection that is created as a result of UNION operation. Those files or folders that are absent on some nodes will be sent to them by the nodes that have them.

And even more interesting behavior is observed if you rename some subfolder. What happens if it was already uploaded to other r/w nodes is that now you have two different folders with the same contents, the one with old folder name on other r/w nodes, and the other, with new name on your node, and, since your new folder contains the same exact files as the old one, then it will immediately start uploading that folder to other r/w nodes as a new folder, and the other r/w nodes will send you a copy of the folder with OLD name. Once the sync process for these folders is complete, ALL the r/w nodes will have two copies of the same folder with the exact same contents, but with two different folder names.

And, even more interesting, once you notice that you have 2 copies of the same folder contents but with different name on your share, you may try to delete the folder with the name you do not like, not even suspecting that it came from other r/w nodes and wondering about where did it come from?. But it will be all in vain if other nodes have that folder. They will simply resend it to you again.

So, you MUST use .SyncIgnore if you decide to delete or rename some files/folders in case you have multiple r/w nodes, and some of them have already synced with those folders/files that you have deleted or renamed. Furthermore, all other nodes must also have them in their own .SyncIgnore file and they may not even suspect that they have some duplicate files/folders as a result of actions on other r/w nodes.

So, before deleting or renaming some files or folders, first, enter their names in the .SyncIgnore file and save it, and only then you can delete or rename it.

Using .SyncIgnore Master file

In order to prevent some unusual or strange effects arising as a result of deleting or renaming some files or folders (see: "Fighting" r/w nodes and magical appearance of duplicate files/folders ), what you can do is to create the .SyncIgnore.Master file on one of r/w nodes.

Also, in order for all the nodes to be in a consistent state and contain the same exact information as all other nodes, in case some files or subfolders are excluded from syncing for all sorts of reason, such as containing the private information, or scratch pad files, not meant to be propagated to other nodes, ALL the nodes need to have the same .SyncIgnore file contents. Otherwise, some of them will be updated with unwanted files and will propagate them to other nodes.

Note: .SyncIgnore file does not propagate between the nodes automatically. Because it is an O/S dependent file and Window's version of it won't work with Linux and vice versa.

But the .SyncIgnore.Master WILL be automatically updated and propagated, just like any other file, if ANY r/w node modified it or extended it.

The beauty of this approach is that nobody can dictate to your node which files you do or do not want to update and propagate, as it would be the case if .SyncIgnore was automatically propagated. Because that would be imposing someone else's opinion on what you do or do not want to carry on your node, which does not make sense and is not a "democratic" approach.

The only thing to be kept in mind is that it is an O/S dependent files and so on Linux, all the backslashes ("\") in the file names have to be replaced by the forward slashes ("/") and all the blanks in file or directory names have to be replaces with "\ " to follow the Linux convention.

What it means is that all the nodes can be automatically informed about the master nodes' "latest and greatest" .SyncIgnore contents via .SyncIgnore.Master file. So, they can simply copy/paste its contents into .SyncIgnore file and do some editing for the Linux version of it in order to comply with the O/S file naming convention.

This should work great and prevent all sorts of files or subfolders being distributed unnecessarily. Furthermore, all the discussions and arguments about whether to automatically propagate the .SyncIgnore simply resolve without a single effort of the part of BTSync development efforts and everything "works as is".

And even more important for general public distribution, where one node has no idea about the other nodes and where the information may be updated dynamically, at any point, there might be some changes desired to the .SyncIgnore file. But how do people know about the "latest and greatest" unless the .SyncIgnore is automatically propagated from the master nodes? And if it IS automatically propagated, then the latest master version of it may "step" on your local changes that correspond to your personal view on what to carry and what not on your node.

What will be shown in the Devices tab if some files are excluded via .SyncIgnore?

Q: What will be shown in the Devices tab if some files are excluded via .SyncIgnore? Will it still show that the node is fully synced? Or will it show some size still remaining to be uploaded?

FAQ recommends to keep .syncignore files same on all peers.

If this is true:

If some files are excluded, BTSync won't calculate them in files amount value and summary data size in UI. Also, it will show peer as fully synced when all non-excluded files are synced.

If syncignore is different, next scenario applies:

Say, we have PC A and PC B with some folder synced.

PC A has *.abc files ignored. It also has bunch of other files as well as several .abc. After sync done, PC A will show that it is synced with PC B, amount of files and volume includes all files except .abc.

PC B has bunch of files as well as several .abc. After sync is done, PC B will show that it is yet to upload a number of files to PC A, their amount and volume is a sum of all .abc files that are on PC B.

This implies that if you see some peer node in your GUI that shows that there are still files remaining to upload to it, it could be that they are in his SyncIgnore.

What does it mean if some node used to show that it is fully synced and then it just keeps showing that there is still some size to be uploaded, and does not show that it is fully synced ever since?

It might mean:

1) remote node is RO and some files are deleted / modified.

2) remote node has some files ignored and simple won't accept them.

3) remote node is mobile device and synced only part of files due to selective sync.

4) some issue occurs that prevents files from syncing completely (permissions, letters case bug, locked files, out of free space, etc.)

Q: Does it mean that the user on r/o node deleted, modified or moved some files out of the sync path?

A: Yes. One of the cases is when user changes / deletes the file on RO peer and settings prohibit from re-syncing it.

[Answers provided by RomanZ of BTSync support]

See:
.syncignore Behavior Not As Documented?
http://forum.bittorrent.com/topic/28148-syncignore-behavior-not-as-documented/#entry79714

See also:

http://forum.bittorrent.com/topic/14260-syncignore-is-great-but/?&hl=syncignore#entry38652

http://forum.bittorrent.com/topic/14260-syncignore-is-great-but/#entry43952

Excluding some files from syncing by modifications

You can also exclude some files from syncing on the r/o nodes after they have been downloaded, by either simply deleting them, in which case you will no longer receive any updates on them, or by renaming them, or by modifying their contents, which is only applicable to non-binary files in most cases.

But this trick is not a clean solution. First of all, it won't prevent those files from the initial download. Then, if, at some point, some other files may stop syncing for you for a variety of reasons, and if you have excluded some files by modification tricks, other than renaming the file, and you decide to restore their original update/propagation mode via Folder Preferences -> Advanced tab -> "Restore modified files to original version" checkbox, then all those files that you have excluded via modification trick will be restored to their original state, which is exact copy of the file on the master node.

But this should work fine and clean if you rename the existing files in order to fix some version of the file so that it won't be updated from other nodes and not propagated to other nodes as well.

But file deletions and modifications could not be considered a clean solution as a general case. So, use the .SyncIgnore file to make it clean and compatible with BTSync documentation and recommendations by the team members.

How to run BTSync as a service

fronik
Posted 03 November 2013 - 10:05 PM

Create portable version

  1. First Create a BitSync folder on your PC or portable USB device
  2. Put the "BTSync.exe" inside the folder you created
  3. Inside the same folder create a new empty "Text Document" file and rename it from "New Text document.txt" to "settings.dat"
  4. Start the BTSync.exe
  5. Config all settings

After you run BTSync as any services (instsrv srvany - work, nssm - work) from any user (example system account).

I have everything working.

Sorry for my bad English...

Start Sync as service on Windows
http://forum.bittorrent.com/topic/18326-start-sync-as-service-on-windows/page-3#entry71462

Encrypted folders

You can create the encrypted folders with a little bit of funky manipulations. BTSync does have this ability even though it is not presented in a clean way via UI. Furthermore, creating the encrypted folders on nodes that do not have the hardware support for encryption will significantly slow down the process and will load the CPU significantly. Nevertheless, here is the instructions to create the encrypted folders as presented on BTSync forums:

Updated Instructions

You can generate encrypted read-only secrets [folders] using the normal btsync client without any API key.

  • Do the normal "Add a Sync Folder",
  • click "Generate", but change the first letter of the "Folder sercet" from "A" to "D" (see 1 and 2),
  • set the "Folder to sync", click "OK",
  • right click on that folder from the list,
  • click "Show Folder Preferences",
  • copy the "Read only secret" (see 3),
  • paste it into Notepad(or other text editor),

"Encrypted Read-Only Secret" is the first 33 char of that string with the first letter changed from a "E" to "F" (see 3 and 4)

real example this time.

1) AR7GC6JIVCTKG2XNPM7GGOSV3FI5BDDNH
2) DR7GC6JIVCTKG2XNPM7GGOSV3FI5BDDNH
3) EYF7Y3OOYZEZALVLFDQDAARXQTV2HO7IZCO2V2ZRZEAN32VY7RFH7HGOKRI
4) FYF7Y3OOYZEZALVLFDQDAARXQTV2HO7IZ

Important

You can not have same folder more then once per machine even if it is in a different mode(RW,RO,encrypted). Try the "Secret" on an other machine or remove the RW folder from the machine before you try to add the "Secret" or you will get an error "Selected folder is already added to BitTorrent Sync."

Fun Fact

RW Secret can be "A" or "D" followed by any 32 character from upper case A to Z and numbers 234567.

So this works

DABCDEFGHIJKLMNOPQRSTUVWXYZ234567
EYRCW2XHXF3NRDT4Z44CL45Y5ZH2HO6ADW33KXUGY4EZN4B5RQDELP7IQQE
FYRCW2XHXF3NRDT4Z44CL45Y5ZH2HO6AD

Hope this helps

(Red Diode)

Generate Encrypted Read-Only Secret Without Api Key
http://forum.bittorrent.com/topic/25823-generate-encrypted-read-only-secret-without-api-key/#entry76262

Couple of comments on @Red Diode instructions. 

The Encrypted nodes functionality is implemented in code and therefore is working. There is no UI for it now, so it can be accessed either via modifying Secret's prefix (as in instruction above) or via API. Note, that adding even a single encryption RW or RO key increases CPU load significantly as BTSync has to calculate also hash for all the files in the folder when they are encrypted.

While this load can be handled by modern i386/x64 CPUs due to hardware encryption ability, it can be extremely hard for ARM, PPC and other CPUs for low-power platforms. So I strongly do not recommend using encryption keys if any mobile / embedded platform is involved, only for desktop / server PCs.

The encrypted read-only Secret is very long as it consists of 2 parts: the access Secret (first 33 letters including the 1 symbol prefix) and the decryption key. So, by taking the first part only and putting the "F" prefix you indicate peer that the folder is going to contain encrypted files and peer will have only access key without ability to decrypt (as it physically does not contain decryption key). So it is safe from security POV.

Putting the whole long key (when adding new folder) starting with "E" will make a read-only node, which decrypts files.

Again, use with care due to high CPU load.

(RomanZ - Sync Support)

Generate Encrypted Read-Only Secret Without Api Key
http://forum.bittorrent.com/topic/25823-generate-encrypted-read-only-secret-without-api-key/#entry79112

Now a few questions. In the above:

2) DR7GC6JIVCTKG2XNPM7GGOSV3FI5BDDNH
3) EYF7Y3OOYZEZALVLFDQDAARXQTV2HO7IZCO2V2ZRZEAN32VY7RFH7HGOKRI
4) FYF7Y3OOYZEZALVLFDQDAARXQTV2HO7IZ

Can all three of these secrets be used usefully at the same time on different machines:

Does #2 still work for normal client sync?

Does #3 still work for Read Only sync?

And if Machine A is using #2 and Machine B is using #3, can Machine C use #4.

#2 is Read-write secret with encryption. It behaves completely as usual secret, but Sync will also calculate encrypted hash tree, so encrypted nodes can connect to such folder.

#3 is a read-only secret with decryption. It will work exactly as regular read-only peer, but can connect to encrypting peer with secret in #2.

#4 is read-only-encrypted secret. Peer with such secret can receive files. store them on disk but can't decrypt them and show their content.

#2 and #3 are loading CPU more due to necessity to encrypt / decrypt data.

(RomanZ - Sync Support)

The main caveats for user (average user, not very tech savvy) are:

1) complexity of usage (more kinds of secrets are confusing)

2) deadly CPU usage of mobile devices due to lack of hardware encryption.

(RomanZ - Sync Support)

I decided to be the guinea pig and tested this. The directions posted by Red Diode work.

Now I have an unencrypted folder (RW) on my local device, a Win PC. It's synced with an encrypted folder (RO) on a remote Linux VPS. I made edits on the PC and the encrypted version was instantly updated on the VPS.

Now for the questions:

(1) I am pleased that the VPS host can't see the files stored on their server. However, neither can I. Let's say my PC was lost, and I wanted to recover and decrypt the files. How would I do this?

(2) Since the remote VPS is "always on", I would want it to be the device in my mesh that propagates syncs to multiple users on their devices 24x7. Won't they be receiving an encrypted version? How would one set up the BTSync mesh so that all users see the unencrypted files EXCEPT for the VPS?

Just to verify your comments above: the VPS is E-RO (encrypted read-only), not RO, correct?

Scenario (1): If you were to lose your primary PC and the only copies left are the E-RO backup all you need is the original RW secret. When you place that RW key on a new PC it will automatically download and decrypt the files from your VPS. In other words PRINT and SAVE those keys somewhere really secure. Something like a safety deposit box.

Scenario (2): It sounds like you're already doing this. When you add the sync folder to the VPS make sure to use the E-RO key. Any other computers that you want to see the actual files (decrypted) should use the RO or RW regular keys. The VPS will still hold all the files, it just doesn't have the decryption key to see them.

I've been doing the above with a VPS at backupsy. It runs E-RO secrets for all my important files so I can edit something on my desktop in the morning, shut it down, then pick up editing the same file from my tablet sometime later in the day. All seamlessly synced to Backupsy.

Generate Encrypted Read-Only Secret Without Api Key
http://forum.bittorrent.com/topic/25823-generate-encrypted-read-only-secret-without-api-key/#entry81177

Here is a shell script to automate the generation of both standard and encrypted peer Bittorrent Sync keys. Works for me, your mileage may vary.

It's not the cleanest script - any edits or improvements are welcome!!

Hope it's useful.

https://gist.github.com/anonymous/10002835

Generate Encrypted Read-Only Secret Without Api Key
http://forum.bittorrent.com/topic/25823-generate-encrypted-read-only-secret-without-api-key/page-2#entry81923

BTSync miscellaneous notes

Are claims of security and encryption real?

Well, this is one of those issues BTSync is not likely to make a public statement about, and even if they do, how much do you think you can trust all those claims about traffic encryption and security made by BTSync? Consider this:

As a key part of a campaign to embed encryption software that it could crack into widely used computer products, the U.S. National Security Agency arranged a secret $10 million contract with RSA, one of the most influential firms in the computer security industry, Reuters has learned.

Documents leaked by former NSA contractor Edward Snowden show that the NSA created and promulgated a flawed formula for generating random numbers to create a “back door” in encryption products, the New York Times reported in September. Reuters later reported that RSA became the most important distributor of that formula by rolling it into a software tool called Bsafe that is used to enhance security in personal computers and many other products.

Undisclosed until now was that RSA received $10 million in a deal that set the NSA formula as the preferred, or default, method for number generation in the BSafe software, according to two sources familiar with the contract. Although that sum might seem paltry, it represented more than a third of the revenue that the relevant division at RSA had taken in during the entire previous year, securities filings show.

A secret sweetheart deal between NSA, technology firm
http://www.blacklistednews.com/Secret_contract_tied_NSA_and_security_industry_pioneer/31362/0/38/38/Y/M.html

Interestingly enough, one individual created a thread on BTSync forum where he brought up an issue of cooperation of BT with NSA. He says the thread was pretty popular, but not a single comment was made by BT about the issues he brought up, and those issues were related to security of BTSync and claims made as far as traffic encryption goes. Finally, that thread was simply deleted without comment. Among politicians, such things are equivalent to "no comment".

Here is a little article on BTSync forums about FSF attempts to create an Open Source BTSync clone and ramifications of the fact that BTSync is not an open source in the context of the NSA and other agencies' global and invasive spying.

http://forum.bittorrent.com/topic/21048-defensive-open-source/#entry55992

Which hosts are silently contacted by BTSync?

BTSync silently contacts a number of hosts for various reasons without informing the user. Some of those hosts, such as trackers and relays, are automatically contacted and there is a specific URL which accesses the lists of automatically contacted hosts (see below).

BT claims that they do not collect any information beyond traffic statistics. But, it knows all your keys to all your shares, and so, from then on, to get any kind of information about any nodes and shares represents no difficulties whatsoever. And if you look at it from the standpoint of allegation of cooperation with NSA, then spying is simply implied with all its consequences.

Here is some information on it from BTSync forums:

Add This To Your "hosts" File and BTSync Can't Call Home

"127.0.0.1 t.usyncapp.com"
"127.0.0.1 usyncapp.com"
"127.0.0.1 www.usyncapp.com"

http://forum.bittorrent.com/topic/17002-btsync-calling-home/#entry69356

$ curl -L http://config.usyncapp.com/sync.conf

{
"trackers" :
[
   {"addr" : "54.225.100.8:3000"},
   {"addr" : "54.225.196.38:3000"},
   {"addr" : "54.225.92.50:3000"}
],
"relays" :
[
   {"addr" : "67.215.231.242:3000"},
   {"addr" : "67.215.229.106:3000"}
],
"mobile_push_proxies" :
[
   {"addr" : "54.235.182.157:3000"}
]
}

http://forum.bittorrent.com/topic/17002-btsync-calling-home/#entry89117

List of domains may be accessed by btsync

    $ strings /home/btsync/bin/btsync \
      | grep -E '[^ ]{3,}\.com' \
      | sed -e 's#.*://##g' \
            -e 's#\.com.*#.com#g' \
      | sort -u

    apps.bittorrent.com
    bench.utorrent.com
    config.usyncapp.com
    download.utorrent.com
    gui.com
    link.getsync.com
    raptor.utorrent.com
    remote.utorrent.com
    router.bittorrent.com
    router.utorrent.com
    search.conduit.com
    syncdev.bittorrent.com
    sync-help.bittorrent.com
    tracker.openbittorrent.com
    tracker.publicbt.com
    update.utorrent.com
    www.bittorrent.com
    www.usyncapp.com

http://forum.bittorrent.com/topic/17002-btsync-calling-home/#entry89974

What do the device icons mean?

The icons that appear to the left of the device name in Devices tab mean:

BTSync Direct connection icon Direct Connection - BitTorrent Sync is able to establish a direct connection to this device

BTSync In-direct (Relayed) Connection In-direct (Relayed) Connection - BitTorrent Sync isn't able to establish a direct connection to this device, and therefore has connected via a intermediate relay server. It is preferable to have direct connections wherever possible, as this will generally provide faster transfer speeds over that of a relayed connection.

If you change, delete or modify some files

If you change, delete or modify some files in any subfolders, the effects will depend on whether you did it on the master (r/w) node or the slave (r/o) node. For more detailed description see:

BTSync behavior table

Generally, if you did it on the read-only (r/o) node, then you will no longer be updated on any modifications or updates of those files. They will be considered as non-syncable or "ignored" from then on.

Also, your changes will not be propagated to other nodes and will be considered non-propagatable to other nodes. These files will become your private version of whatever information they contain. Furthermore, at least in some cases, their original propagation status can not be restored from then on, no matter what you do. The only way to do it in some cases would be to remove that share/folder from BTSync (but not physically from the file system) and recreate it again in the same exact folder with the same exact key.

But if you did it on on the read-write (r/w) node, then the behavior of btsync will be quite predictable, just like if you did it in the O/S file system.

File update history: Finding out which files were updated or modified on the master side

BTSync has the ability to keep history of updates. Whenever some file is changed and a new version of it was downloaded, the latest copy of it, just before the change, will be saved in the .SyncArchive folder at the same place in the directory hierarchy as the original file was. Its file name will have the numeric file version number appended to the file name.

There is also an option "Store deleted files in the SynchArchive". It can be accessed by clicking on the Folders tab, right clicking on the folder, selecting "Show folder preferences", then Properties tab and then clicking on the "Store deleted files in the SynchArchive" checkbox. If this checkbox is not checked, no previous revisions will be kept.

All the files in the .SyncArchive directory tree that are older than the default value of 30 days will be automatically deleted. Alternatively, you can delete them by hand without any harm to operation. You can even delete the entire contents of the .SyncArchive folder. But do not deleted the .SyncArchive folder itself.

There is a parameter that can be set - sync_trash_ttl from Preferences tab. Click on Advanced button at the bottom and then you can change the number of days to keep your previous versions. After that many days, the oldest files are deleted automatically.

How BTSync knows about new files?

BTSync uses 2 mechanisms to detect new files in sync folders:

1) system notifications. Very fast technique, but it is not fully reliable and hardly depends on OS notification mechanism. To a make sure that all changes are detected, there is a second way:

2) full folder rescan every 10 minutes (configurable).

(RomanZ, Sync Support)

http://forum.bittorrent.com/topic/28071-starter-problems/#entry79054

Normally, on Windows and on some recent Linux versions btsync knows about new files and updated or deleted files within a few of seconds. Then, after a few more seconds, typically within a minute or so, it performs the transfer to all nodes that have that file updatable.

What are .SyncID, .SyncIgnore, .SyncPart, .SyncTemp .SyncOld and .!Sync files and the .SyncArchive folder?

When you add a new folder to BitTorrent Sync a number of hidden files/folder are automatically created in the folder. The purpose of each are as follows:

.SyncID = A file containing the unique, internal "ID" of the folder. This file should not be manually modified, or deleted. If you do, the folder will no longer be recognized in BitTorrent Sync.

.SyncIgnore = A user editable file allowing you to "exclude" certain files/sub folders for being sync'd

.SyncArchive = BitTorrent Sync, by default, won't actually delete any of your files/folders. If a corresponding file/folder is deleted on another device, it will simply be "moved" into the .SyncArchive folder on all other devices, rather than being permanently deleted.

(In Sync pre v1.1.40, this folder was named .SyncTrash and was later renamed to .SyncArchive as this folder now stores both local files deleted on remote

See: http://forum.bittorrent.com/topic/17782-bittorrent-sync-faq-unofficial/,

When a file changes, does BitTorrent Sync transfer the entire file again, or just the part that's changed?

Files smaller than 4MB are transferred again in their entirety when changes are detected.

Larger files, however, are split into 4 MB "chunks" of data, if one chunk changes, but the others don't, only the chunk that's changed will be transferred (instead of the whole file again).

(Source)

Source:
BitTorrent Sync FAQ *unofficial*
http://forum.bittorrent.com/topic/17782-bittorrent-sync-faq-unofficial/

How long are files kept in .SyncTrash/.SyncArchive for?

From v1.1.30 onwards, deleted files are kept in .SyncArchive (or .SyncTrash pre v1.1.40) for a default period of 30 days, after which they will be automatically removed.

You can change this default period by adjusting the advanced "sync_trash_ttl" setting.

See: http://forum.bittorrent.com/topic/17782-bittorrent-sync-faq-unofficial/,

What data is collected/seen by BitTorrent Inc when I use Sync?

According to BitTorrent Inc, "There are 3 points of contact with BitTorrent infrastructure: tracker, relay, and check for update service. If you don't use any of these - we will not get any information about you."

For more information on what data is seen by BitTorrent Inc, please see this thread

Source:
BitTorrent Sync FAQ *unofficial*
http://forum.bittorrent.com/topic/17782-bittorrent-sync-faq-unofficial/

Note: well, this is not as simple as it looks on paper. The thing is that btsync has a number of BitTorrent Inc's trackers, that are prewired and/or hardcoded, exist in the database or various config files. One of them is r.usyncapp.com.

This means that no matter how much anyone tells you that if you turn the BT trackers off, they "do not collect anything", how much can you trust them, especially in the context of NSA type of agencies that some claim are in fact involved with them? Basically, even if you disable anything you want in your configs, they can still connect to their servers and simply flip a bit in btsync, perhaps even temporarily, and then they have you basically totally open to whatever they wish to do with your node.

See: About minimum firewall requirements...

picosync / workingDraft
https://github.com/picosync/workingDraft/wiki/Protocol

malwr analysis
https://malwr.com/analysis/NjUwMjM5YzJlNDVhNGJhZDhhZDY1NTg1MzQyYzAyYjI/

Edit History

You can keep adding your entries to the list below. Identify your node, so others can communicate to you via their Device name message. Currently, there is other way to communicate in BTSync, even though we have proposed it to them on the forums nearly a month ago. But they don't seem to care about this issue.

You might want to add the GMT/UTC time stamp to your entry.

BTSync Attitude

In this section we will try to clarify the issues of "attitude" of BT, the state of the product and issues of "priority items" on the list of priorities, at least the way it all manifests itself as you can see yourself on the BTSync forums.

Is this a real product, or just a prototype

This is probably one of the fist things anyone would like to know when dealing with some new product.

"The bottom line", as far as BTSync as a product and its state of completeness go, it is FULL of problems. Literally "full of it", as they say. Luckily, most of those problems are not so bad as to prevent one from using the program, but MANY of them create the situations for the users that it simply does not work at all, and because of the most irrational reasons.

Furthermore, displaying the important information to the user, so he could understand what is going on and track some issue down is simply horrible. And it is as primitive as it gets. Even to classify BTSync as a professional product from this standpoint is simply laughable, if not pathetic.

And the program is full of bugs. But, at least if things are done with understanding, it does work eventually, if you know how to avoid those mines while walking on a mine field, which it is.

So, we have tried to provide the most necessary advice and our opinion on BTSync forums, but all of it was simply ignored, and it eventually even went as far as "last warning" threat, which is simply nothing short of insane.

We have done such a tremendous amount of work on their pathetic documentation that even a representative of BT corporation contacted us with thanks and asked for permission to use it in their own docs, which we initially gave them. But when all this insanity and utter denial of the issues and problems that are as real as your own five fingers, which ended up int the most disgusting insults and threats, revealed the true nature of what we are dealing with. At that point our support was totally withdrawn and they were told specifically that they have no permission to use our documentation in any way, shape of form.

Why should we spend months of time of the developers of the most senior level there is to provide any "free" help to those, who keep their sources closed, exhibit "piss on you" attitude and go as far as threats?

So, all the information and assistance to BTSync in our most detailed posts with analysis and proposals and requests for the most basic changes, that would take in most cases minutes to implement, were edited out and the most valuable content was removed.

Let BT know this: you guys have to do your work yourselves. You have already received the assistance worth a mid 4 figure dollar amount, if this were done under the contract. So, enough for you.

Your attitude "sucks" and your product "sucks" just as well. But it does work, even though just barely. Unfortunately, at this particular junction there isn't yet a product that is ready and cross-platform compatible, even though Hive2Hive http://Hive2Hive.org/ library, written in Java, is already available in its initial final release version, which means that we are going to see the first open source projects appearing in no more than 3 months. Actually, with all that library has in terms of features and interfaces, it should take about 1 month to create a product compatible to BTSync in terms of features and power. And that product is an open source library and it is much better designed than BTSync can even dream of, at least from the preliminary estimate and initial look at it.

So, as far as BTSync goes, it is our opinion that this product has no future, regardless of how many zombies are going to use it to download some psychotic "music" to their mobile devices. Because it is a closed source product, and on the top of it, it barely works, and, instead of fixing bugs, they keep adding new bells and whistles to the product they have announced to be in beta phase, which means nearly stable product, ready to be released.

The rules for the beta products are simple enough: You do not introduce ANY new "features" or bells and whistles. Beta means exclusively bug fixing and problem resolution phase. But the way BTSync looks to us, it isn't even in alpha stage. Basically, it is in experimental or prototype phase, and we are seeing somewhere between 30 to 50% of people having various problems and can not even initially sync, which is simply bizarre.

Under no conditions, except of some network related issues, should the user be prevented from the initial sync, that is from downloading the whole share as it exists on the master nodes.

As far as computer time difference errors, we have proposed a solution that would fix this problem in minutes. But they simply exhibited a "piss on you" attitude, like this problem is purely a user error issue and said something like: "we'll see what we can do to HELP users to deal with this problem".

WHAT? HOW can you possibly "help" general public that just installed your program and it does not work? First of all, it is not users that need "help". It is YOU that need all the help you can get. Because there is sufficient evidence that it is not a user problem. We have seen cases where we observed this error with several nodes simultaneously and when we adjusted our clock time to match two of those nodes that showed this error and that error was the same for them, still, one of them would show the error while the other one no longer showed it. Yes, this issue is not as simple as it might look without detailed analysis, but it does not look like a simple user error where user is simply so dumb that he can not see his computer time clock. It simply makes no sense.

Secondly, how can you "help" some user if you don't even know who that person is? You can not send him a message with advice and provide him a link to a detailed document which explains the nature of his problem. HOW can you possibly "help" him? By doing what kind of miracle?

So, we will present a few selected posts here and then will do some editing and leave only the most interesting things in it.

A member of BTSync support team said: "we do care about our customers". - No, you don't. You don't even care about the highest level developers giving you a free advice on how to fix your most ridiculous problems literally within minutes.

Compared to documentation we have created for YOUR product, your documentation, if it can be called such, simply sucks worse than a black hole and it does not answer a single question on the most critical issues that prevent users from even doing the initial sync, not to mention the most ridiculous problem with incomplete syncing and so on.

Shame on you, BTSync and double shame on you, BT. You even have guts to claim the increased security as a result of encryption of traffic. But, as one security professional said: "you can not possibly make any claims regarding security and encryption for a closed source product". Yes, PRECISELY correct. And a hole in security may turn out to be in the places no one even suspects, especially in light of some claims that BT is cooperating with evil agencies of this satanic NWO scheme of taking over the world, such as NSA, for starters. One person even said that he has observed certain things happen that clearly indicated to him that they are in fact cooperating with the NSA.

Yes, to verify it from the outside is not an easy job to do. They are not going to announce some things in public statements, but the very fact that some thread dealing specifically with the NSA cooperation was removed and not a word of comment has been provided by BT staff regarding the extent of their cooperation with NSA, if any, simply indicates the same thing as "no comment".

So, this is going to be the context for what you will see in the following chapters. Except we do not have time at the moment to organize it all and make it nice and clean and simple and clear to understand. So, if you are interested in reading it, just keep patience. We'll get there eventually. We are not going to let these issues "slide".

But some things might be too technical for some users and you might find too much detail in descriptions of the problems and the solutions proposed. So, it might not be such an easy read for some, unless they are curious to know more of what they are dealing with.

We are going to discuss the following issues:

At the moment, we will only drop a few posts from the BTSync forums here...

Time difference error issue

Here's just a few posts from BTSync forums. This needs an initial edit. Right now, it is just a "dump" of sorts.

Device time difference more than 600 seconds

Posted 26 Mar 2014 12:33 PM

May be we should switch the subject to something more entertaining.
So...

[the rest of this post has been removed]


Edited by GreatMarko, Today, 12:49 PM.

stanha, please keep your contributions relevant, on-topic and avoid spamming the forum with irrelevant/off-topic posts

Nice, eh? Well, the original post he had edited contained the following (in response about "we are going to see what we can do to help the users" in dealing with timediff problem):

You really have a sense of humor.

May be we should switch the subject to something more entertaining.
So...

There was one man, U.G. Krishnamurti, who is considered to be enlightened. In one of his interviews, after hearing the same kind of stupid questions again and again and again, he finally told the reporter:

You are not really interested in hearing anything new with your questions. ALL you want to hear is that which reinforces that, which you already know.

In my personal opinion, this is one of the most brilliant things that has ever been said by any man in the entire history.

http://forum.bittorrent.com/topic/21850-device-time-difference-more-than-600-seconds/page-2#entry80759

Mind you, we are talking about one of the nastiest problems of BTSync as a result of which new users that just joined some share would not even start to download it because they would be presented with the error message "Sync stopped: Device time difference more than 600 seconds". The reason for that error is that BTSync "thinks" that the user's clock is set incorrectly and, therefore, it was impossible for BTSync to determine if his version of some file is newer or older than the version of the same file on other nodes.

It is quite a nasty problem as far as technicalities of it go, and it would require quite an explanation of what is taking place and why, which we will omit here to avoid bringing up purely technical issues. But "the bottom line" is this: this particular error is as real as your own five fingers and at different times we have seen up to 30% of users on some shares that would simply sit there and their nodes would not even perform the initial download of the share, which is a disaster of an issue. Because to them, this whole thing would simply look like: "well, this thing does not really work", and, as a result, after spending some time sitting there and waiting for the sync process to start, they'd simply leave the share eventually.

And about the most ridiculous aspect of this error is simple: Under no circumstances whatsoever the node should be prevented from the initial full download of the data. Period. Simply because there exist no technical reasons to prevent even the initial download, regardless of which version of which file is seen by BTSync as "newer" or "older" than on some other nodes.

The initial sync simply MUST be able to download even those files that could theoretically turn out to be "older" versions as seen by some nodes. Because initially, it is utterly irrelevant which version is newer or older, at least compared to utter inability to download the file itself. After initial sync, BTSync would be able to update some file to the newer version and the process is fully automatic. There isn't even a thing to be done for it to happen, or even talk about. It already works "as is". Because this is the very nature of the sync process.

So, what could possibly be the technical reason to prevent even the initial download? Therefore, this entire issue is simply ridiculous, if not outright bizarre, from the standpoint of program logic and function. It is a purely artificial issue which can only exist in a totally broken and utterly inconsistent design.

But when we did some investigation on this issue and have discovered some pattern of behavior indicating that it is not merely a user problem but rather a flawed design, we were simply laughed at. So, eventually, seeing all the hopelessness of the situation and utter denials, we have posted the above message which pointed out that they simply deny the problem as such and were trying "to talk it away".

So, what was so deserving to be edited, which is considered to be an intellectual crime among the writers? Because by editing someone else's post, you may distort and pervert its very meaning and the essence of the issues, which is EXACTLY what happened in this case, and in the most perverted way, done by a man of profound dishonesty, and making it look like not an insult of intelligence of an author by the most ridiculous responses "we'll see what we can do to HELP the user", knowing full well that no such help was possible.

And that is precisely the reason it was said "you do have a sense of humor, I have to admit".

And a quote by U.G. Krishnamurti was "right on the money" in the context of utter denial of the issue as such, without ANY investigation.

What "spamming the forum with irrelevant/off-topic posts" this bozo, full of self-pride and having access to the big red button, labeled "kill", is talking about? You see, he started imagining that he has "power" with his "kill" button, and he can edit the posts of other people, and he can delete the posts, which is what he has done in an attempt that exposed his dishonesty when he lied and tried to make this timediff error issue look like something no one cares about, which isn't the case. Because, in reality, quite a few people read that thread about timediff error.

And so he simply deleted the post that contained a single line of text essentially, saying that only during the last 24 hours there was 90+ views of that thread. What a coward!

So how it could be so insignificant if nearly 100 people read it every day? What does it tell you, unless you are a complete zombie having a black hole in place of a head? How many thousands of people do you need to see reading that thread before you consider it as something significant?

Because the original post was "right on the money", and it was addressing the way this most ugly issue of timediff error was affecting about 30% of all the people on one of the most important shares, not counting those, who came, wasted a few minutes, and not seeing it "working", simply left.

Interestingly enough, this issue is known to BTSync for at least 8 months. But it has NEVER been really addressed or investigated, at least until we pointed out that this must be a time zone issue related to DLS (daylight saving) time adjustment, and, furthermore, there were indications that it would be present even in the situations where users' clocks were in fact set correctly.

And every time this issue was brought up, their canned reply was simple and essentially said: users are stupid and have no idea of how to set their clock so it would show the correct time. Can you imagine this level of "brilliance"?


Device time difference more than 600 seconds

stanha
Advanced Member
88 posts
0 warning points

Posted 26 March 2014 - 10:50 PM

[the rest of this post has been removed]

I do not see how this post is relevant to anything beyond personal vendetta and desire to dominate.

Normally, in the Silicon Valley, which is where all my clients were, no testers and even customer support reps are allowed to contact developers directly. They can do it ONLY through their own manager, and if they do that, they get one and last warning, and if they do it again, they are fired on the spot.

And I happen to be a senior level developer and a contract consultant. So, this whole thing is something unreal to me. I simply have no idea how do deal with the guys like you because I have no personal experience of the frame of mind of people like you.

But one thing I know, is that deleting the posts that contain a single sentence as to your misleading representation that this issues is something of not interest to anyone, while in fact, there IS quite a number of daily views on this thread, is simply a futile attempt at "covering the tracks" and it unethical and dishonest.

And so is editing the posts of others. Among the writers, it is considered an intellectual crime.

I am really surprised that people like you are given "authority" to delete and/or modify the posts of others.

And you are following me with your pathetic attempts at showing your "superiority", just because of you have access to the big red button, called kill.

Look, pal.
This isn't funny.

And now this issues is going to get into the Detailed User Manual for BTSync, no matter what you do, change, edit or delete.

And the more you do it, the more your destructive energy may be reflected back to you, only in a magnified way.

To me, you are not even funny, but a nuisance that hurts the product more than it helps anything.

So, I hope someone at BT will be able to put things in proper perspective and stop this harassment.

I am here to HELP the product and to provide the advice an opinion for which I normally paid the kind of money, you probably never heard about.

So...

Get off my back, will ya?

And stop interfering in the affairs of others, just because your mind is full of hate and personal vendetta.

Because at this point you are HURTING BT as a company and BTSync as a product, whether you realize it or not.

I do not wish to waste my creative energy on people like you.

http://forum.bittorrent.com/topic/21850-device-time-difference-more-than-600-seconds/page-2#entry80840

Device time difference more than 600 seconds

Advanced Member
Members
PipPipPip
88 posts
0 warning points

Posted Today, 10:50 PM

[the rest of this post has been removed]

<again removed because it's an abusive personal attack instead of an on-topic point for the thread. You were told to drop it and you haven't. Final warning.>

Edited by Harold Feit, Today, 11:22 PM. Removed abusive personal attack.

http://forum.bittorrent.com/topic/21850-device-time-difference-more-than-600-seconds/page-2#entry80840


Device time difference more than 600 seconds

stanha

Advanced Member
Members
PipPipPip
88 posts
0 warning points

Posted Today, 10:17 PM

All in all, this version seems a big step backward in terms of stability. I think you should pull it back.

Nicola

I would agree to that.

There is one interesting release related issue here. In my experience and understanding, beta is basically a nearly stable official release of the product.

At beta phase, no new bells and whistles are added and no new functionality and, especially, the changes of behavior are to happen.

The very definition of term beta is bug fixes ONLY. It is a final problem resolution phase of the product.

Secondly, not making releases in small and gradual improvements, and accumulating massive amounts of changes and/or bug fixes, you risk to be exposed to the bomb effect, when things massively do not work and you see tons of issues appearing out of nowhere, as it happened with 1.3.67 release.

For me, personally, probably the worst "improvement" was replacing folders in both, Devices and Folders view with only the last element of the path. That is a disaster for me, since I have several shares that now have exactly the same name and once I switch to one of those tabs, I have to start thinking, moving my mouse around and trying to recall what relates to work.

That is just an additional load on perception. There is such a notion as "burden of perception". Perception is not "free". It takes energy and concentration, and, in this case, utterly unnecessary. I work with several totally different projects in totally unrelated area as a normal mode. That means, when I switch from the text editor, or coming back from site status statistics and some other things, it is like coming from a totally different world. Sometimes, it is not even easy to type as you just worked with totally unrelated language.

And, most importantly, what does this new folder presentation IMPROVES?
What does it add?
What does it clarify?

Instead of providing MORE useful information, it makes it LESS useful.
When I switch to Devices or Folders tab, which, in my opinion, more appropriate name for what is called My Sync. Because EVERYTHING in BTSync is about "My Sync", every single parameter and every tab in every dialog page.

When I switch from one tab to another, I'd prefer to have as complete of a picture of everything as possible without making ANY additional efforts on my part. I do not have spare or idle energy to throw around.

So, to me, to call this product beta stage is simply misleading.

About the only thing that is of any use or changes anything, as far as our own situation is concerned, is that additional tooltip that shows pending transfers. I like that one indeed. Because it really ADDS more information and gives a fuller picture of the state of the nodes.

I helped me to see exactly the reason I consistently see the incomplete sync with couple of nodes. Because it shows exactly which files remain to be uploaded, even though they were uploaded at one point, but were modified on the r/o node and they do not even realize what they did, most likely.

So, if BTSync still continues to develop and/or add all sorts of new bells and whistles, for marketing purposes or otherwise, it would be more appropriate to reclassify BTSync as Alpha state.

In beta state, you are dealing with basically a releasable product where only the final touches remain in terms of fixing bugs ONLY. It is a state of the product when massive testing occurs with all sorts of test suites or otherwise.

I am still willing to run it for a day or two, the way it feels right now, just to see what other changes of BASIC behavior I see in other nodes. And it looks like I am risking some very unpleasant potential consequences, looking at all the posts on this tread and other 1.3 related threads.

http://forum.bittorrent.com/topic/28597-bittorrent-sync-13/page-3#entry80837

User interface design issues

File size and transfer progress in Transfer tab

stanha
Posted 29 March 2014 - 11:40 AM

It would be very useful to see the File size and transfer progress in percentage points in the Transfer tab. Otherwise, it is problematic to estimate various things and see the progress.

Plus, it could prove useful to developers from the standpoint of optimizing the performance related issues, especially in cases of smaller files (between 30 and 200k approx.)

Update #1

It seems the contents of Transfer tab should be logged into a separate log file so that one could see what was actually transferred. Just a few minutes ago I noticed one node coming on line after several days of absence and there was quite a bit of changes in the files since then. He had over 100 megs to download, which is strange. When I returned from the kitchen, he was gone and was fully synced. I'd like to know what did he actually transfer because it might indicate some issue.

It looks like he began to transfer some pretty big files that he should have had when he was fully synced. But there was an issue with duplicate folders at one point as a result of renaming, and I'd like to know what happened with this particular transfer today.

User Interface Design

Well, I'd like to discuss some issues with BTSync's GUI.

At the moment I'd like to cover the following:

1. General design issues
2. Device tab
3. History tab
4. Preferences tab

General design issues

These issues include the common accepted standards in GUI design, such as ANY displayable field, no matter what kind of field it is and in what column of what tab it is displayed is subject to copy to the clipboard operation.

That is the rule, and not some "luxury".
And it is the BASIC rule.

Many, if not most, well designed programs have the menu bar and a status bar.

In order to enable/disable logging or pause/resume the sync process, it is highly UNDESIRABLE to be performed via right mouse clicking on BTSync icon in the Windows task bar and then select a property choice.

Because it is HIGHLY unintuitive. It took me several days to realize that there are features like these available in BTSync, but ONLY if you use the right mouse click on a task bar icon. And I stumbled upon it only after reading lots of articles on BTSync forums. Why? Where is it described in the manuals?

But what do you have a concept of a program menu for? Why not utilize the STANDARD tools, methods and principles of well designed GUI? Why should I leave the main BTSync window to exit the program? What does it "buy you"?

Why can't you copy ANY field from ANY column from ANY tab in a dialog box to your clipboard, considering the fact that some of the things displayed you will have a hard time to replicate or copy to your editor or to the message on this forum?

Device tab issues

There are 3 columns currently on this tab. The column status is used to display either the amount to upload or the error message(s), in such cases, for example, as "Device time difference more than 600 seconds".

The nasty thing about that error message is that it is not displayed constantly, but toggles every 30 to 60 seconds, to the "Amount to upload", which, in case of timediff error shows totally incorrect amount that is equal to the total size of files in the share, even if that share is FULLY synced.

This design is not only "wrong". It is simply horrible. First of all, because of this totally wrong and horrifying message that tells you that your whole share have just disappeared, even though it is fully synced and all the files are there.

So, what could be a solution?

Well, first of all, if you decide to toggle the messages in the same column (Status), then you have to toggle them much faster, say, every 5 to 10 seconds, so the user has enough time to read it.

Furthermore, when you toggle it from the error message about timediff, to amount left to upload, you do NOT, under ANY circumstances show the user utterly incorrect data indicating that has to download the entire share, even if he is fully synced. If you can not estimate what would be left to upload in case of timediff error, you either show the CORRECT amount, if you can calculate it accounting for all the files that could be affected by the time diff range, or you simply say something lie: "?????", or "Can not be estimated in case of timediff errors".

But you do NOT, under ANY circumstances, present the user totally incorrect information, as it is done today. First of all, on the basis of what kind of logic you have made such a conclusion? Is there ANY logic involved in the process of displaying this utterly erroneous error message?

Or, you simply have 2 separate columns: 1 for the amount left to upload/download, if you can calculate it correctly and reliably, and then you have a totally different column for the error messages, that could also be used for the status messages, such as "Last contacted at ...", or "Fully synced at ...", or some meaningful things like these.

History tab issues

The message "Finished syncing with xxxx" is insufficient in the applications where there are multiple shares. Because it does not tell you which Share it was syncing with. So, it places additional drain on the energy, and, in case some Device has left the swarm when you are reading it, you have no idea which folder they were syncing.

This tab should show not only the messages that were the result of full sync, but also the messages of last contact and the amount left to be up/downloaded in case it could not fully complete the sync. Otherwise, you have no idea of whether the contact was actually made with some node, and whether that contact was successful, at least to some degree and resulted in some transfer. It would probably be great to see like "transferred NNN Kb, NNN Kb remaining".

Because the way it stands currently, is that you do not get ANY messages in History tab if you were not FULLY synced upon the last contact.

So, it seems to me, the message in Event field would look like:

Finished syncing "Folder name" with "Device name" transferred NNN Kb, NNN Kb remaining.

So, this message would provide the complete picture as to the activity and would also give you additional information on the state of the link with that node, such as whether it is functioning or not, whether files are indeed being transferred, even if link is OK, WHEN the last contact was made and so on.

For example, in the case of "Mac mini", described in "Device time difference..." thread, the other day, I have seen this node joining the swarm. So, I decided to help him out and changed my clock time by 1 hr. so he could sync, ans so he did. I was busy with all sorts of other things so I could not sit there for hours watching him to sync 1 Gig. After a while, he was gone, and when I looked in the History tab, there was no record that he did indeed completed the sync. I wanted to know how much is left for him to sync, but there is no information available, which means the operation was partially successful, but not entirely. But to what extent - only God knows.

Or, alternatively, you could add a new column(s) for "Folder name", "Transferred" and "Remaining", so, in case the device is too long to be displayed, it could be expanded in the maximized dialog view and Device column size could be enlarged by the user instead of being simply chopped off and replaced with "..." in case device name is too long.

Also, as far as user alterations of the files (delete, modify, rename) should also be recorded somewhere, even though may be not necessarily in history, simply because of the bulk of it. May be, may be not.

Because those kinds of alterations have profound effects and could have consequences that might not be easy to comprehend at the moment.

And all these modifications are simple, really, and could be implemented in a matter of hours, if not minutes, at least for the most part of it. But the benefit and informativeness of them is orders of magnitude more beneficial in terms of providing the user with as complete picture, as possible.

Does it worth the effort? - In my personal opinion, yes, indeed. Because the payback is something like hundreds to one.

Preferences tab issues

Device name edit field should be expanded, to say the least. Or, alternatively, the Device Message is to be added to this panel to be displayed either in Devices tab separate column, or, like in any half-decent program for Windows, in a STATUS bar at the bottom.

Status bar is one of the required elements in a well designed Windows programs according to Microsoft's design guidelines, known for tens of years.

The user simply MUST be informed about any messages to him, considering the number of various error conditions or undesired or unpredictable effects of various user actions, such as file modifications, deletions, renaming and so on, the results of which may be simply unpredictable to the "newbie" and he might not even suspect that he has done something that will prevent his node from being updated in the future.

So, there needs to be a way to communicate to him a message from other nodes, especially from the master (r/w) nodes, that could give him an advice on what to do in current situation in order to correct the situation and bring the sync to the normal and expected status and condition.

User Interface Design

#3 [User Interface Design: post #3] stanha

Advanced Member
Members
PipPipPip
107 posts
0 warning points

Posted 24 March 2014 - 12:44 AM

The contents of any user *editible* field should be able to be copied to the clipboard - which it can be!

This is obvious to a five year old. Because it is a default behavior of the O/S GUI.

There is however no "rule" that states that every non-editible GUI element should be able to be copied to the clipboard! The information shown on the folders/devices/history tabs isn't user-editible, therefore, there is no "requirement" for it to be copyable to the clipboard.

Yes, the folders tabs themselves are not copy/pastable. But there is a catch here: they are not DATA. They are just like labels, even though I, personally, would prefer to be able to copy the labels as well. For example, if you look at the Windows file - Properties tab, you will notice that generally, data itself is highlightable and copyable. But the labels to the left are not. Why?

Say, for example, I want to copy the entire contents of the Properties dialog box. I may have a huge tree of folders and I'd like to know the exact state of those folders and contained files written down somewhere in one of my documents, such as revision history, major updates to the system, and so on.

Now, do you think it is REASONABLE for me to be able to CTRL-A even on the entire dialog box and it should highlight all the text fields and labels, or to region-highlight any section of the dialog box and copy all those fields at once? - Well, why not? It would be great indeed.

Well, you can certainly maintain this position. But I suggest to look at even Windows explored property dialog, or nearly any dialog or window in the Windows GUI. Just try to mark some text, no matter what it is, and do a CTRL-C to try to copy it to the clipboard.

Once you do that, it would be great if you provide the statistics of all the cases where you could do it in relationship to the cases where you could not do it.

What you might find is that, even though it is generally true, that it is not a required operation, but, nevertheless, in any situation where you have things like labels: and any kind of message or data, you are in fact able to highlight that text, which means it is copyable to the clipboard, which you can verify.

The point here is: since some of the thing that are displayed in various columns of various tabs in GUI, contain text, that could be highly desirable to copy to the clipboard and paste elsewhere, such, for example, as hash codes or long device names or the state of the sync completion and so on, which you may preserve somewhere, but you can not copy/paste those things, it becomes a royal pain on the neck and royal waste of energy trying to save that information elsewhere.

The question here is simple: is it really NECESSARY? How much of a development effort would it be to make ANY data fields, editable or not, copy/pastable?

I agree a menu bar and a status bar are traditional - however, more and more software is moving away from this traditional concept. Take web browsers for example, it's now more common for there to not be a traditional menu or fixed status bar by default.

Yes, I am fully aware of that "new wave" trend, and Windows-8 is a perfect example of making a computer screen look like a smart phone, which, in my personal opinion is simply insanity. I have not installed the Win-8, but I hope the user is at least given a choice to go back to the "classic GUI" version, instead of being forced into this NWO insanity of imposing the idiocy on everyone and forcing them to submit to it, not even giving them a choice.

And I hope you don't mind me deciding for myself things like these.

Note: some further quoted text will be in italics because of the board limitations...

In Sync the traditional "menu bar" has been replaced by simple to navigate tabs, allowing direct access to the key functions of Sync.

But then you have a problem of sorts. How do I turn logging or transfers on and off and exit the program? Well, you need to KNOW that right clicking on the BTSync's icon in the Win task bar there will be a property choice popup.

Question: how would new user know that there is such a functionality available, but ONLY from that icon? Initially, and for days, I had to stop BTSync by hard killing it from the Task Manager, because I did not even notice the Exit option on the BTSync icon. And terminating the process from the Task Manager may have some very nasty side effects, such as abandoned named mutex or other synchronization objects and so on. Or, some drivers may be left running and all sorts of other nasty things.

I hope that when I exit BTSync from the BTSync icon Exit choice, it exits gracefully and fully restores the system to a stable state and unloads all the objects from memory.

A "status bar" of sorts is coming in Sync 1.3. This will show the current up/down speeds and be visible across the bottom of all tabs.

The Debug logging feature is not an "every day" function that every user will need to access. It is an "advanced" debugging function, that will be used infrequently for the primary purpose of debugging Sync problems. Given that Sync is currently in Beta, this option may well not even be present in "Stable" builds! (or perhaps it might then move to the "Advanced Settings" dialog)

Well, to some extent you can claim that debug logging is something for the "expert" users and so on.

But if you look at functionality of debug logging, currently it bulks everything into a single file. But, in my opinion, there is a need for some log of the transfers and their completion status, and, especially, of all the transfer errors discovered.

So, fine, debug logging, just as its name implies, might be considered as something for those who know more than the plain, ordinary users.

But transfer/error logging is something that is a must, in my opinion, and, probably, should be enablable/disablable individually for each folder, and, alternatively, in bulk - for "all" or "none".

But I am not going to argue this case at the moment, because it is nothing more than a royal pain on the neck to look at those logs, containing ALL sorts of information of not very useful value for the bulk of it.

But what I'd indeed like to see in some log, "debug" or some other, is:

1) Connection attemps with node names and hash.
2) Transfer completion entries for each file and their status (success or error, in which case, what was the cause of that error).
3) Definitely errors
4) Descriptive messages of transfer attempts, such as:
Sending the file xxx from folder yyy to peer "pear name". Something of this sort.

So, by merely looking at this log, any user, even not "expert" would be able to see exactly what is going on.

We need to keep in mind that BTSync is much less predictable in terms of the results than, say, torrent programs, and side effects of various user actions on the files locally may have some unpleasant side effects that he might not even realize.

So, it seems to me that informing the user in the most detailed way is one of the key points.

But some, if not most of those things that I see in the log, such as ping this, pong that, received response this, contacted the relay that, are not necessarily the things I like to see in the operation/transfer/completion log.

These are two separate animals to me.

It is described in this pinned thread. The manual directs users to the forum if they have problems, and as I say, the "Debug" function isn't something that most regular users will need to use, unless they encounter a specific issue, for which debug logs are required and for which instructions are provided in the forum.

To tell you the truth, "I could care less" about debug log as it stands right now, except for clear error messages, such as the case with timediff problem.

But I do care for the operation log. That is just the way I am, I guess. Actually, the last program I have developed in Jave automatically generates the running logs and you can click on the GUI checkbox in the main dialog and enable the "detailed" version.

And you know what? Since implementation of that feature, I was able to even find some bugs in a manner of minutes, that would otherwise take me hours, if not days.

Because all the exceptions of relatively important operational significance are ALWAYS automatically logged. And so when I finish the run for some job, which takes hours, and if I have any doubts of anything whatsoever, all I have to do is to open that daily log and look and find exactly the issue of my interest, down to the amount of memory taken by the process and remaining available to the JVM. Because those jobs are humongous in size and process hundreds of thousands of articles.

If I would not have that logging, I would not be able to develop that program to the level which it is in right now, and it is a tank grade program. The last time I have seen a bug, not even talking about crash of any kind, was years ago. I don't even remember how many years.

That is the point I am trying to address.

Proper logging is a tremendous help to both, the plain users and the "experts".
Simple as that.

And sorry, I am not willing to accept the arguments justifying the present state and quality of logging. To me, there is plenty left to be desired, saying it in the mild terms.

Because Sync is designed to always be running in the background, if you've got lots of windows open/programs running, it's often easier to exit Sync by right-clicking on the tray icon (which would take two clicks, 1 - right click on the tray icon, 2 - click "Exit"), than it would be to first locate/open the Sync UI (1 click), click a "File" option in a menu bar (another click), then choose "Exit" from a file menu (another click) = 3 clicks vs 2 clicks!

Well, I do not mind having "Exit" feature on the icon in a tray. But the problem is "newbies" are likely to have "no clue" about it. How do you assure that the most basic things are obvious to ANY "clueless" guy?

Well, the "classic" approach is the Program Menu bar. And I fully agree with it. At some early point, Win-95 period, Windows released a document or an article for the developers with the guidelines for "good program design".

To me, that document was probably one of the best documents and/or advice provided by MS in their entire history. And that document even specified what is the "must have" tabs on the program menu bar. They were, File, Edit, View and Help tabs, if I recall correctly. I was even baffled at that time being surprised with requirements to always have an Edit tab. In some applications, it seemed, there simply no such functionality available, like for example in the image viewers and so on.

And now I am saddened seeing this "new wave" approach in GUI design. To me, quite often it looks quite insane.

You see, with program menus that are consistently used across the multitude of apps, user eventually "gets" the basic idea when using any program, intuitively knowing what and where functionality is likely to be found where in the maze of the program dialogs, functions and features.

But with "new wave" approach, it seems "anything goes" for as long as you have some graphical representation of everything and all the program functionality is accessed like it is done on smart phones.

But quite often, those images are meaningless and are not consistent across the apps.

When I see a text, like "File" in the main menu, I already know, without even accessing that menu that there are going to be the standard choices available there, like open, save, save as, exit and so on.

But hey, everyone can do what they want and go insane with all those flashy images. And the only question I'd have is: why do YOU have to follow the pack? Is there any REASON that provides some ADDITIONAL benefits or clarifies something, say, for example, in the current version of BTSync GUI, where main program windows looks like it was designed on the base of the dialog class, which is not a good idea in my view? Because the most important program functionality of the GUI design is not available to the dialog derived classes, assuming they did development in MFC.

Dialogs are not designed for that, generally speaking. That is why they called dialog boxes, and not windows or views. Sure, you can design anything in any way you like. No one can force you. But... Well, this would be enough for now.

Screenshots are a good way to provide this information in the forum if you need to do so. Additionally, further information will be available in the Sync log.

The problem here is that you have a "newbie" that just installed your program and has not a slightest clue of anything. He may even have a hard time entering the key ("secret"), not realizing that is is done via Generate button, just for the sake of argument.

So, he fires up the program for the first time and manages to create his share. But then, magically enough, nothing happens, or, if he is lucky, he will see the other nodes joining the swarm. But it is not even intuitively obvious that his main operating tab should be a Devices tab. So, he might be sitting there watching the Folders tab and expecting some miracle to appear there. But he does not even see the main action. Sure, eventually, he'll get curious and will look in the Transfers tab and so on.

But, what if he has a problem? What is he to do?
Does he have to come to the BTSync forums as a must?
Does he have to read those sketchy "manuals" that are, sorry to tell you, are simply laughable as they basically describe nothing of the "real world problems" and issues.

As to the sync logs, you yourself just said above that logs are not meant for the "mere mortals". They are meant for the "elite" users and "experts".

So, what are you advising here to the plain, ordinary "newbie"?

There are improvements coming to the devices tab in Sync 1.3. Most notably, you'll be able to see exactly which files are awaiting being synced, or which are out of sync.

I'd LOVE to see that kind of level of detail. Again, with BTSync approach and dynamic nature of the process and unpredictability of some user operations and actions, it is simply a must. The more, the better.

As long as it is not all bulked in one huge log, but is properly and functionally separated into different kinds of logs, and I'd recommend an running rotating daily log, where new log file is automatically created when the clock time passes 00:00.

And again, in case of any kind of difficulties or problems the user may experience, the "Detailed log" checkbox would help indeed in my opinion.

This would be something more for the "Devices" tab - which shows current/active information, whereas the "History" tab shows action which has already taken place/completed.

I agree that a "time remaining" indicator would be a useful addition to have, but as with any suggestion/feature request - please post it in the dedicated wishlist thread.

I did post a few things on that thread with some ideas and/or suggestions. But, at this point, my feeling is that it is nothing more than a garbage dump that no one reads from the developers team. Some of those suggestions were written over a month ago I think and, since then, there were two or three program updates. But those ideas or suggestions seem to have gone straight into a black hole.

So, I no longer have any interest to post to that dump, filled mostly by all sorts of wild ideas and dreams of the people. Yes, all of this would be "nice to have", at least eventually. But right now there are CRITICAL issues in BTSync, that need URGENT attention in my opinion, and if even the most basic things and suggestions simply go into a black hole, then what do you expect me to do?

I can not even convince people that timediff error is nothing short of the disaster, at least in my case, and I mean it in quite a literal sense.

About all I see in terms of response: "tell those clueless to set their clock right". But HOW do I tell them?

I proposed some sort of primitive messaging facility how long ago?
And what is the result?

So, HOW do I talk to all those "clueless", and what if it turns out that their clock and even time zone are set correctly? And now I am nearly convinced that they are in fact set correctly. But which node sees what time zone is not a universally and globally available from some central "authority" for all the places in the world. For example, in Windows, the time zones of several cities in different countries are bulked together. But this may be utterly incorrect. Because in some countries the politicians constantly pass the laws about the daylight saving time, and so the Microsoft time zone, chosen for your city may be incorrect, compared to other sources or the local laws.

??? It's already pretty long! (94 chars max on the Windows app): device_length.jpg

The problem here that yes, it is already pretty long. But if there is some problem with some node, you might have to somehow explain to them and possibly give them the URL where they can look it up, and that message might be longer than 94 chars, which, btw, has been even shortened in later versions.

The problem we are dealing here with is the case where BTSync refuses to move a finger in order to provide even the most primitive facility to display some more or less descriptive message in the device name, which is visible to all other users. This way, they have a chance to see that message. So, that was a "solution" that would take about 5 minutes to implement, but would allow to display a much longer message. Because I want to tell them what is wrong and also provide the url to exact chapter and topic in the detailed user manual where they can read a full description of their problem. And for that, I'd like to see preferably about twice as long the device name edit field.

Technically, the physical size of the edit field does not imply the maximum amount of text you can put there. It is already auto-scrolling. So, you can put even a 1 k long string into it and if you want to read all of it, you can simply do a standard select-all and copy operation to copy it it to the clipboard and then paste it into your favorite text editor to see the full text.

What is the problem with this?

And again, it was meant for the case that BTSync is not willing to move a finger to provide any other way to display some device message. But even having a separate column on device tab for device message should take no more than a few minutes to implement.

Please see earlier comment in relation to the "status bar" coming in Sync 1.3.

Also, please note that Sync isn't a dedicated Windows product - it's cross-platform and the desire is to have a similar looking UI across desktop platforms.

Yes, this is clear even to the five year old, without even mentioning. But you need to first of all create a design, as quickly as you can, that fulfills the most critical needs of informing the users in case they have ANY kind of problem or difficulty.

And you need to provide them with the FULLEST picture of everything.

Question: how do you do that?

What EXACTLY do you propose to be done, if anything, but waiting for new instructions of the party line kind, telling you that you are simply in a state of delusion, and everything is nice an rosy?

There are a number of improvements coming to notifications in Sync 1.3

Well, I am glad to hear that indeed, and I hope when I finally see it, I will not be disappointed.

But I've been hearing these "real soon now" messages for over a month and it reminds me the politicians forever talking about the "bright future".

But where is the substance I ask you?

Where's the beef?

What are the top items on your "priority list"?

I'd LOVE to see that list. Who knows, may be my time would be better spend doing my main things that I do, instead of trying to prove that I am not a donkey with long ears.

The love affairs come and go, and you can spin your wheels, getting nowhere really fast, for only that long. And then come the "boom" part of the affair, and you simply decide to switch to Java Open Source project, such as Hive2Hive, where you can throw together an application compatible in functionality and with even better and more reliable security arrangements in less than a month, in my current estimate.

My hands are beginning to itch. But not enough at this moment and I am already overloaded with things that MUST be done with other areas of my activity.

So...

User Interface Design

stanha

Advanced Member
Members
PipPipPip
107 posts
0 warning points

Posted 24 March 2014 - 02:13 AM

Stanha,

Everyone is welcome to contribute to any topic within these forums, and everyone's opinion is as equally valid as anyone else's.

If you have a differing view to another forum contributor, that's fine - constructive discussion and the sharing of ideas, knowledge, and solutions is what makes these forums a fantastic resource for the Sync community! :)

However, personal attacks, "flaming", threatening or abusive language towards other contributors is not appropriate in these forums.

Your previous post has therefore been edited accordingly.

Please help to keep these forums a friendly and welcoming place for the Sync community.

Thank you for your co-operation.

What?

"However, personal attacks, "flaming", threatening or abusive language"?

Hey, I tell you what, now I really feel that energy going full blast.

Well, what can I tell you, but: it dissolves, no matter what.

Yes, you did indeed help a little bit for my fingers to move.

But, you see, it is simply dishonest to edit the posts of other people.

Yes, you can delete them as a whole, and it seems to me that you have given that "authority".

But I operate in different lands and their name is:

Truth.

Let me ask you: was it you, personally, who has deleted one of my posts yesterday on the other thread?

It seems you have some problem here of personal kind. But you see, in my opinion, you did not provide any solution to any of the issues that are there on the table, regardless of the "priority list".

As I said before: I am only interested in solutions and I like to see things working on the proper, and that is fully professional level. Because just today I have seen 3 people out of 5 on one of my active shares having a problem.

And I do not know how many other people have abandoned the efforts to download the information and receive live updates.

My rough estimate is that it could be 50% of all the nodes I have seen on that share. Sure, statistics is just that - statistics.

And I have provided so many versions of solutions to some of the most critical issues for the general users, that, sorry, but you have to have some other reason to concoct your last message.

So, Be it.

User Interface Design

Harold Feit
Community Manager
Administrators
4,582 posts

#6 [User Interface Design: post #6] Harold Feit

Posted 24 March 2014 - 02:19 AM

You have been warned multiple times by moderation and administration members. Drop it.

What can a poor guy like me say, but "Seig Hail, my fuehrer"!

User Interface Design

stanha
Advanced Member
Members
PipPipPip
107 posts
0 warning points

Posted 24 March 2014 - 02:41 AM

You have been warned multiple times by moderation and administration members. Drop it.

Unfortunately, you are talking to the wrong party in this.

The message I saw as a followup to my post was a total insult.

And this is "official" now:

The support of BTSync product, in any way, shape or form is totally withdrawn by this party.

Whether it means something to some of you or not, this is not of our concern.

BTSync was given what I would classify as tremendous degree of support by the powers that I would not even like to mention here.

That support is no more.

Incomplete sync issue

Node Update Logic Is Profoundly Broken And Needs Redesign

Basically, from what I see, the node update logic the way it stands right now it fundamentally broken as a result of bad design.

It inherently creates the inconsistency across the nodes. The very concept of database to accommodate and save the update state of the share is not a reliable concept in case other nodes do file deletion, renaming of modification.

If, for whatever reason, the database gets damaged for example on the r/w node and one has to delete and re-add the share, all the memory of the last state of the share is lost and all the nodes become totally inconsistent in respect to what they should have in terms of the latest state.

Therefore, unless you implement a database journaling system, you can not possibly guarantee the consistency or database recovery. If database is damaged, which happens no matter what, how do you know which exact files are subject to update and propagation?

Secondly, as it stands right now, there is no such a concept as a "reference node". Therefore, it is impossible to determine what should be the correct state of all the files that might have come and gone as a result of user modifications, deletions and file renaming.

Furthermore, the r/w node can not be considered as a "reference node" or a "master node" with present logic, as it can not fully control what happens in the r/o nodes if they have made some file modifications, deletions or renaming.

Once the r/o node has done some of these operations, it is no longer controlled by the r/w node in respect to updates of the modified files, and, therefore, it is no longer consistent with the r/w node.

From what I see, with current update logic, about the only way to make it work reliably is to prohibit stopping of updates in case r/o nodes performed any file modifications. Those modifications should be undone and files should be restored to exactly the same state as they exist either on r/w nodes or their exact or the "best fit" copies.

If user renames some files, those should remain in renamed state, but the files with the original file name should be re-downloaded.

If user deletes some files because he does not want to download them any longer, he should add them to the .SyncIgnore. Otherwise, they will be re-downloaded again.

If user wants to modify some file under original file name, his modifications will be lost as the original versions will be re-downloaded. Therefore, if he wants to modify them, he should rename them and then edit them the way he wants. But the originals will be re-downloaded in exactly the same version as on the r/w nodes.

Therefore, any files that are not in .SyncIgnore, will be restored to their original state as on r/w nodes.

Else, I see no way, even theoretically, to eliminate what is called a "database inconsistency problem" as known in relational database theory.

It is simply logically impossible with current update logic and design. The very idea of BTSync database to maintain the latest state of the files is inherently unreliable, unless journaling system is implemented. Because, first of all, databases eventually get damaged for all sorts of reasons and only journaling systems may prevent the effects of damage and restore the database to its correct state.

Furthermore, there are two copies of the state - one in the physical file system, as eventual reality of what is going on, and the other one, that may differ as a result of user file modifications, in the database.

On the top of it, several nodes may have totally different state of various files as a result of local file deletions, renaming and modifications.

As a result, you have a total inconsistency across all nodes and, in case of general public situations where you can not even contact the other nodes and tell them what to do to fix the situation so that their nodes fully update, what you have is unresolvable problem logically and you can not possibly assure the consistency.

From what I see, in current design, about the only way to resolve the consistency issue is to make it a MUST to use the .SyncIgnore file in case users do not wish to receive the updates on some files for whatever reason.

All other user modifications that are inconsistent with the r/w nodes should be undone by redownloading the user modified files if those modifications are not in the .SyncIgnore. That is the only way to assure consistency across all nodes as far as I can see, at least with non-journaling database system.

The "bottom line" is this:

The current design and update logic will never work, in principle, at least in general public situations where you don't have access and control of all the nodes and their data.

Defensive Open Source: Proposal to go Open Source

Posted on:
May 15, 2014 08:53 AM

If I could support a kickstarter type of campaign for opening the source, I'd virtually be throwing money after you, knowing that I'd be solving one of real problems of the internet still around.

Please please please! I love this product so much, but I can't ignore the closed source, it really matters, as you can imagine.

OK guys. This is beginning to get on my nerves.
BTSync is FULL of bugs and it is probably the most unreliable and inconsistent program I have ever seen.

And the problem with it is this:

I have provided quite a lot of proposals to BTSync with specific suggestions, analysis of the MOST critical and literally vital issues and all of it was ignored. In some cases, they even promised to take care of one of the nastiest design flaws related to timediff error, which is clearly a flaw in design in my opinion.

But ALL of it was simply ignored, and its been months now since these issues were raised.

And the sorriest thing is that we can not even fix their bugs because the source is not open and those bugs are most critical, at least to our applications and use. So, they simply MUST be fixed, or else...

Now, here's the kicker:

BTSync may be reverse engineered in no more than a month from preliminary estimate.

The bulk of the program is based on Torrent protocol, which is an open source by now. The rest of the program is not that well designed logic and some minor encoding, hashing and node update logic, which is simple enough as it looks right now.

So, what we have on the table is the proposal:

Proposal:

Either BTSync goes open source IMMEDIATELY or it will be reverse engineered by the most experienced and senior level developers in the Silicon Valley, which means in the world who does not know what it means.

That source code will be released to the public and if something is found in it that even remotely hints on their alleged relationship with the agencies, such as NSA, the Bittorent as a company will be disgraced and the whole world will know about it.

So...

The clock starts ticking NOW.

My humble suggestion: think well before you try to challenge this offer.

Edited version

Posted on:
May 15, 2014 08:53 AM

If I could support a kickstarter type of campaign for opening the source, I'd virtually be throwing money after you, knowing that I'd be solving one of real problems of the internet still around.

Please please please! I love this product so much, but I can't ignore the closed source, it really matters, as you can imagine.

OK guys. This is beginning to get on my nerves.
BTSync is FULL of bugs and it is probably the most unreliable and inconsistent program I have ever seen.

And the problem with it is this:

I have provided quite a lot of proposals to BTSync with specific suggestions, analysis of the MOST critical and literally vital issues and all of it was ignored. In some cases, they even promised to take care of one of the nastiest design flaws related to timediff error, which is clearly a flaw in design in my opinion.

But ALL of it was simply ignored, and its been months now since these issues were raised.

And the sorriest thing is that we can not even fix their bugs because the source is not open and those bugs are most critical, at least to our applications and use. So, they simply MUST be fixed, or else...

Edited by GreatMarko, on May 15, 2014 , 12:47 PM.
Post truncated due to inappropriate, agressive, and threatening tone in the remained of the post

Final message to BT

Posted on: May 16 2014, 08:22 PM

I don't know that threats are really going to help anything...

OK, I am willing to put my energy into it, for the last time and try to explain what that message meant in reality and considering facts, and not fiction and fabrications of all kinds.

First of all, it isn't a treat. There is a difference between a threat and reality. This is how it is and once top level developers commit themselves to making a compatible product, that's it. Nothing can stop them and it becomes just a matter of time. Here's just a few facts.

Fact: OSF (Open Source Foundation) has made a BTSync compatible product one of their TOP priorities and "hot" projects. That, by itself, means that some of the best developers in the Silicon Valley will certainly get involved, and I know some of them personally, including the inventor of the UDP tunnel, some creators of computer languages and so on.

Fact: To experienced developers decompiling the BTSync represents no problems whatsoever. In fact, just within the first day of analysis, all the "bowels" and key algorithms and some "esoteric" and even some quite shocking things were discovered.

Considering the program size of about 2 megs, decompiling and reconstructing the program should take no more than one month approx, and that is based on knowledge of specifics of the task.

Fact: It will be impossible to create a BTSync compatible product without decompiling because of some of the specific code, related, for example, to hashing, name mangling, some semi-encryption (the encryption itself is handled by the standard encryption related libraries), etc.

Fact: Sync approach on P2P basis is vitally needed, for example, in global information warfare, carried out by the "dark forces" in attempts to suppress the truth of what they have done on this planet.

So, as far as copyright type of issues and well-being of humanity and its future go, there is an "individual good" and there is a "common good" on behalf of all mankind. When "individual good" of any individual, party or even a country goes against the good in terms of mankind as such, the "individual good" can not possibly prevail, for this is the Law, Law of Life itself. It isn't an "esoteric theory" of "talking heads".

From the legal standpoint, even though am I not competent in that area, but did have to deal with one competent attorney working exactly on the expert systems project related to copyright issue, there are provisions in the law to accommodate for principles of the "common good". Furthermore, competent attorneys are of the opinion that there exists no copyright violation in cases where the amount of creative work "borrowed" is less than 30% of the total volume of creative work.

That directly translates into near certainty in legal terms that to "borrow" small sections of code, particularly of those convoluted hashing, mangling and semi-encoding, raise no legal issues.

Furthermore, there exists a possibility that BT itself is in violation of the copyright law in the Linux version of their product. Because they, nearly inevitably, used various tools, compilers, development environment, libraries that are subject to Open Source provisions of their licenses, some of which may require the source to the BTSync to be opened. Else, they are in violation of the licenses. Certainly, this issue has to be examined further by the competent attorneys.

Fact: If BT corporation does not release the source code, or at least the key issues and algorithms related to security and those convoluted things, then those developers that have committed themselves to create a compatible program will have no choice but to look at decompiled binaries, which is not illegal.

Fact: the result of their work WILL be released to public by the sheer fact of the Open Source principles, and this is not a threat, but reality.

Now, if BTSync does decide to release the source code or at least the vital parts needed for creation of a compatible product, it may be to THEIR benefit. Because, again, just from the first look at the code it has been discovered that there are some "unusual things" being done from the standpoint of security, privacy or tracking, to put in the mildest terms. But before the conclusion can be made, a detailed code analysis has to be performed.

This, in turn, means that if BT releases what is needed to create a compatible product, they may avoid that close scrutiny, which, in case there ARE some "hidden tricks" present indeed, this information will become public. And again, this is not a threat, but merely an inevitability of the outcome.

It is not likely that BT has any intentions to release neither the entire source code, nor even the relevant parts to creation of compatible products. Simply because they are likely to hope that they can "corner the market" in the "entertainment industry" which they use for the purpose of commercial gain by distributing various "packages" or "bundles" of indie artists, and so on. So, it is likely that they do not want anyone to be able to also provide the tools with which these "bundles" may be distributed by others. Probably because of the fact that all the nodes by default are known to BT trackers and it is likely that they get the percentage of of the "pie" based on counts of transfers or at least their estimates.

So, here you have in addition a monopolization of the market issue with all its consequences.

Now, when you put all these things together, and there are more issues, then what do you have as a result?

"THREATS"?

or

REALITY, that is about to bite the BT pretty hard if they maintain their present stance.

Because what we have right now in BTSync is pretty immature product with ALL sorts of problems and you depend on the "priority lists" of BT and what ANY of you think is a VITAL issue, may be not even on those lists, and even if they are, how can you possibly know when these issues will be resolved, if ever?

Furthermore, even to classify BTSync as a product in Beta stage is misleading and may have legal consequences, especially if combined with the issues of "cornering the market". Because in beta stage what you are dealing with is a product in near-ready to be released state. In beta stages, there is no longer any development done, nor there is any adding of "bells and whistles" to existing code base.

Not many people would dare to download and install a product in Alpha stage of development. But nearly everyone would be willing to download the betas, in the understanding that this is already a nearly-ready to be released product and, therefore, worth a try, which is NOT the case with BTSync, by ANY measures.

Therefore, it raises the legal issues of "bad intent" and luring the potential customers into a trap and thus gaining "unfair advantage", at least over those manufacturers and developers who are honest and do not advertise their products as nearly ready to be released, while in fact they can hardly be classified to be even on the level of Alpha stage.

As it stands right now, even if you COULD fix some of those critical issues yourself, or in a joint Open Source community effort, you simply can not. There is NOTHING you can do, but to cry and complain and bitch and moan, even while you see nearly half of the nodes on some shares either in the state of incompletion, which is a disaster, or stuck for days and not downloading the share, or showing the timediff errors and so on.

How do you like to see people LEAVING the share simply because they think the whole thing simply does not work? Well, in OUR case, it is not only a disaster, but a disaster on a global level.

"The bottom line" is this:

No "individual good", such as desires, greed, ambitions or attempts at "extraction of individual benefits" may go against the COMMON Good of Mankind as such.

For this is the Law of Intelligence itself, the law of Life itself, the law of Creativity itself, which can no longer be violated by "the dark side".

That game is over.

Now is the time for recovery, reconstruction, healing of nature abused by these forces and reestablishment of the Laws of Intelligence. The present corrupt system of control, domination, greed and "ones own interest" is being disassembled as we speak.

I hope this post clarifies the matters and gives the BT staff yet another chance to have a hard look at the issues. Otherwise, the "interests" of BT as individual entity will not and can not prevail, even in principle. This can not possibly be allowed under the circumstance. And this is not a "threat" but inevitability.

The original message of the party that writes this to you, which was interpreted by some as a "threat", was in fact an energy cluster and a message, that was meant "to ring some bells" in your consciousness.

Because the individual that writes this to you made certain commitments which he so far honored in full account and which he has no intention to dishonor as of this moment.

Even if the "dark forces" decide to KILL him, it won't change even iota. Because all the necessary measure have been already implemented and the processes that have started as a result are unstoppable, and for several reasons.

And this message is likely to be one of the last messages these forums will see. The work has been completed and the rest will proceed in "automatic" fashion.

Original:
Defensive Open Source http://forum.bittorrent.com/topic/21048-defensive-open-source/page-2#entry84076

"Munched by admin" a abusive

[Note: this is what was left of the previous message (above), within few minutes after its publication.]

Posted on: May 16, 2014, 08:22 PM

<munched by admin - abusive>

(Edited by Harold Feit, May 16, 2014, 08:40 PM.)

Original:
Defensive Open Source http://forum.bittorrent.com/topic/21048-defensive-open-source/page-2#entry84076

[Note: "ABUSIVE"? To whom or to what? Where is the "victim"? "Abusive" in what respect? Harold Feit also uses a seemingly interesting term: "munched", which clearly indicates this individual operates according to principles of the "dark side" or "destructive" side as it is known. Because it is the most violent way of expressing destructiveness.

Secondly, it indicates that he must be considering himself to be a representative of some kind of "superpower", "above" others, "undefeatable" and "omni-powerful".

And, interestingly enough, he classifies the original message as "abusive", even if his own most abusive act imaginable! Well, this is not only a plain ordinary illusion, but nothing less than DELUSION itself, a "case of psychiatry".]

"Empire strikes back": BT response to the "munched" post

Posted on May 16, 2004, 08:38 PM

Fact: you are now on moderator preview for 90 days.

Original:
Defensive Open Source http://forum.bittorrent.com/topic/21048-defensive-open-source/page-2#entry84077

Question arises: are these people REALLY insane, or just pretending?

Useful information and links

Protocols

DHT

Torrent files

Magnet links

Forums