Contents

BTSync

Contacts: Where to get help?

If you have any problems using the BTSync program, send the email to here: preciseinfo [@] mail [.] ru.

Get involved, instead of sitting on your butt

Get involved, instead of sitting on your butt, hoarding your creative energy and waiting for some "Messiah" to come and "save" you. If you only take, but never give anything, you will never get enough to satisfy you finally. Because in that case you are simply wasting your life in endless turns of the Wheel of Karma, as they call it, or are "spinning your wheels, getting nowhere really fast".

No matter who you are, you do have something to contribute. For this is the Law. The Law of Life itself! All you need to do it to look inside yourself and pay attention to that which "turns you on", something that attracts you, something that excites you and makes you want to be.

And then you simply give whatever "turns you on" without worrying of condemnation and denials of others, done, primarily, out of their of blindness and self-denial. You shalt see this one day. It is just a matter of time WHEN you do, at least if you are sincere and are honest enough to yourself.

If all those of you, who really understand and know something, simply add what they know to this manual and make sure to add the new chapter/topic to the TOC, can you imagine how much you help all sorts of people struggling "out there" with artificial problems that only exist because they simply do not understand how it all works? Well, all that wasted energy can be redirected to CREATIVE things, instead of wasting it, struggling with the paper dragons.

So, let this manual become truly alive with every drop you contribute to it, and, with it, YOU will. For it is the Law, the law of Life itself as expressed by the Infinite Multidimensional All-Pervading and Ever-Unfolding Intelligence.

The purpose of this manual

The purpose of this manual is not to merely duplicate the information available elsewhere, including the "official" BTSync manuals. But to provide the comprehensive and most detailed coverage of some of the tricky issues and describe as many different aspects of BTSync behavior and its effects as possible.

This means that not every single conceivable issue is covered here, even though we attempt to cover as much as possible.

The second point is to hopefully give an example to other software manufacturers or developers of how do you write documentation which covers all the minute details of your products in such a way that even users with the most basic understanding of computers could use your products without going to all sorts of forums and asking about those things that should have been explained in the first place.

Many, if not most, software manufacturers and developers in most cases provide only scratchy documentation that leaves way too many questions in the minds of ordinary users. Instead, they keep placing almost all of their efforts on developing all sorts of bells and whistles and "features" of all kinds, as a result of their never ending market department driven attempts to grab a bigger and bigger "market share", and BT is about as perfect example of it, as it gets.

So, instead of covering the most important issues in their documentation and fixing the critical bugs, they dedicate most of their efforts on adding more and more "bells and whistles" to their product while urgently needing resolution issues remain unattended. Their basic product is not fully described in clear and concise and detailed way.

If you look at their blog, you might get amazed at the amount of hype they keep producing in their massive marketing efforts.

http://blog.bittorrent.com/

And if you subscribe to their RSS feed, you'll be getting the announcements on all sorts of "bundles" packaging all sorts of zombie "entertainment" nearly every other day.

http://blog.bittorrent.com/rss/

But if you look at their BTSync forums, you'll be amazed, if not "blown out of your chair" outright seeing the amount of problems users experience, and plenty of those problems are of CRITICAL nature, the issues that have to be resolved ASAP. But many, if not most of them, simply remain unattended as you can see by the age of some of those threads.

The result of this is the products that work in most cases, but in some cases of "this should never happen" kind they either do not behave correctly or simply do not work - period.

For example, we have observed a countless cases where new users join the share but never actually start downloading it. So, they sit there for a while and then leave, never to be seen again. At the moment, we are seeing over half of the nodes on one of our main shares not being in the state of 100% completion, and they stay that way permanently, never completing, at least as it looks to other nodes. We have also seen cases where some nodes begin to download, but then stop for some reason and never resume the download and never completing the initial download. And we have seen the cases where they do manage to complete downloads if you restart BTSync on a master share. And we have seen plenty of cases where nodes permanently remain in a locked state as a result of time difference issue between the nodes, which is also one of the most critical issues that had to be resolved immediately, but remains unresolved for nearly a year now.

Now, every single issue above is of CRITICAL level and has to be resolved immediately. Because the program simply does not work or works in fundamentally incorrect or totally unexpected ways.

The users install the program, try to join some share and sit there for few minutes waiting for something to happen. But, since nothing happens, they simply leave the share and probably simply deinstall the BTSync thinking that "this stuff does not work". And what could be MORE critical of an issue than not being able to perform even the initial download? How is this even possible technically?

They do not seem to understand that about the MOST important criteria in any product is not necessarily the quantity of bells and whistles, but the robust and predictable behavior and the ability to recover from any situation, conceivable or "impossible".

And the second most important property of any product is precise and detailed documentation that could be understood by "mere mortals", and not just some one-liners of a smart kind that could be understood only by professionals.

Documentation should be as easy to read as you read a newspaper. You should not be sitting there, scratching your head and wasting your mental and creative energy on trying to think "what do they mean by that?".

So, by reading a good documentation you effectively become and expert as far as product use goes. Simple as that.

And, finally, if not most importantly, the overwhelming volume of problem posts on their forums, for the most part, is a result that users simply do not know or do not understand how things should work under normal circumstances, simply because it is not documented in the manuals. When we were introduced to the BTSync, we could not quite believe how scarce and sketchy is their documentation and how little of some of the most critical issues of the program behavior were described.

Forums are not the place to provide the documentation. Else, you see the large amount of traffic on them that should not be there if it all described somewhere. As a result, their customer support on the forums is simply overwhelmed with amount of questions and issues.

The end result of poor documentation is wasting a lot of energy that could be used for creative purposes and not merely wasting one's time struggling with paper dragons and purely imaginable issues that shouldn't be there to begin with.

Building the sophisticated information distribution systems

You can build the sophisticated information distribution systems of a synthetic kind where one part of your system exhibits the private network behavior and is not visible to others, who are not the part of your private network, and the other part of your system exhibits the purely public aspect and can be accessed by anyone as long as they have a key to some share/top folder/collection.

Here is one example related to "hot" and "not so hot" information distribution via web sites:

In this example we will use the combination of BTSync and Syncthing (Pulse) working together.

Assume you have a web site on a server somewhere and you have a working system, possibly at home. You might be doing all sorts of updates and edits on all sorts of information in all sorts of subdirectories of your site which you would like to sync to the site as soon as you push the save button in your editor. This way, you do not have to worry about remembering which exact subdirectory of your site the information belongs to and you don't have to waste time on using the FTP client to update the server, which requires a number of steps before you can actually transfer the files.

So, you can use the BTSync via privately known key to update the site automatically and in "hot mode", meaning that as soon as you push the save button in your editor, the information gets shipped to the server within seconds, and without any further action on your part beyond pushing the save button in your editor. The rest of the process is fully automatic.

But then, you would also like to make this site available on the sync basis to the general public so that they could receive the "latest and greatest" updates nearly instantly. Except there is one "but". In case you are working with some document in active mode, which might imply a lot of updates and save cycles, possibly every few seconds, you would not necessarily want to update the sync clients as often as you update your site.

So, what you do is to create another folder on your disk which contains the same exact copy of the site, but in "not so hot" mode, so that you could decide at some point: ok, we are kind of done with this cycle of updating this document and now it is ready to be synced by the whole world.

To do that, you establish another connection to that 2nd folder, but with using the Syncthing this time. What it does is to feed your 2nd folder from the "hot" site back to your home box.

Finally, you feed that 2nd copy of your site to the whole world via BTSync, only on a different share/folder. So, as soon as your site on the web server is updated, it feeds the info back to the 2nd folder and is served to the whole world via BTSync.

So, to make that 2nd folder to be not so "hot", all you need to do is to stop the Syncthing that feeds back the information from the site to your 2nd folder. Then, once you feel this edit session is complete enough, you enable back that Syncthing channel and it will bring back the site to the 2nd folder that is serving the world now.

One other benefit you receive with this approach is that you also assure a certain guarantee that your original information does not get damaged because of the actions of other users from the general public that might do some crazy things with their copy, which, in turn, might update all the other nodes with wrong or bad information. This is quite possible with BTSync and we have seen it many times in different situations, either because of program misbehaving or because some users do things not really understanding what they are doing. And we have also seen plenty of cases when some users intentionally try to attack the share. In this case, under no circumstances would you like your original information to get damaged. That is why you establish a private channel only between your working system and your server. In this case, you know 100% that you do on that link only those things that will not cause the damage to the original information.

So, the private side of your system can be controlled by you in a predictable and reliable manner, because you control and have access to both ends of that link, but the 2nd folder, which is connected to the general public, might and is likely to get damaged, sooner or later, because of all sorts of reasons. Moreover, some degree of damage may be observed on more or less periodic basis. We have seen some shares getting damaged every few days, mostly because not that many people really understand what they are trying to do and the consequences of it.

But you can recover from that damage since you also control the 2nd folder and the original state of the data can be restored within seconds simply by copying the original data over the damaged copy. This can be done with a single command. You just need to remember to copy the entire directory tree preserving the file modification times.

So, what you have effectively, is a synthetic system that exhibits the behavior of totally private connections, but, at the same time, any part of your information may become public any time you want and all of it will be updated automatically without any need for you to do any additional things in order for the entire system to function as a whole, and in any mode of operation you might think of, "hot", "not so hot", or even in "slow motion".

Interactive information systems with nearly instant delivery

What the above mentioned synthetic information system also means is that you can get really creative and can concentrate primarily on the creative aspects of working with information, instead of being bogged down by purely technical aspects of information management and the routine work. This way, your creative energy could be used at its peak, without worry about all sorts of routine technical things and tasks.

Once the system is set up, you no longer have to worry about many things that assure the whole thing works. Everything, but the creative aspect itself, becomes fully automatic and nearly instant delivery and distribution of information becomes guaranteed and fully automated.

In other words, now has come the time, finally, "to get down with it" and investigate and expose the REAL issues facing this planet, as you no longer have any excuses of not "doing a good job" because of some bizarre or overwhelming technical aspects and issues.

What else can we say?

What does sync have to do with New World Order?

Note: the following chapter(s) were originally written in the context of conversion of Syncthing to Pulse and, shortly after that, change of copyright license, which has the profound effects on the entire project. So, the implied reference to be kept in mind is the Syncthing/Pulse project.

Well, in a single word: EVERYTHING! Believe it or not.

In relationship to Amazon.com showing up in the whois request, whatever that means or implies, one can mention this kind of thing:

If, by any chance, Amazon.com is in fact involved in any of this, then the question arises: But who stands BEHIND the Amazon curtain?

For example, it is alleged that behind the BTSync stands no one less than the NSA, and that is pure grade evil and global, all pervasive surveillance and identification of the "undesirables" to be placed on the so called "red list" of those individuals that are considered "dangerous" in the view of the powers of evil and this NWO thing.

Interestingly enough, even the veterans returning home from the war zones are considered to be "the most dangerous" kind, if you can imagine such a degree of evil. Those that are promoted as "heroes" and "fighters for freedom and democracy", in reality, are in fact considered to be the "most dangerous" of all. And those soldiers do not even realize any of this until they return back home and cold and brutal reality hits them between their eyes.

Most people childishly believe that there simply must be some limits to evil and it can not go against the most basic things. Well, sooner or later, they'll have a chance to realize that the evil has no limits whatsoever, ethical, moral or otherwise". That is its very nature. It does not even have a concept of some bounds or limits, just as shown, for example, in a book called "Atlas Shrugged" by Ayn Rand, who was one of the mistresses of Rothschild and this book, according to John Todd, a former member of "the council of 13", was created as per request of Rothschild as indoctrination book for the Illuminati. Kind of mind programming book in a psy-ops kind of scenario.

John Todd's introduction to Atlas Shrugged
(about the famous Illuminati mind conditioning book by Ayn Rand)
https://antimatrix.org/Convert/Books/Ayn_Rand/Ayn_Rand_Atlas_Shrugged.html#John_Todd_introduction_to_Atlas_Shrugged

And that opens the whole Pandora box that leads to this thing called the NWO. Actually, anything one deals with in the modern world, eventually and inevitably leads to the same thing, and that is takeover of this planet via "The Plan" of planetary takeover, which has been nearly completed in full at this junction.

And so let us diverge a little bit from the "main subject", and attempt to cover just a few basic things about that scheme, because, first of all, these are some of the most vitally important things that are happening on this planet, by far. Anything else simply pales in comparison, in terms of significance or the impact, at least from the standpoint of the global information war raging on as we speak, and if one underestimates the consequences of that, then, as they say "you are as good as dead".

And about the sorriest thing is that not that many people even realize what is REALLY going on. They just buried their heads in the sand and "play dumb", like it is "not my problem". So...

And the idea behind the so called "red" and "yellow" lists is that all those that end up on the "red list" are to be exterminated first, or thrown into the concentration camps according to the NWO plans. In the USA alone, at this junction, there exist over 100 concentration camps, with gassing chambers, efficient furnaces to burn the bodies and the rest of it. The "red list" as of couple of years ago contained about a MILLION people, if you can comprehend the grandness of the scale of this operation code named "Project ENDGAME". Barak Obama has issued a secret order to activate that operation back in 2009. This information is publicly available, from Alex Jones, for example.

"The bottom line" is that one can not forget that we live in the world of evil most profound, and all the so called governments throughout the world and even the international organizations, such as UN, EU and so on, are merely the puppet theaters, stuffed with puppets that sold their souls for the benefits of stuffing their own bellies at the expense of all others. For example, Obama bluntly, and even jokingly states: "there is a theatrical aspect to politics" and governing. One can only add that it is all MOSTLY theatrical. The reality aspects of it are minuscule.

Look for FEMA concentration camps. Btw, hundreds of thousands of caskets, made out of special plastic and capable of storing at least 3 bodies in each one are stored all over the USA. Over 30 THOUSAND guillotines are stored in two states in the US alone. With that many guillotines you can "process" people at the rate of chickens on a chicken factory. So, we are talking about MASS murders of genocide scale, unseen in the entire history, done in the name of "protecting freedom and democracy" and "fighting terrorism". Can you imagine millions of "terrorists" or "extremists" in the USA alone?

So, the same exact people that were called the "dissidents" before, are now classified as "the enemies of people", the "extremists" and even the "terrorists", all of which is pure grade lies and fabrications, just as was explained by Nicholas Rockefeller to Aaron Russo in their friendly meetings.

(Note: you can find the interview with Aaaron Russo by doing a search on "Aaron Russo: Reflections and Warnings - Full Transcript". In this profoundly shocking interview he talks about his friendly meetings with Nicholas Rockefeller, who, for example, explained to him the scam of 911 and how it was meant to create a totalitarian state in the USA. He also talks about the mechanisms used by "the banking mafia" to rob the world and create mountains of fake "money", not backed by anything other than thin air.)

This is not just a bad joke, by ANY means. Except you need to study these subjects to even BEGIN to comprehend what might be taking place with this seemingly strange "conversion" of Syncthing into Pulse brand and all the alleged "support" by the corporate "sponsors".

The problem for the NWO "elite" is true information that discloses their evils and exposes their scheme of world takeover. That is why it is being suppressed on a global scale and the so called governments, who, in reality, are nothing more than mere puppets of the NWO puppet masters, have already passed all sorts of "laws" to suppress the true information by classifying it as either outright "terrorism" or "extremism". Interestingly enough, most of those "laws" and "executive orders" go against The Universal Declaration of Human Rights in the most direct and blatant ways.

We are in the global information war right now and of the magnitude never seen by the mankind in its entire history.

And that is precisely where sync technology comes into this picture.

Sync on P2P basis is the most urgent need in today's Information Technology

It is very unfortunate, and even sad, that even the "advanced crowd", the developers, do not seem to be "getting it". They seem to look entirely oblivious of the facts and events in the world that have the most profound significance and the impact on the present and future of the entire planet. They watch the puppet theater on TV and think that this is something that represents or corresponds to reality. Because via mass media their minds have been zombified to believe so. And, since their minds have been programmed with the ideas of "I don't care, this is not MY problem", they believe that "their world" extends to the surface of their skin, nothing more than that.

It seems that "the smartest of all", at least as they think of themselves, the developers, are either "playing dumb" and pretend that the issues of public distribution of information on P2P basis do not have any merits and "use cases", or have simply burred their heads in the sand, pretending that it is all nothing more than "none of my business", "let it all go to hell for as long as I can save my own skin and stuff my own belly" kind of thing.

It is like the only things that everyone is interested in is to sync their laptop to their camera, their server and a chip in their shoe, if not in their brain. That is ALL they seem to "care about".

Yes, all this sync stuff is good and all that, and it does serve some local, individual purpose. So, now you can push a single button and you get all your devices automatically updated with the latest collection of the zombie images, sound and video files and you name it. Push one more button, and it all gets delivered directly to your ears and eyes, so that you could get zombified faster with all sorts of images of the most horrendous violence, wars, killer "heroes", conmen of all kinds and the rest of the zombification menu of the NWO grand "plan", of which very few even know about, or even care or suspect that such a thing actually exists, and even if it does, "hey, what does it have to do with me and 'my interests'? It is not MY 'problem'"!. Then WHOSE "problem" is it? Do ANY of you know, by any humble chance?

And so, it is not entirely incomprehensible that the "developer-in-chief" even gets vicious in his utter denials of the very need for public information distribution issues, which is probably the most significant development this planet is going through at this junction, as far as purely technical issues are concerned.

Because how can you defeat the evil if you have no clue of what it is, how does it work, what are its tools and methods, who stands behind what, what are the "plants" and what kinds of schemes have been developed and implemented for centuries?

There is an ongoing massive attack on the Internet going on as we speak, and all the corrupt puppet governments and their agencies issue orders to block more and more sites, and even worldwide, without any court orders or proper legal proceedings in gross violation of the The Universal Declaration of Human Rights, articles 10, 11, 18, 19, and 30.

http://www.un.org/en/documents/udhr/

Because the powers of evil that have already taken over this planet for the most part and control over 80-90% of the world's gold, precious metals, diamonds, energy supplies, banking system, media and publishing, and have taken over the money printing business for all the major world currencies with the help of the FRS scam, know very well, that their demise and inevitable defeat begins with true information, disclosing the nature and the mechanisms of evil and principles of world takeover.

They were able to suppress and destroy all sorts of most vital information about the nature and mechanisms of evil, and instead, via the zombification machine called mass media, were and are feeding everyone lies, perversions, deceit and the ideology of violence and destruction, and have programmed the human minds to be interested only in the most idiotic, perverted, corrupt and violent things, promoting greed as some kind of "objective necessity" and a basic mechanism of "survival", and so they were able to do to this planet anything they wish in pursuit of their final goal of taking over the world via the scheme known as the NWO.

But, even despite their most vicious methods of destruction of information and the authors and publishers, all of a sudden, it all floated back up to the surface, seemingly out of nowhere. And that is their most deadly problem, because it has spread out the waves of energy of awareness throughout the planet and the people throughout the world have begun to revolt against the entire satanic doctrine of Luciferianism, that has been ruling this planet for thousands of years.

But to stay well informed about methods and history of evil, you now need the tools and methods that allow you to switch to a different mode of operation. Instead of attempting to access the sites and collections of information directly, you need to be able to access it no matter what, on P2P basis. There is simply no other way to prevent and stand against all the tools and methods of blocking the information of all kinds even bypassing the law itself, and unless millions upon millions of people know exactly what they are dealing with, there is no way they will be able to defeat the most powerful and sophisticated system of evil that has been developed and perfected for thousands of years.

Before you can stop the evil, or cut off the parasitic entities that suck all your resources of all kinds like a black hole, and restore the environment of this planet and human dignity and real fairness in all sorts of things and activities, you need to KNOW how that system of evil works. You need to study it all, which used to take years until now, but now can be done in months, if not days and some things - in minutes.

Or do you expect that some "noble knight on a white horse" will come and "save" your sorry rear end, while you are sitting on it, doing nothing and pretending you are merely dumb as a piece of wood, not to insult the piece of wood?

WHO is going to come and "save" you if you are not willing even to move a finger to save your sorry ass which is on fire, regardless of whether you realize it or not?

For how long you, the false "kings", are going to deny the utmost need and urgency of implementing the global public sharing of all sorts of information via sync technology, which is the only technology that allows you to stay current even with the immense amounts of information via single collection and a single key?

The thing is that even torrents can not do that. Because torrents are static. Once created, that's it. They can not be updated, extended, modified and so on. How many times do you need to be told all this before it clicks in your "sophisticated" minds?

Look, you "smart guys", even a FIVE year old child can comprehend it. Why can't you? Are your brains so profoundly damaged and zombified to the point that you can not put two and two together? Are you going to ask "but what are the use cases for that"? Try to "prove" that you need to even bother about two plus two. "Who NEEDS it"? What are the "use cases" for that?

Well, what can we say but "it depends on how dumb you are", or pretend to be! It is not inconceivable that by now even Life as such is nothing compared to all the images of death they feed you non stop, any place you look.

Yes, the situation on this planet has never been as pathetic and as hopeless as it is right now, and that is precisely why you have a chance to correct it, once and for all. You need to come to a state of utter hopelessness and desperation before most of you even move a finger, and even then, only for the exclusive purpose to "save their own ass", pathetic as it is.

This is all you get from us in this transmission. Either you wake up, or be prepared to face the consequences of your denials, utter insensitivity towards each other, and even Life itself, and utter absence of care within you. Simply because your minds have been totally zombified with the ideas and images of death, destruction, violence, parasitism, cunningness, corruption and the rest of it.

It is up to YOU to decide which world you wish to live in. No one can do it for you. Because it is against the Laws of Free Will and Free Choice to interfere in your affairs, whatever they might be, unless it has all come to its final, logical fruition, at which point, it can no longer be allowed and the forces beyond visible simply have to get involved directly and in the most tangible ways conceivable.

Comparison of file synchronization software

Here's the link to Wikipedia article about different sync packages.

Comparison of file synchronization software
https://en.wikipedia.org/wiki/Comparison_of_file_synchronization_software

Open source sync alternatives

Introduction to open source sync alternatives

This is just a stub for now. This section will begin to accumulate the information about the open source alternatives to BTSync.

It is our opinion that BTSync, at least the way it is right now, has no future, and for quite a few reasons:

1. The last updates from 1.3.109 to 1.4.72 and higher include the major GUI change and the position of the management is "this is the way it is going to be from now on, regardless of anything anyone says or thinks". And that means a catastrophe. The "latest and greatest" GUI is web browser based and it is a huge step back from the previous design, which provided a reasonably detailed view on dynamics of nodes update.
   In particular, what used to be the Transfers tab was very useful to see which file is sent/received from which node. You could see the simultaneous transfers to/from different nodes and that gave you a sufficiently good idea of what is going on. That tab is gone now and you can not see the actual transfers taking place.
   The Devices tab in old design would show one all the shares/folders and all the nodes participating in those shares. So, in a single screen you could have a pretty good overall idea of what is going on. That screen is gone now and you have to click on things just to open up another screen or semi-dialog, where you still would not see what you could in previous design. And so you have to click more and more times to see anything useful, and even there, its usefulness is questionable and information provided is clearly insufficient.
   This fact alone is a killer. If that is the way it is going to stay, then BTSync is dead. Quite a few people have decided to go back to previous version.
2. It is a closed source software which implies that you do not really know what happens "under the hood" and you can not trust a word about claims of encryption of traffic or increased privacy, first of all because no claims of encryption or increased security or privacy can be made in a closed source code, pretty much by definition. Any security specialist knows that as good as 2+2=4.
3. The incredible number of bugs in BTSync and its generally poor stability and consistency of behavior, combined with the fact that it is a closed source code, makes it impossible for other developers to fix piles upon piles of bugs, problems and the issues.
4. BT strategy on BTSync seems to be a desire to corner the sync market via distribution of various "bundles" of zombie/biorobot level of entertainment. As a result of this, and, possibly because BT may have some contract arrangements to get "the piece of the pie" of total volume of traffic in the "entertainment industry", it is only "natural" for them to hard-wire the mechanisms of tracking the traffic of any node running the BTsync. Therefore, whenever you run BTSync you leave your footprint at BT.
5. There are reasons to believe, or at least suspect, the cooperation of BT with NSA and other agencies engaged in global surveillance and you can find some references about it in this manual.
6. BT is primarily interested in creating as much hype as possible, instead of fixing bugs in the software to make it stable, reliable and have a predictable and consistent behavior. Instead, they keep developing all sorts of new bells and whistles to entrap more and more unsuspecting customers or clients in order to control as much of global sync traffic as possible. They regularly brag about the mind boggling statistics of global volume of sync traffic, and to have that ability, they'd have to hard-wire the mechanisms of informing BT about any and all sync traffic that is tracked by BT servers, trackers and relays.
   In fact, you can not even turn off the ability of being tracked even if you disable their trackers and relays. Because the program simply downloads the list of servers automatically, without informing the user. Even if you disable their trackers and relays, still, for some magical reason, you can discover the other nodes, even if you disable the DHT.
7. BT keeps bringing up new and new features and "improvements" in a product which they themselves classify as Beta, which implies that the product is not only becoming more stable, but just the other way around, less and less stable. In beta stage, no development or new features are allowed. Otherwise you will never get to the state of the final version. Because bringing in new "features" will bring in new bugs. But they not only keep bringing new "features", but even do some major and quire radical redesign of the most basic mechanisms, which is an absolute "no, no" in genuinely Beta phase of development.
8. BT is not willing to disclose the necessary information to create the compatible products, which, by itself, is an evidence of their desire to monopolize the sync market.
9. The project management, supervision and technical competence of their developers are of such a low grade that it isn't even laughable. It is simply pathetic. For experienced developers, some of the design errors and incompetence in technical sense are simply obvious, and would have never be allowed if level of technical competence would be on the competent, professional level.
   Just looking at one obvious example makes things pretty obvious. As it stands, BTSyncs begins to "forget" the nodes it knew about on a particular share if you keep the program running for several hours. But this should never happen, simply because once the node discovers any other nodes there is no logical way for it to "forget" them. How could you possibly forget something you knew? This is evidenced by the fact that if you restart BTSync it will again find those nodes that it "forgot" about before you have restarted.
10. There are other numerous examples of poor or even bad design in GUI, and we have mentioned a few of examples of it on BTSync forums months ago, but no improvements can be seen even today. In other words, they did not even move a finger or even looked at any of it, like it is something utterly insignificant. But to the users it would provide a much better and more complete and detailed picture on the whole thing and some of the ideas we proposed would even help to identify numerous bugs much quicker then it is possible now.

We could continue this list for quite a while, but what's the point?

As a result of this, it is pretty clear that one can not expect BTSync to become a stable and reliable product, at least in the foreseeable future.

That is why we began to accumulate the information on developing the open source alternative, and it will start right in this manual, which is no longer dedicated exclusively to BTSync, but, rather, to the sync technology as such.

Highly desired features

This is just a stub for now. This section will begin to fill with those features of sync programs that distinguish it from the "stone age" of syncing.

First of all, it is not a bad idea to keep in mind that syncing, as such, is still quite an immature technology. Furthermore, it is a highly complex from the standpoint of all sorts of logical cases or consequences of all sorts of conditions and events, and there are so many of those, that we can not quite enumerate them all.

Some of the most critical parameters of the sync process are as follows:

• Stability, consistency of data across all nodes and predictability of behavior.
• Detailed and precise reporting of various operations and their results, be it warnings, errors or success results.
• Being able to see the entire process in as precise and as detailed way as possible.
• Precise and detailed view on dynamics of the process, especially during the active periods when a number of nodes begin to upload or download the entire folders/shares/collections.
• The ability to control the process, down to the level of individual nodes. This includes putting some share on hold or putting the individual node on hold, possibly only on one particular share. This includes the extreme cases of suspending or even banning the nodes for various reasons, such as when you see some people that are essentially the parasites, getting as much as they can and then splitting and not giving anything to others.
• The ability to customize the user interface, such as adding, removing, rearranging some columns in various dialog boxes, resizing the GUI elements, setting fonts and colors and all sorts of other visual or convenience things.
• Detailed and precise and complete error and warning reporting.

Additional problems of an open source implementation

This is one of the aspects of the open source that could be easily overlooked, which might cause the most drastic problems. It is applicable to the situations where the information exchange does not necessarily happen within some trusted group of people or between different devices belonging to one individual. And this is particularly applicable to any public information exchanges, regardless of anything else.

The thing is that in open source alternative you have to be extra attentive as to how you design your algorithms and various data structures passed between the nodes and how do you identify various things with confidence that they are not faked.

For example, in BEP (Block Exchange Protocol) definition there is a concept of a local database, in essence, which is sent out to all other nodes upon request. Now, in r/o nodes, there might be some files that are not present in the original master node (r/w), and so, various decisions might be made by other nodes as to ways in which that additional information may affect the other nodes. Because that information may look like a new version of some existing file, or a new file that has been stamped as being present on the master node, while it was merely faked.

In that case, some attacker, who does not even have a key to be the master node, may effectively cause an update even of other master nodes, just because, for example, he might stamp some files with some flags that indicate that they originally came from some master node, which is not true. In that case, one master node may decide that some other master node have added a new file, which may cause a download from the r/o node of any garbage files created for exclusive purpose to cause damages to the share/folder/collection.

Therefore, an extra rigid update logic needs to be implemented in order to assure the validity of any files and data structures floating around, and and the overall consistency of data across all the nodes in a share.

Generally, no r/o node should be able to update any r/w nodes under ANY circumstances whatsoever, even if some file looks like it was obtained from some master node by faking some bits in the data structures passed around.

Secondly, no file may cause an update of other nodes unless there is a guarantee that it definitely originates in some master node. That means that the file hash on the originating master node has to be generated in such a way that faking the originating source becomes impossible, because the originating flag (master version) is encoded in the very hash and could be checked after its decoding and, possibly, comparing the hashes of the file pieces or even the entire physical file against the hashes.

Basically, no information, bits or statuses of nearly every single data structure passed around may be trusted unless they are encoded with the necessary identification stamps.

The only data that can be trusted is the data that is guaranteed to come from the master nodes, or guaranteed exact copy of it.

This all means that even if someone modifies the sources for the program and recompiles it and then tries to use his version as an attacking weapon, he should not be able to do any harm or damage to the data, nor to inject, delete or modify the originals. At the most, he might cause some extran network traffic, which is also undesirable as this could be used as a permutation of some bandwidth attack that might choke the nodes with utterly fake data.

Guaranteeing the master version

In order to fulfill the requirement of data consistency across all the nodes, about the only realistic alternative is to make sure that the only files that are propagated around, including from the r/o nodes, are the files that are GUARANTEED to be the exact copies of the master version. And that is easier said than done in an open source setting. Because the attacker may modify his version of the program and set any elements of the data structures and any flags in any way he wants.

This means that no transfers of any files should be possible unless there exists a guarantee that either at least one master node is on line, so things could be checked against it if needed, or there exists a copy of the master node database, which has all the necessary information about files, hashes and so on. That means that every r/o node first of all obtains a master database, which may even come from some other r/o node if there is a guarantee that it was not faked, which is possible. The master database needs to be stored permanently on each node, so that it survives across the program restarts.

One thing we know about the master nodes that make them different from the r/o nodes is the KEY. No r/o node knows about that key. Otherwise, it can become a master node any time it wants.

This provides us the hook via which we can assure that the things really originated at the master node. So, when we hash or encrypt things on the master node we can use the master key similar to security certificates, private and public. The mechanisms are pretty much the same.

We may not even have the latest version of the master database on our node because no master nodes are currently on line. But that is not a problem. We can still operate and transfer files even between the r/o nodes that might even update the other r/o nodes with new files, because those files are stamped in unforgeable way to prove that they came from the master node.

But when at least one master node comes back on line, we can update our r/o node databases with the latest and greatest version and possibly update some files that are newer or fresher than those we currently have.

With this strategy we can be sure that even if some attacker uses a custom version of the program, modified for the purpose to attack, it is not going to work. Because he will not be able to encrypt the things the same way as the master nodes do, simply because it does not have the master key.

Synthetic synchronized torrent approach

One of pretty interesting and not quite apparent approaches might be simply using the existing open source torrent code and extending it with sync functionality.

For example, if we take some portable torrent client, written, for example, in Java, such as Vuze/Azureus, then we already have the bulk of the work done, including the DHT discovery, UTP protocol implementation and so on.

From there we have several alternatives:

• Use the existing torrent mechanisms and construct the temporary torrents that represent a set of files to be shipped to some other node.
  Basically, to synchronize with some node via procedure of gradual and incremental discovery of changed files, a node, trying to sync with some other node, first of all requests the top folder hash. If that hash is the same as yours, then you are in sync with it. Otherwise, you request the list of hashes for the files in the top folder, and if some of them differ from your own, you need to request those files from that node in case their version is newer or you are missing some files.
  Some of those files may be subfolders. In that case, you descend deeper and request the list of files in those subfolders in the same way.
  Once you have gone through all the files on that node, you send a request to deliver those files to you either via single torrent or in batches, as you keep traversing the top folder and discovering more files that need updating. This is just a matter of efficiency.
• Construct a single temporary torrent for the entire collection on some node, which represents the state of that particular node, and ship it as a whole, regardless of whether you are in sync or not. That torrent should contain the necessary information about the entire tree of files/folders, such as file paths, individual hashes and file update time stamps in the universal UTC time.
  Basically, what you are doing is sending the entire state of your node to others. They can then compare your state with theirs and, possibly, request some files from you, whether they represent the newer version or new files. You can do the same thing, only the other way around, requesting their state of the node.
  This might actually be not as much of an overhead, as it might look, because, first of all, most of the nodes on some share will be in sync with other nodes and so, there is no further activity required. But the overhead would probably be there for the nearly synchronized nodes because you are sending the info for the entire node while in most cases you are out of sync with them in the amount of a few files only. But this needs further analysis.
  One of the advantages here is that you construct this torrent ONLY when there are updates on your node. Once constructed, you can keep it around until your node gets updated for whatever reason. So, at least for the master side and for most nearly synchronized nodes you save some time by constructing such a torrent only once. But you sacrifice some time because you need to send out more data then necessary.
  It might turn out that your overall overhead is not as significant as it might look on paper. Because shipping even a single torrent for the entire collection is probably comparable to shipping a single file from that collection. And, if we are talking about the audio, video, PDF or picture collections, then a descriptor torrent is very likely to be comparable to the average file size, if not significantly smaller than that.
  But the benefit is that you only do a single transfer which contains the complete state of your node. So, it looks like a statistical issue and it will highly depend on the collection type.
• When some files are changed/updated/added/deleted on some node, you send the "updated" information notification to all other nodes. After that, they send you back the requests to deliver those files to them. It could be a number of files simultaneously, if, for example, you have added the entire subdirectory to the share.

Basically, there are variations on this theme and things need further consideration and a more detailed analysis. But what we achieve with this approach is quite a bit. First of all, we would already start with a fully functioning torrent program, and, if it is portable across several platforms, we can immediately start working on implementing the sync functionality in small incremental steps instead of starting the whole thing from scratch.

Single modifiable torrent approach

Let us consider the single modifiable torrent approach to syncing. The goals of this approach is to minimize the amount of development effort to get the initial version going, while, at the same time, assuring that all the nodes are updated to the "latest and greatest" version.

Furthermore, it has to provide the guarantee of data consistency across all the nodes, meaning that all the nodes contain the correct and "latest" version of all the files. (Note: latest is in quotes because the very notion of file time stamp is quite illusory, at least in cases of a deliberate "time stamp" attack on some share.)

Also, it would be interesting to investigate the possibility of uniting the ordinary torrent functionality while extending the torrent client program to also do the sync operations. There is quite a bit of common information GUI-wise between the regular torrent operations and syncing, which makes is worth of investigating it further.

It might turn out that the easiest way to implement such an architecture would be to stay as close to the original torrent transfer concept as possible, and that is - to use the torrents for the entire collection rather than doing it in a more fine grained manner by constantly constructing smaller torrents for only a part of a collection. But there might be potential performance degradation in case of using the single torrent for the entire collection versus a number of smaller torrents that only contain the modified or new files.

But single torrent approach might exhibit the additional difficulties of synchronizing the updates in case there is pending transfer of some torrent while some file was updated in the middle of transfer operation. But this is one of those more "esoteric" aspects which, by the way, would have to be dealt with even if we handle transfers by construction a number of smaller torrents representing the deltas only.

This approach assumes that all the top level communication exchanges begin with top folder hash. Top folder hash is the combined hash of all the files in a share/top folder/collection. Any further action is required if those hashes differ on different nodes. Else, the sync state has been achieved and there is nothing to do beyond periodic scanning to recalculate all the hashes and, ultimately, the top folder hash, which is the main or "key" hash for the share.

About hashing.

The idea is that you gradually and incrementally reach the state where all the nodes on a share contain the same collection, represented by the same torrent file. That is the stable and final state of the share and it implies that all the nodes are in sync.

But there might be a difficulty here because there is a contradiction. On one hand, we are trying in effect to create a single torrent that represents the master collection. That would imply that all the nodes would eventually end up with the same torrent file when they are all fully synchronized.

But, since any node may ignore some files from the master collection, that would mean that their version of the torrent might be, and is likely to be, different than on a master node. But there might be a solution to that, which allows for a SUBSET of the master node files to be distributed. Because it still valid, correct and current with the exception that it is not complete.

But it is just one node's view. When you contact the other nodes, you are likely to see a slightly different version of the same collection, except THEIR ignores are likely to be different. But it does not matter. What matters is that they might have some files that are missing in the first node, and that is what matters.

Because you enlarge the view on the complete collection by using the union operation on all the data on all the nodes on line. The more nodes, the more chance that your own version will be as close to the master version as possible.

But the files themselves are exactly the same as on the master node(s). This needs further analysis. One of the solutions might be to have the ability to cancel the pending transfers once we detect the change in the file system and then restart some transfers once new torrent file is constructed.

• Again, the 1st step in this procedure is to request a top folder hash (or a magnet link) from some node and if that hash is the same as yours, there is nothing to be done with this node.
• If top folder hash is different than yours, then you request the torrent for the entire collection from that node.
• Any master (r/w) node, at any given time may produce changes to the collection by modifying, adding, deleting, restoring or renaming files. So, in one way or another, that node will look "newer" than others. (According to data integrity across the nodes rule, the r/o nodes may not perform any modifications on any files. Even if they do so, their files will be re-downloaded from the master version).
  But it does not mean that ALL the information it has needs to be distributed to other nodes. For example, some r/o node may decide to modify some file, but that file is not necessarily to be delivered to other nodes because that file has different hash than the same file on the master node, which means that his modification is in fact invalid and is to be overwritten with a version of the master node or the reference node, the original supplier of the information.
  But if some file is on the ignore list, then it can be modified, deleted and even restored, none of which will propagate to other nodes, nor will it be updated from the other nodes (this is the BTSync version of logic). Because it will be considered a private and "untouchable" version on this particular node. Ignore means this file does not exist for any purposes whatsoever, as far as this share and the sync process is concerned.
  Actually, this case may get complicated by the fact that there may be several reference nodes for the same file. Meaning that even though there was some originating node for this information, later on, it decided to pass the reference (or master) status (for this particular file) to other nodes, so that they were allowed to update the same file(s). For example, a publisher may have an original writer. But once the article is written, it then goes to a number of editors, for style, syntax, the volume and so on. Which means that all of these people are now the reference nodes and the latest version of the file prevails and is distributed to other reference/master nodes.
• Once you request or are informed about the top hash modification, your node than requests the torrent file from the node that notified it about modification. At that point, you analyze this torrent according to the rules and decide if you want to update some of your files according to that torrent. In case you do, you request those files, and once done downloading them, update your own torrent to correspond to the state of your node.

  Note: there is one uncomfortable element as far as torrent files go. The standard torrent file does not have the field with file update date/time. In this respect the syncing process in this particular design differs from the purely torrent approach in that in torrent clients, there is no need to know the file update time. As long as they differ, the file needs to be updated. All you need is the hashes of the individual pieces. But the update time is irrelevant.

  But there is a solution to the time stamp issue in a torrent making it "non-standard". You can still use the standard torrents that do not contain the time stamps, but only contain the list of files and list of hashes. If you find that some file in a torrent has a different hash, then you request the supplier to provide you the time stamp in a separate network exchange, if you decide to honor the very idea of a time stamp as something significant and reliable in your opinion. So now, since most of the nodes are usually in the state of full sync and any updates usually happen on individual files, this additional overhead of requesting the time stamp via separate network exchange become a matter of statistics and may not impact the overall performance in any measurable degree. Furthermore, this can be done not just for a single file that differs, but for any number of files in one go, and then sending the request to get their mod-times in a single network exchange.

  But... Basically, the very idea of time stamping the file modifications is not a trustworthy mechanism, and for several reasons. For example, in BTSync we constantly observe the nodes that completely refuse to sync if their clock differs by more than 6 minutes from the other nodes. What we have seen is up to 30% of all the nodes might show up as being out of range in terms of acceptable time difference. Interestingly enough, most of those nodes have a 1 hour difference, which looks like the time zone issue. Furthermore, we have seen the nodes that have their clocks off by DAYS! So, how can you trust the file modification time in those situations? What if their time is set in the future and everyone starts syncing with them? Mind you, this is not even some attack from them, but something they are simply not aware of or pay any attention to for whatever reason.

  But in terms of syncing, the mere fact that your file hashes are different is still insufficient in terms of figuring out which version of the file is either more current or more "authentic", or more "correct". Basically, if the hashes differ, about the only way to know which version is "legal" is to compare those hashes to the version on the master or reference nodes or the guaranteed copies on other r/o nodes, as long as there exists a guarantee that they represent the exact copy of the file.

  Again, the idea here is that no "newer" version of the file is considered to be valid or acceptable unless it was modified and supplied by the master nodes, regardless of the "newness" of a time stamp, which is guaranteed by default in a "hardcore logic", which implies that the r/o nodes may propagate only the master node versions of the files. All modifications by the r/o nodes are considered to be the local or private copies and are in fact subject to being overwritten by the master node versions unless included in the SyncIgnore files.

  Because in case of an attack on a share via "time stamp attack" the file(s) may look "newer", but in fact they are not even the same files, but nothing more than fake data. They have been modified by the attacker to sabotage the share.

  But, "any way you cut it", merely carrying the file update time stamp with torrents does not really resolve the time stamp attack situation, because time stamps can not be trusted, especially in public applications. This means that about the only way to detect such an attack is to refer to the hash of the master version of the file.

• At any given point, any torrents "out there" that are different from the master node can not be classified as "authoritative". The ULTIMATE reference is the reference node. After that, goes the master node, which contains the exact copies of every single file in the collection. So, this is the protective mechanism to assure the integrity and consistency of data across the nodes.
  It is possible to design a mechanism where files on r/o nodes can be trusted even if there is no master nodes on line. For example, we can add a flag "master version" which will be set in case this particular r/o node can verify at any given point that this particular version of the file is indeed an exact copy of the master version. There are several ways to design such a mechanism.
  Actually, with "hard core" data consistency logic, every file in the database of the r/o node is AUTOMATICALLY guaranteed to be the master/reference node version. Because it won't be propagated to other r/o nodes in any other version since its key is different from the master node version and any modifications of the file by the r/o node in the file system, won't change the status or its file piece hashes and the file hash.
  So, the only files that are propagated are guaranteed master node versions.
• So, by shuffling these per node torrents around, you are informing the other nodes of a total and final state of the share, which, ultimately, is determined by the reference or master nodes.
• No r/o node may update any other r/o nodes with any files that do not have the same hash as on the master (r/w) nodes. And under NO circumstances whatsoever the r/o node may update the master nodes or the reference node.
• So, eventually, the collection state will stabilize and all the nodes will update all those files that they wish to update and to be updatable on their nodes. But even if some node notifies the other node(s) of updated files, it does not imply that those nodes are obliged to download those files to update their share/collection (take, for example, the case where the r/o node has this particular file in its ignore list). Except logically, no r/o node may modify any files and expect them to be propagatable to other nodes.
• The main difference between this approach and pure torrent approach is that you do not exchange the torrent files themselves between the users in order for them to be able to join the share/collection. Because they are dynamic and change all the time. In fact, those torrents are temporary and are invisible to the user. They are used by the internal program mechanisms and are not meant to be advertised as an identity or a key for the share. But you can exchange the master node versions of torrents. This would mean that every node may also have a copy of the master torrent in addition to their own local version. (This needs further analysis.)
  But you identify this collection/share by the share key (or a magnet link), which is the same for everyone regardless of the state and condition of the data on any nodes. The share key identifies the collection, and any node, even an empty one, may join it and immediately start downloading the files from all the nodes on line, just like with torrents. Except in our case we discover "the latest and greatest" version of a modifiable torrent via this key, then node discovery, then requesting the latest torrent guaranteed to be the exact copy of the master (r/w) or reference node(s).
  But the underlying mechanisms are exactly the torrent based.
• It has to be noted that any files added by the r/o node do not automatically get included in the collection. The same thing with deleted or renamed files. The deleted files still remain in the collection database and will not be deleted on the other r/o nodes. In fact, they will be re-downloaded from the nodes that have the master copy of them. The same with renamed files. The files with old names will still remain in the collection, and will be handled in exactly the same way as deleted files, and that is, restored. But their new copies will NOT become a part of the collection. Simply because they do not exist on the master (r/w) nodes. The same thing is with modified files. ANY modification of any files on the r/o nodes will be overwritten with the master version.
  All of these are the "hard core" data consistency rules that can not be violated under ANY circumstances. Because they guarantee the consistency (the same exact copy) of the files across ALL the nodes. Else, the eventual data degradation is virtually guaranteed, no matter what other rules you might implement.

  The perfect example of it would be the sync inconsistencies and update issues in BTSync. In particular, their design principle "the user knows better" is a logical absurd as far as data consistency goes and virtually guarantees the eventual data inconsistency across the nodes, and even worse than that, some files may stop syncing completely and you won't be able to restore their normal sync state no matter what you do. You'd have to delete that share from BTSync and re-add it again. But then you loose all your local file modifications and deletions, in case you have deleted or modified some files before you readded the share and expect them to remain deleted. ALL of your files will be restored to the master node's version.

  BTSync behavior table

This is just a rough sketch which will require much more detailed analysis of all possible cases. But this is something that might turn out to be the easiest and most effective and simplest way to go about it by starting with some open source torrent client code base.

Single modifiable torrent approach via magnet link

One interesting aspect of using a single torrent file for the entire collection is that if that torrent file is in standard .torrent format, then it can be either made directly available to download by anyone, or even better, you might refer to it via magnet link, which is effectively the same thing as a key to the collection, except it does not have the torrent file itself. But the magnet link itself provides further information for you to recover the torrent file itself. The link is the same, but the content of what it points to is different. That makes it like GUID of some sort. The link itself is unique.

That means that you simply provide the magnet link and it is eventually resolved into a torrent file. In the most general case, the magnet link contains only the hash or a key for the collection. The peer discovery happens via DHT protocol. But it has the provisions to provide the addresses of several trackers, just like torrents. If no trackers are provided in the magnet link, the only way to discover the peers would be via DHT or via exact or the fall-back exact address of a torrent file. You can specify those in the magnet link text.

Basically, the magnet link differs from the direct torrent approach in that it represents the mechanism of mapping a unique torrent content to the universal and ever valid identifier to the collection. There is really no need to carry around the static torrent files that inevitably get outdated. All you need is the universal one-to-many mapping mechanism, which is what the magnet links attempt to provide.

The actual torrent file corresponding to the magnet link is obtained directly by the peers. This means that in our approach we do not need to pass around the torrent files themselves, and, secondly, as soon as we discover at least one other peer, we will obtain "the latest and greatest" torrent file from them directly. Now, once we discover at least one peer, from then on, regardless of how "old" or outdated our own torrent file is, regardless of how we obtain it, we, from then on, are guaranteed to eventually obtain "the latest and greatest" version of the torrent and collection.

This approach could be quite interesting to consider. Yes, it would work fine if collections do not change. Otherwise, you have to make sure that torrent clients always refer to this share via magnet link, and not via torrent, every time they join this.

So, this means that by simply modifying the magnet link text you effectively refer to a different torrent file. This needs more analysis though.

Sync general design issues

Several things may be noted here.

The basic idea used by BTSync is to use torrents as the underlying mechanism and an engine to do the transfers and updates.

This is a pretty good idea, and for several reasons. First of all, there is no need to "reinvent the wheel". The entire transfer mechanism is well developed by now and is available in open source versions.

The basic idea is simple enough. You simply use the same torrent transfer engine except you create the dynamic and temporary torrents on the fly instead of creating a single torrent of fixed contents for the entire share or collection of information.

The MAIN disadvantage of static, per collection, torrents is that it can not be updated, extended, modified and so on. Yes, its main advantage is that it guarantees you the integrity of a collection and gives you an assurance of reliability of data and guarantees that no data within the collection may be modified. But that is not what is needed in the modern information age.

The nature of information in general is dynamic. It changes all the time. And that is why torrents get outdated. But the problem with them is that they can not be changed once created. You'd have to create new and new torrents for new versions of information, which is highly undesirable. It would be preferable to keep a single version that could be updated, extende and so on.

Now, since torrent transfer engines and mechanisms are a mature technology by now, it does make sense to use them instead of inventing the whole engine again. And that implies that at least 30 to 70% of the work is done once you can use the same technology and create the dynamic and temporary torrents to perform the transfers.

What remains at that point is the high level control logic and GUI. At that point, you have about 90% of all the work done. The remaining mechanisms are key mapping to share/collection, node discovery, such as using the DHT or "predefined hosts", used as a general purpose trackers, and various hashing algorithms and traffic encryption. And that is all there is to it, pretty much.

None of it is that "esoteric" once you start looking at it with a competent enough eye.

So, this is the direction we will address in this document.

Update logic engine

One of the most critical aspects of sync process is stability and predictability of data in the context of update.

It is a tricky process if you allow all sorts of funk and bells and whistles, like it is done, for example, in BTSync. The main problem with their update logic is that if the r/o node modifies the files in any way, then it will no longer be updated from the r/w (master) nodes. And this is a disaster, at least from the standpoint of knowing what is going on, and the effects of such a design are profound and totally unpredictable. BTSync logic is described in the transfer table.

BTSync behavior table

Current file update and propagation logic problems

It is proposed to use a much more strict update logic, based on few very predictable criteria that assures the stability and consistency of data across all the nodes. It is based on several principles:

• No data modifications on the r/o nodes is allowed. Any modifications, file deletions or creation will be overwritten in such a way as to restore the same data as on the master (r/w) nodes.
• No r/o nodes may update the r/w nodes under ANY circumstances, even if it looks like the file version on the r/o node is more recent than on the r/w nodes.
• The r/o nodes may update the other r/o nodes with the new versions of the files, but only if there is a guarantee that the files are exact copies of the reference or master (r/w) nodes. That means that you do not send the updates of the files (if you are r/o node) unless it can be guaranteed that the file is the exact copy of the master version.

  That means there must be at least one r/w or reference node on line (if we chose not to store the master node database locally) and the file on the r/o node is exactly the same as on the r/w node. In that case, the r/o node exhibits the bandwidth contribution behavior. Basically, the ONLY reason for the r/o node to transfer to some other r/o node is bandwidth contribution, assuming the presence of at least one r/w node. Otherwise, the most reliable supplier of information is the r/w node.

  There may be a solution for the r/o nodes updating the other r/o nodes in the absence of either reference nodes or master nodes, or both. For example, if we have a state bit in a the database for the r/o node that says: "this file is guaranteed to be an exact copy of the reference node or a master node", then this r/o node is allowed to update the other r/o nodes even if no reference or master nodes are currently on line.

  So, any r/o node that does not have its file marked as "guaranteed to be good", can not copy the file to other r/o nodes regardless of the time stamp or anything else. This way, you never propagate potentially bad or damaging files and the r/o nodes can not possibly have any files that are not guaranteed to be the exact copy of a reference node's version. This could solve the consistency problem across all the nodes. But this has to be given more thought...

  The "guaranteed to be good" file flag is valid only if file hash is the same as on r/w nodes. (This flag may be cryptographically signed with a key for example). Alternatively, instead of this flag, you simply carry the original file hash from the master nodes. This is a verification mechanism for the situation where some r/o node either modifies the file or copies a different contents to it for malicious intent. In that case, the flag alone is not sufficient and will simply become invalid after file modification. So, you also need to perform the contents check before you are allowed to transfer this file to other r/o nodes.

• The new version of the file "wins" ONLY if it is possible to determine with pretty close to 100% certainty that the "new" version is indeed newer than some other version. For example, if you use the file update time, then, potentially, you are inviting a disaster or even devastation of data on the entire share in case of malicious intent. The point here is that any node may set the file update date to anything it wants, including that in the future.

  So, if you use the file update time as any kind of criteria of a file version, then, in case no master (r/w) or reference nodes are present on line (and you do not chose to store its database locally), you may overwrite the newer file with the older one if its file update time was deliberately set in the future, compared to the existing version. So, if the file update time was originally set to a year ago, and you present the fake file with the same exact file name that has an update time of a year ago plus one day, or even a minute, then you can force all the nodes to be updated to a maliciously created file.

Higher level logic engine issues

Core principles

First of all, we will use the term "system" instead of a "program", as in our understanding, the programs better be viewed as systems and not merely the collections of code. The very definition of a system implies much more rigid requirements, the main of which is system STABILITY. No matter how "sophisticated" is the system, if it is not stable under any conditions whatsoever, it will, sooner or later either degrade or collapse entirely.

The second criteria, which is of vital importance, is data consistency. In case of a sync system, consistency is applicable across the nodes and means that there is only one valid copy of data floating around. That is, in a stable state, meaning no updates are in progress at the moment.

It is highly undesirable, and in fact fatal in some sense to have different versions of the same file be available on different nodes, even though during the file modifications or updates we do have some files that are inconsistent with the rest of the system. But those states are transition states to a new stable state, and are subject to the most rigid update logic to make sure we do not propagate the inconsistent data around.

This, first of all means, that the r/o nodes are not allowed to modify, delete or add any files that are not present on the master (r/w) nodes. This is what we call here the "hard core" logic. If this requirement is not followed, then disastrous consequences are pretty much inevitable, and for quite a few reasons.

For example, in BTSync, as of versions 1.4.nn, the r/o nodes are allowed to modify, delete and add files, and under some circumstances, as a result of what seems to be clearly a bug, they may even update the master nodes, which is nothing more than the invitation to disaster. Their reasoning is based on a rule "the user knows better", which is fine and dandy, except it does not work, and for quite a few reasons.

So, in case, for example, when some user deletes some file, which immediately creates a data inconsistency across the nodes, his deletion is considered valid, except it is only applicable to his own node. This means that this file will remain in its own database but will no longer be updated from the other nodes. This, in turn, creates a nightmare. Because, first of all, the master node can no longer report the status of completion of that particular r/o node an so, the "main" node - the master node, does not really know what is going on with the share, and from several different points of view. But let us not diverge from the main thing here. BTSync update rules and logic is described in:

BTSync behavior table

Higher logic operational components

First of all, the higher level logic engine is the most complex part of the system. Basically, the operation of higher level logic extends to scheduling a transfer. Beyond that, higher level logic is no longer applicable end we enter into a phase of plain, ordinary issues of of assuring the delivery of a correct copy of some file. So, once the file is queued for transfer, that's it, "we wash our hands".

So, in effect, the higher level logic deals with the issues of discovery which files are to be updated and scheduling of updates.

Now, the issues of discovering which files need to be updated are, for the most part, the issues of efficiency and performance. Because fundamentally, there is a "brute force approach", which means, that during the periodic scanning to find changes, we may simply go through every single file of the share and compare the copy (of the file and block/piece hashes), available on disk, with the copy available in the database, and if there is a difference, then transfer is scheduled. Yes, it is the least efficient way of doing things, and yet, it is most reliable and trustworthy.

Master and local databases

In our design, we chose to store two separate databases for each share/folder/collection. First of all, we must have the local database, which contains the latest state of the share from the standpoint of local view on this particular share. But we also chose to store a master database on a local node as well. This is quite valuable because of "hard core" logic, which says: You transfer ONLY the files that are guaranteed to be the exact copies of a master node(s) version. No file transfers take place if we do not know for fact that what we are about to transfer is an exact copy of a master version. Else, the eventual inconsistency is pretty much inevitable.

But we do not need to store the other nodes version of their database on any other nodes, but their own, on a permanent or even on a temporary, per session, basis. Because "we could care less" of what they have. About ALL we "care" is whether we are consistent with them, and if not, then do they have something to contribute to us and vice versa.

This could be done via procedure of incremental discovery (of differences) by first examining the top level hash for the entire share, and then, if it is different, comparing the hashes of subfolders. If any of them differ from our version, then we descend into those folders and now check the individual file hashes. This optimizes the overall performance and reduces the need to carry around useless information about anything.

And contribution does not require a complete knowledge of their entire database. It only needs deltas, the differences, which are, in vast majority of cases, are but a very small part of total database. Furthermore, it is virtually useless to store their entire database on our node. For what? Most of it is exactly the same as our version.

So, what kind of information is stored in the database? Well, we certainly need a file name, actually full path relative to the base folder of the share. Then we need the file hash, and, possibly, for efficiency reasons, the individual block/piece hashes for the file. We also need to store several state bits.

If we decide not to store the the file pieces hashes in the database, then we will incur an additional performance degradation and, in some cases, may not even be able to make the correct decisions as to what to do next. Because there is no guarantee that the master node is on line, and so, we can not even query it from the r/o node in order to get the individual block/piece hashes for this file (This is applicable if we do NOT store the master database on each node. Otherwise, we could query any r/o node about their view of the master version and base our decision on that information).

Secondly, even if we could, we would create the otherwise unnecessary network exchanges and traffic, which could significantly impact the overall performance, especially for the cases of small files. With large files, the additional network traffic to get the block hashes could be minuscule compared to total file transfer time.

(Again, most of this logic is related to efficiency issues and not something fundamentally necessary or required.)

Yes, if each r/o node also stores the master database on a permanent basis, then we do not really need to refer to the master node directly. We can obtain this information from any other r/o node provided that its version is guaranteed to be correct. This is the key. We do not store, nor carry around anything that we can not trust.

File state flags

• "File was physically present on disk during the last scan".
  So, when we refer to some file in a database and we find that it no longer exists on disk, then, depending on whether it is r/w or r/o node, the file is either to be removed on all the nodes (r/w node case), or it has to be retransmitted again (r/o nodes). Because removal of file on the r/o node is not allowed. Otherwise, we immediately create the inconsistency with other nodes.
  The same thing, only in reverse, happens if some file did not exist in the database previously, but it was found on disk, which means it is a new file. For the r/w nodes, this means that this file is to be scheduled for transfer, and for r/o nodes it means that we either have to query the master node directly, or query the other r/o nodes to see if they also have it, in order to find out if this is a brand new file and our r/o node could be the first one to receive it. If this is a brand new but local file, which only exists on our own r/o node, then we mark that file as "non-propagatable" and "non-updatable" and it is considered to be our own private copy not subject to distribution.
  (There is an argument here of whether we store the information about the private files [files that do not exist on the master nodes] in the database. Basically, we may chose not to store the information about any files that do not exist in the master nodes in the database. Because we really do not need that information, since these files are essentially not a part of the collection. Only the master nodes may define what is a part of the collection. Anything else is pretty much irrelevant from the standpoint of the share. Because we do not propagate it, no matter what. But we will skip that argument for now since it is not a fundamental issue, but a matter of efficiency and overhead.)
• "File is propagatable".
  File is propagatable only if it also exists on the master node(s) and it is guaranteed to be the exact copy of the master node's version. If some file is propagatable, then the r/o nodes may transfer it to any other r/o node, (but not to the r/w nodes).
• "File is updatable".
  This means that this file may be updated from the other nodes (r/w or r/o). File is updatable only if it also exists on the master node(s). Otherwise, it is not updatable. For example, a user have added some file locally on his node. But this file does not exist in the master collection. In this case, even if some other r/o node has a file with the same name, we are not going to update our local copy (In fact, if we query some r/o node about any file that does not exist in the master collection, they should simply return "file does not exist", or "file is not a part of this collection"). It is considered to be a private copy, outside of the main collection.
• "File is the exact copy of the master version".
  This is debatable, primarily because of the fact that a share may be attacked by someone who modified his version of the program to behave differently than it was designed to behave. So, this information is basically the same thing as information delivered when file hashes are exchanged. According to the scheme we adopt here, this flag is carried with and encoded into the very file hash. They are inseparable, and, therefore, this bit can not be forged. (Alternatively, the master node flag may be cryptographically signed).
  The encoding can be performed correctly only on the master node that knows the r/w version of the share/collection key, which is used in encoding. The r/o nodes may only decode it and find the value of this flag, and, therefore, even a doctored copy of the system can not affect the other nodes and fake the "master version" flag.
• "File update time".
  This is highly debatable and, probably, the most complex issue among all, related to the file updates and file authenticity and version (or degree of "newness"). Because the file update timestamps can not be trusted, because the user, innocently or intentionally, may set the file update time to anything he wants, and, if we use this time to see which version of the same file is "newer", then we might get a totally distorted picture of reality. Basically, in our design, the main reference is the master node version.
  If we find any conflicts as to the file version or correctness of its data, then there exists no logical way to determine which version is newer or correct, because nearly everything may be faked. Therefore, about the only reference we have is the master node's version of it, and, even if some r/o node's version looks "newer" than even the master node, that changes nothing. The master version stands no matter what. This is a part of the "hard core" logic. So, let us skip this issue for now. We will discuss it elsewhere.

So, this is pretty much all the information that needs to be stored in the database about each file.

Cryptographic information signing

One of the issues of critical significance in the context of public information sharing and open source nature of the system is the authenticity of things. The assumption here is that everything may be forged by the custom build version of the system. Therefore, things of critical nature need to be processed via some cryptographic technique, such as PGP for example. In this light, let us see which information is of critical nature and significance and needs to be signed.

• Identification of the master nodes.
  In the "hard core" logic approach, the main reference point in the system is a master node. Therefore, it is necessary to validate that any and all the data exchanged between the nodes corresponds to the master node version. Simply because there exists no other reliable way of validation. Anything else can be forged and can not be considered as something reliable or authentic.
  No r/o node may elevate its status to the master (r/w) node level, no matter what it does. Because it is pretty much the same thing as gaining a superuser access on the O/S level.
• Exact copy of master node files stamps/hashes.
  The fact of the authenticity of the files may be propagated from the master nodes to the r/o nodes. But ONLY if we can verify that some file(s) indeed originated on the master nodes. In this arrangement, even the files on the r/o nodes may be trusted as thought they have come directly from the master nodes.
  Therefore, master node does not have to be physically present on line and no queries are necessary to be made to it in order to find out if some file is an authentic version. If no master nodes are on line then the only degradation we may experience is not having the "latest and greatest" version of some files, which isn't such a big deal.
  Specifically, the file and block/pieces hashes, passed around, need to be signed.

File scanning

File scanning is an attempt to find out if any files have changed locally on a particular node. It is scheduled on a periodic basis, and, even if our O/S supports the file system notification events, some of them might have been missed, if, for example, the program was not running and file modification happened during that time. So, regardless of anything, we do need to schedule the periodic scanning which is meant to detect any file changes with certainty. No other mechanism will provide us the definite answer on any file modifications.

File scanning is also performed on the program start as a default. During that startup scan we verify the validity of our local database for this node. Alternatively, we may decide not to perform the startup scan, but rely exclusively on the periodic scans. It really does not matter eventually, because sooner or later we still have to rely on the periodic scans.

Mostly, the strategy of scanning reduces to optimization of efficiency, because, otherwise, we could simply use the "brute force" technique and perform the physical reads of the files on disk and compare the piece/block and file hashes calculated during the read to the hashes available in the database that tell us what was the state of any particular files. If our newly computed file hash is different from the one in the database, then we certainly have the file modification case. The same thing applies to pieces.

Basically, during the scanning we are looking for the changed, new or deleted files. In case of deleted files everything is very simple. We simply can not find some file on disk that is present in our database. Therefore, we need to pass this information to other nodes and let them decide what they wish to do with this case. In particular, it depends on whether the deleted files are or are not on the other nodes ignore lists.

In case of new file creation, things are still relatively simple. Some file did not exist in our database and now it is present on disk. Or, alternatively, it did exist in our database, but was marked as "not existing on disk", which means that it did exist at some point, but then it was deleted. No matter what, we need to pass the new state of this file to other nodes and let them decide whether they want it or not.

In case some file was modified, things get substantially more complex, especially in the light of "hard core" logic we adopt here. Several things may help optimize the process:

• File size has changed. In that case, we definitely have a different file and we have to inform the other nodes about this change.
• File (or directory) update time has changed. Since in public applications we can not trust the trueness of file update time, all it tells us is that something has changed about this file/directory.
  We can not rely on directory update time as, depending on the O/S and how some file was modified, the directory update time may remain the same. But if and when we do know that change indeed took place, it is used as further optimization.
  It is possible that file update time has changed but the file is exactly the same, in which case we still schedule this file for being informed about to all other nodes, and we present the individual piece hashes with it, so that the receiving node could look at them and decide to request either the entire file or some pieces that have changed or not to request anything because the file did not change, but only file modification time did. We do not necessarily have to transmit the entire file. All we need to give them is the pieces that have changed. Files may be edited at the end of file or a few blocks/pieces might have changed, but the larger part of the file is still the same. This optimization could be useful for very large files, such as video or audio.
• Pieces hashes have changed (compared to those in the database for this file). In this case, we clearly have a file modification case and the file is to be scheduled for distribution (together with its piece hashes, file size and update time, whichever some node might find relevant).
  To find out if piece hashes have changed, we have to physically read the entire file, rebuild the hashes and compare them to those in the database. If any of them are changed, then we recompute the file hash as well (out of individual piece hashes). Then we schedule this file to be announced as modified to all other nodes on line.

The special case of detection of file modification is the O/S file system events, such as delete, modify and so on. But some operating systems do not have such a facility, so this becomes merely a performance optimization and we can not rely upon it as a general case.

Once we detect the file change(s), we simply schedule the node notification mechanism. But we are no longer concerned with how and when it is done.

One other optimization is to try to schedule several small files and pack them into a single delivery via building a single torrent for a number of small files. Because the network overhead, especially with TCP connections, may be significant if we keep bouncing back and forth about every individual file. But again, this is merely optimization.

Polling queries for updates

We may also perform the polling queries to see if some other nodes have something different than what we have. Since we are dealing with on/off line situations all the time and any node may come and leave any time they want, and so we are, we might like to explicitly query the other nodes to see if we are not in sync with them.

For efficiency purposes, we don't poll for every single file in a share. First of all, we poll for the top folder hash and only if that hash is different than our own, we need to perform the more detailed discovery and request the sub-folder hashes, and if at least one of them is different than ours, then we request the individual file hashes for that folder.

In most cases, polling for updates is a waste of resources since we are in sync with all the other nodes most of the time. But it does not hurt, at least if used in efficient way.

If we find some file hash "out there" different than our own, we then request all the different particular pieces of that file.

Basically, file transfers happen upon request. No node forces any other nodes by transmitting anything to them, unless those nodes specifically request things. It is up to that node to figure out where it is and request for any further information to clarify the states of the particular files.

Polling can be performed either on periodic basis, like scanning, or during the system startup.

The first choice to poll is the master nodes, if there are any on line. If we are consistent with ANY master node, we do not need to do any further polling, at least according to "hard core" logic. Because once we are on line and any node has any changes, we will be informed by it as to those changes. So, there is no need to poll.

But during the startup, if we find that we are inconsistent with our own database, meaning that some file modifications, additions or deletions were performed while the program was not running, then we may chose to do some hard polling in order to reconcile with all other nodes as soon as possible.

Share/Collection degradation problem

The main disadvantage of the sync approach compared to torrents is the collection degradation problem. Which means that once all the master nodes are gone and no longer appear on some share/collection then there is no certainty that any r/o nodes that visit this share do have the entire collection as it existed on the master nodes.

Furthermore, an extra measure of care needs to be taken in order to make sure the new r/o nodes may even start downloading the share. For example, if it is required for the master node to be present on line in order for the r/o node to construct the correct database, then the r/o node can not even start downloading anything because it never has a chance even to build the database.

Now, if we carry the cryptographically signed "master version" flag in a database for each file, then any r/o node may contribute its part to the new r/o node database, even if their part is not the complete master node database. This means that even if some r/o node on line has a single file and no other r/o nodes are on line at the moment, the new r/o node can still build its database even though it contains a single file, and it can download that file since we know for fact that this is an exact copy of the master node version. Later on, when some other r/o nodes come on line, then the database for the new node gets expanded.

Basically, with "hard core" logic, we can assure the theoretical maximum fitness of the resulting share to the share on the master nodes, which can assure that the resulting database will indeed be a union of all the individual files on all the nodes that ever visit this share. We can do that because in our logic we are guaranteed that the only files that propagate are those that are cryptographically signed to be the exact copies of the master node version.

This means that the very notion of share degradation becomes somewhat inapplicable. Because the very fact that we do no longer have any r/w nodes on this share, even permanently, we still have the latest snapshot of the master node version of the share, which, in turn, may be logically equated to the "latest version of this torrent ever released" in torrent terminology.

And again, probably the main advantage of this approach is the fact that we can start downloading the new node(s) even in utter absence of the master nodes on line, and so this share in reality "never dies" or "never degrades" as it will live on in as close version to the master node as possible.

Actually, the behavior would be exactly the same as if you got some partial torrent. Meaning that at some point when the torrent seeding node was last seen on line, there was only one other leecher node that was able to download only a part of total torrent. From then on, if no seeder node ever comes back on line, this torrent will remain partially complete until the end of times, regardless of how many other nodes join that torrent, unless they have some files to contribute to the incomplete torrent.

This is excellent and quite an unexpected result that removes probably the main disadvantage of the sync approach vs. pure torrent approach. Furthermore, it removes the second disadvantage of the sync approach, which is the collection is not guaranteed to be healthy, unlike with torrents. With torrents, all the files that were downloaded completely are guaranteed to be healthy, which would be impossible with other sync designs that do not utilize the cryptographic signature of the master version.

This design once more reasserts the design principle of "no r/o node, under ANY circumstances, may distribute any non cryptographically signed versions of the files" guaranteed to be the exact copies of the master node versions.

Furthermore, no r/o node even informs any other nodes about the fact that it may have some other files in the share directory tree that are not a part of the master collection. Those extra files are considered to be a part of the local version of the file tree in the file system and have nothing to do with the share as far as sync process is concerned.

Any existing file(s), that were renamed to the new names, effectively "move out of the share" and no information about them has to be kept in the share's database, and, if they are renamed back to their original name in the future, they are still to be re-downloaded as though they did not exist in the share. Because once they are renamed to the new name, they "fall off the share" and we loose their cryptographic signature kept in the database only.

The logical requirement for such seemingly harsh treatment of renamed files (vs. merely renaming them back without the need to re-download them again) is that we also need to keep their cryptographic signature when they are renamed, if we expect them not to be re-downloaded once they are renamed back to original name. So, if we keep them in the node's database with their signatures, then we create the new files with the master node status and they become subject to the propagation, unless we utilize a special mechanism to handle that case possibly with the help of some extra bits in the database. One way to resolve this conflict would be to also mark the newly named files as "non-updatable" and "non-propagatable" in the database.

In that case, even though we do keep those files in the database, we no longer either update them, nor propagate them to other nodes, nor do we even inform any other nodes that we do in fact have these files under new file names. But probably the simplest and most straightforward solution would be to simply move them out of the share and "forget about them" as far as this share is concerned, because unless those are huge files, re-downloading them again should not represent such a big problem. Plus, the solution becomes much simpler and we do not introduce any probable inconsistency issues if we have to juggle a couple of extra bits in the database. But this can be argued.

Share/Collection index and database

There is a need to keep the state of the shares/collections on per file basis because during the time the program is not running there may be some changes in the file system, compared to what we knew about the share, and we need to detect the changes since the last time the program was running and watching things.

When the program starts up, it can perform the startup scan to determine "what's new out there". During this scan we can use either the database for the share and check and see if the physical files in the file system are the same as in the database. We may also be interested in scanning the file system and check it against the database in order to discover new or modified files. We can not discover the new files if we only use the database as a reference, that is, we only check those files that do exist in the database. We have to do it the other way around as well.

We need a high performance database engine because we may have some huge shares with hundreds of thousands of files, which is not uncommon. Also, the files may be huge in themselves. So, we need a high performance indexing mechanism to quickly check some file and get the information we need to see what to do next.

Each share/collection has its own individual database. When some share is deleted, all that happens is deletion of its database. No actual files are deleted for variety of reasons. If user wants to delete some share entirely, first he has to delete its key/folder in the program and then physically delete the files from disk.

The database needs to store the file tables using the file path (relative to the root folder of the share) as a primary key. Here is what we need to store in the file table:

• File path relative to the root folder of the share. This is the primary key and all the searches are done on this key.
• File hash which stores the hash of all the block/piece hashes.
• Block/piece hash array/list which stores the individual hashes for each piece of the file.
• File size.
• File update time (normalized to UTC).
• File state flags

When new share is created on the master (r/w) node, the initial indexing is performed and the database is constructed for the share. The process of construction is gradual and some sync operations may start even before the share is fully indexed. This is important for the very large shares that may take minutes, hours or even days to fully index. Basically, as soon as we fully construct a row/record in the database we can start using this file for sync operations.

On the r/o nodes, the database may be constructed even if no r/w nodes are present on line at the moment (if we cryptographically sign each file in the database to indicate that it is the exact copy of the master node version).

Basically, the r/o node database construction goes like this:

• The r/o node requests the root folder file list from any node, r/w or r/o, does not matter in our design. Then it looks up to see whether this file already exists in its database. If it does not, then it adds this file to the request list to query the other nodes about the details of the file.
  (Alternatively, we may receive the full file information from the other nodes about each particular file that it has, instead of querying for the file name and then sending an additional request to get all the file info. This is pretty much a matter of efficiency and not something fundamental).
• For efficiency purposes, we may chose not to query about each file separately, but bulk the requests for multiple files in order to minimize the network traffic.
• When we receive the information about the file(s) that we do not yet have in our database, we add the new records for those files. Once the record is fully constructed, we may immediately issue a request to download that file. So, the process of constructing the index and downloading files run concurrently, again, for efficiency and user experience purposes.
• As soon as we download some files, even before the entire index construction process is complete, we may respond to other nodes requesting those files that we already have in our database and have already downloaded their actual data.
  (Further efficiency improvement is to be able to start downloading the selected pieces even from those files that have not entirely completed download, but only have a few fully downloaded pieces so far. This is a more complicated procedure since the files are not directly downloaded to the file, but are downloaded to the temporary files first and only if the entire file was downloaded completely, we rename those files to the correct file name).
• Since we add to the database and download ONLY those files that are marked as "guaranteed to be the master node copy", at any given point, our partially constructed database and files become fully functional as far as sync process goes (in respect to those files/pieces that we have fully downloaded).

One of the other reasons the databases are needed is to minimize the amount of core memory we need and release any and all the memory occupied by the things we have already done. This is important in large systems with a number of shares, some of which might be huge.

Otherwise, we may have the memory leaks which will eventually not only lock up the program, but may even bring down the entire O/S or JVM (Java Virtual Machine), just because we ate all the available memory.

(In particular, Syncthing/Pulse as of version 0.10.5 suffers from the memory leaks, it seems, and the program has to be restarted every few hours).

Initial index and database creation

One of the tricky aspects of update logic engine is the initial index and database creation. The complication primarily arises in case where there are no master (r/w) nodes on line while some new node initially joins the share. It is further complicated by the fact that the new node may actually contain most if not nearly all the data from the existing share at the moment when it joins some share. For example, if, for whatever reason, the database was damaged, there might be a need to remove some share from the sync and recreate it again in order to recover the correct state of the database. But all the physical data is already on that damaged node. So, in case of very large collections, rebuilding the entire index and a database may imply re-downloading the entire collection, which is the last thing in the world anybody would be interested in.

So, this means that even if you need to rebuild the index, you need to be able to determine that you do not actually need to download the files that are already there.

This means that you should be able to join some share and reconstruct the index and database in the most efficient way.

But the question is: what if you create some share in some folder that already has some files in it and what do you do with those files as far as adding them to the database? Those files may be a part of the collection as it is known to all other normally functioning nodes, or it may have some extra files on the disk that are not a part of this collection. What do you do about those files and their index?

Well, logically speaking, you do not want your database to contain any files that are not a part of the collection. Otherwise, you might incur some overhead in processing and may create logical inconsistencies across the nodes, where one node thinks this and the other node thinks that, which is an invitation to eventual disaster.

In order to solve this problem we can adopt the following strategy:

• When some node creates a share/sync folder, we can not start indexing and creating the records in its database before we can determine which existing files in the file system for this folder do actually exist on the other nodes and are in fact a part of the share. If they are not a part of the share, we do not add them to the database, but only if we are the r/o node. But if we are creating a master node (r/w), then we can start creating the database immediately upon creation of the share/folder.
• If we are creating some share and there are no other nodes on line and we do have some existing files for this share on our node, we do NOT add them to the database until at least one other node comes on line and so we can verify the files.
• Therefore, for r/o nodes we build the database by first sending the info requests to all the other nodes on line and first of all create those database records that already exist on the other nodes.
• As we keep adding the files to the database on r/o node, we check to see if those files already physically exist in our folder, and if they do, we set the appropriate bits for those files in the database, such as "file exists on disk". The file and file block hashes are computed and are compared to those provided by the other nodes. If those hashes differ from the other nodes, we add the requests to re-download those files from the other nodes. This can run in parallel to creation of the database.
• Since we construct the database by requesting the other nodes for existence of files in a share, we do not need to bother about any other files that physically exist in the file system of our node. Because they are not a part of the share.
• With the master (r/w) nodes, we may decide to speed up the process by reversing its order. Since we are the suppliers of all the files that do already exist in our node, we may decide to first of all go through all the existing files and add them to the database without even bothering to request the other nodes about them. Once we add all the local files to the database, we then, first of all, start informing the other nodes about our files so they could start downloading them immediately in case they do not already have them.
  Secondly, we request the other nodes to provide us the information about the files that THEY have and, in case we do not have those files on our node, we add them to our database and request those files to be delivered to us.
  For r/w nodes, it does not really matter which files we start adding to our database first - those that we already physically have in our file system or those that the other nodes have. It does not really change anything fundamentally. We may as well do the same thing as with the r/o nodes and request the file information from the other nodes first and then and check to see if we already have them. If we do, we mark the database records for those files appropriately, that is "physically exists on a disk", "propagatable" and "updatable", and in case they do not, we put them on the request list to be downloaded to us.
  Actually, for the r/w nodes, these processes of requesting the other nodes about their files and informing the other nodes about our own files may run in parallel. They may be considered independent and do not create the logical problems of index/database creation.

It should be noted that we should be able to build the database and the index for new node even in absence of any master nodes on line. This means that the r/o nodes MUST only contain the files in their database that are the part of the collection and are guaranteed to be "good" (the exact copies of the master node versions). Otherwise, if we are the r/o node, we will not be able to start downloading the share if there are no other master nodes on line, which is a disaster and is totally unacceptable.

Share/Collection hash and a key

There are two separate notions related to each share/folder/collection. One is a folder hash and the other is folder key (which is also a hash, but not uniquely derivable from the folder hash).

Folder hash is used exclusively for node discovery and can not be used to gain access to the share/collection.

Folder key is the key that is used to actually be able to access/join some collection and download it, at the very least. Key can not be used for the purpose of node discovery, just as folder hash can not be used to gain access to the share.

So, folder hashes may be publicly distributed without concern that some undesirable party may gain access to the share/folder.

But folder keys give one the ability to access the data for the share.

Selective sync

For now, this is just a stub with the most basic idea. It needs more thought and work, but it is something to think about indeed.

Selective sync means that you can selectively sync only those files that you are interested in. It is particularly applicable to the situations where some web pages or even the entire web sites may be blocked, (in particular, by the powers of evil to explicitly prevent the distribution of truthful information about the evil "ruling this planet" and its nature, tricks, tool and methods it utilizes).

Many people might not be aware that what we have at this junction is a global information war raging on as we speak. Quite a few very important documents and even the entire sites have been blocked by now and it is only going to get worse and worse, beyond any doubt. It is a huge subject to cover and we are not going to do it here. We have done it in other places and some of it elsewhere in this document.

But, what we would like to discuss is the mechanism to access only the selected web pages on some site as one of the most typical examples of the selective sync mechanisms. Or, we might wish to view some video or listen to some audio files even without downloading the entire collection. Actually, there are quite a few cases for selective sync.

Assume there is a great site somewhere, but it was blocked, at least for some potential visitors either in their country or by their ISP or globally for that matter. So, that site becomes inaccessible and the information is lost for all practical purposes. Because you can not access it by going to that site directly.

But the information is not really lost. It is still available at the source, be it someone's home computer or elsewhere. So, what can be done in this case in order to be able to access the information from the blocked site is to distribute the sync key to the collection. In that case, the site becomes accessible again on P2P basis, directly from some sync node(s). Except the potential viewers might not wish to download the entire site, but would simply like to look at some selected web pages.

In order to do that, you need to provide the ability to selectively sync only some files out of the entire collection. In that case, when you join some selective sync share, first of all, it automatically downloads the main TOC file for the collection. It may also download the common necessary files and/or subdirectories, such as common CSS files and images or other things. Then, once those files have been downloaded, you may open the main TOC.html file, which we call The Navigator and it will show you the list of all the web pages in this collection. Then, you might simply click on any of those links and the program will selectively download only the files necessary to view that page.

There are variations on this theme and different ways to implement this functionality. For example, there is MeshedSites and the XTTP plugin that is meant to work with BTSync as it stands right now. We tried to install it but for some reason were unable to get it working properly and we do not have much time to spend on it at this time. Basically, everything seems to be working, the XTTP plugin installs and when you click on some link for some site it is indeed added to the BTSync share collection. But no other nodes could be seen and, therefore, no download starts. Who knows, it could be because there are actually no nodes present, at least when we tried it, which seems to be strange. But here it is:

MeshedSites
https://www.meshedsites.com/

Basically, once you enter a key for the selective sync folder, there may be a checkbox in the edit box for this functionality. If that checkbox is checked, then sync does not start automatically except for the common files, such as CSS, images or anything else.

You may provide the text file with well known name which contains the list of all the files that are automatically synced.

But if you selectively sync only some HTML, PHP or some other file, then it may contain the references to other files that are required to properly display that page. That creates quite an issue and will require parsing the page to extract the links to other required files and so on, which opens up a Pandora box. A compromise might be to download any subfolders in the directory where this file is under assumption that you have some common subdirectories for images, css and so on. But there is no straightforward and reliable solution to this set of issues.

But, at least static HTML pages should at least display, which may be sufficient in case of some important information where the page formatting is not as significant of an issue compared to contents.

Unfortunately, for dynamic web pages this isn't going to work, at least to some extent or functionality.

But, in the worst case you may simply sync the whole collection or at least select which subdirectories you would like to sync. All of this needs more thought and analysis, but it might be something to at least think about.

Uncensored chat and blogs on P2P basis

Some of quite interesting applications of sync are chats and blogs on P2P basis. For one thing, while working with BTSync we have observed all sorts of problems with other nodes and in some cases they would sit on some share/folder/collection but things were broken on their end, partly because of all sorts of bugs in BTSync that would put them out of sync forever, and they did not even realize it, sitting on some share that would not update for them from then on.

The sorry thing about it is that you can not even tell them about their problem and can not advise them how to fix the situation. Because there is no chat facility available even though sync application could handle things like chats and blogs easily.

There are some attempts to implement the chat facility, like, for example, Vole, which works with BTSync underneath, and there is even an effort on the part of BT to provide a P2P based chat facility as a separate program. But why does it have to be a separate program when you already have the sync mechanisms in place with BTSync?

So, we can say quite a few things about desirability of chat facility combined with the sync main program in a seamless manner, and there are different ways it can be implemented.

But what we can say in summary is that - yes, such a facility is highly desirable, even for the simple practical tasks of communicating between the nodes possibly telling them about the problems they have and communicating all sorts of other useful information.

One thing to be kept in mind is not to worry about full-fledged web-like behavior as far as dynamics of the page update. Even if the display is done in the most primitive way, without any text formatting or styling, it is still quite acceptable, at least at the initial phases of implementation. Because the value of exchanging even the simplest kind of text messages is already quite something.

We've been experimenting with chat in the context of straight BTSync program by simply creating a separate folder/share for the chat. Then we would create a standard HTML page for chat messages. It even has any kind of standard HTML formatting and styling available. It all works fine and syncs between the participants within seconds. But its major limitation is that you need to edit that web page in your text or HTML editor and try to find the right place to put your message in and type all the things besides text itself in order for it to look good. But this is way too primitive to be called the real chat.

In the most primitive way, you can create the simple text pages and drop them into that chat folder and they will sync with all the participants. Except they'd have to periodically look into that folder to see if there are any new text files available. But it does work and we were able to help a few people with their issues by merely editing their text file and providing some answer or advice.

All we can say is that such a facility is highly desirable and quite useful. But what is the best way or a mechanism to do it remains to be seen and investigated.

Preliminary Update logic engine design and strategy

Here, we will present some preliminary ideas on design, architecture and key strategies for the engine. Following posts were made on the Pulse (previously known as Syncthing) forums:

Node discovery and trackers

One of the important issues is node discovery. For that, you are welcome to use quite efficient DHT network approach, which allows for distributed storage of node discovery information across all the nodes of the network. So, this way, you do not carry ALL the DHT traffic on every node and do not need to store the entire DHT tables on your box. You just carry the traffic and a part of the DHT tables of the nodes "closest" to a particular hash you are requesting.

And so is the issue with trackers. Technically, you do not have to provide some centralized, prewired tracker. You MAY, but it is not a predefined requirement. The trackers need to be configurable on per installation basis. Sure, you may provide a default and "guaranteed to work" tracker with a release. But even that one may stop working for whatever reason, and could even permanently go off line in the future.

I also like the approach of any node behaving like a tracker and upon ping would tell the requesting node about all the nodes it knows about for a given share. There could be even two different pings - global - per node, and per share only. You may even decide to utterly prohibit the per node global queries, purely for security and privacy purposes, again, configurable on per node/installation basis. But this needs more thought, especially in the private network applications, where you know for fact that your data for some share(s) is strictly private and confidential and you may not even let the other arbitrary nodes to query some information about those particular shares.

I also like the idea of keeping the node history. Meaning, once you see some node on line, it goes into your history file. So, even if that node goes off line for any length of time, you can still have its information. So, in cases of static IP/domain:port situations, you can periodically query the nodes you saw before to see if they respond. This is a fallback strategy to node discovery.

Basically, I am of the opinion that node discovery needs to be fully decentralized, or at least give you a configurable option to do so. Nodes and networks may have their own trackers or they may share some of them and should not, at least in most cases, rely on some fixed central tracker.

But when you see that the TCP connection may be more beneficial for some specific operation(s), then why not use the TCP? The TCP connection does not HAVE to be permanent to any node, if you find for any reason that you would not like to keep a huge number of connections simultaneously. But I think on modern hardware, you can easily utilize at least 50 simultaneous connections without winking an eye. The additional TCP handshake and connection overhead is not that significant once you start a transfer session. It is only inefficient in cases when you have burps of information, such as various queries that do not necessarily cause any further communication with that node.

Furthermore, BTSync can fall back on alternative protocol. This could become important for example if the company/organization network blocks some protocols for whatever reason. I recall some such discussions on BTSync forums.

Basically, you don't have to get stuck with specific protocol, and even there, for the most part it is just an efficiency issue and not something fundamentally important.

So, all these broadcasts, handshakes and all sorts of bells and whistles can be implemented in variety of ways. But fundamental things, like higher level update logic engine can not and remain the same, regardless of anything. Because they are designed like a tank to handle the craziest conditions imaginable and guarantee the data consistency across all the nodes.

Hope this helps.

Assuring the valid copies of the files in a share/collection

This is a pretty critical issue in public applications. It has to do with validity and integrity of files across all the nodes and making sure the correct version of the files is distributed.

Probably the main problem with public applications is that there is no guarantee that some node will not do some things that are damaging to the entire share/collection. They may not realize what they are doing and do some things the effects of which will be propagated to all other nodes, such as deleting or modifying their version of the file, adding new files and so on.

Or, some share may be deliberately attacked, either in order to create chaos on it or to destroy it entirely. We are seeing this with BTSync every day, literally.

Therefore, there is a need to assure that only the correct copies of the files are distributed. The most "correct" copy is the copy by the original supplier of information, which we call the reference node. Then we have a concept of a master node, which means a node that is guaranteed to have the exact copy of ALL the files on the reference node and has the r/w permissions and is allowed to update the other nodes with newer versions of the file(s).

Beyond that, there exists no logical way to assure that the information is valid.

• You can not rely on the file update time stamps. Because those can be changed any time anyone wants and to be set to any time they wish.
• You can not rely on the file version numbers, like, for example, it is done in the Lampert clock. Because again, those versions may be set to anything imaginable.
• You can not rely on any other r/o node to supply you the "latest and greatest" version of the file, because it can fake any file with any kind of garbage, and that file may even look "newer" then yours, at least according to its update time or its version number.

So, before you can accept any files from the r/o nodes, you MUST be sure that these are not merely fakes.

So, what do you do in that case? Well, you know for fact that the original supplier of information is the only reference version of it. The 2nd in validity would be the master (r/w) nodes. Beyond that, nothing can be guaranteed.

But what if there are no reference or master nodes on line at the time you wish to transfer some files? Well, then you need to assure that any time the original version of the file is being transferred, you also carry a validity flag with it, which indicates that this version can be trusted as valid. But that flag can not be carried separately from the file hash. Otherwise, since it is an open source program, anyone may change how things work and set any flags they wish to any value they want.

That means that when you transfer a file from the reference node to other nodes, including the master nodes, you also include the validity flag encoded in the very key. That means that when you are calculating the file hash, you also prepend or append the validity flag as a part of the file itself, and so the hash is calculated accounting for the flag. The hash itself includes the value of the flag. So, when that key is decoded, the value of that flag can be trusted as not the faked one. This means that even the r/o nodes will carry and transfer the valid file versions ONLY. There is simply no way to transfer the incorrect versions of the files in this design.

Furthermore, with the "hard core" logic, there is no way to transfer the files unless they are guaranteed to be the exact copies of the originals. So, even on the r/o nodes, once the file hash is decoded, the value of the flag may be tested to see if this is just a fake copy that might have come from some other r/o node that modified his own version of the file(s) for whatever purposes, intentionally or not does not really matter from the standpoint of validity of data.

Decentralized node discovery

Here's one article that addresses the node discovery via UDP broadcasts in the context of multiple network interfaces:

The reply to the initial UDP broadcast needs to contain the currently active listening port. So, you send out the broadcast packet on a well known port and all the available and willing nodes reply to it with their listening port. Then, after the followup handshake on P2P basic, the requesting node AND the node that have responded are said to have established the communication and may add the other side to their "nodes seen" lists.

Transfer and higher level logic engines

These are two of the most important subcomponents of the system. The "good news" is that the transfer engine code may utilize a well known and well tested existing code, which is available in the open source. Let us look at something known to be well tested by time, such as libutp library. Again, the idea here is that there is no need to "reinvent the wheel".

The UTP protocol was introduced in uTorrent and since then has been implemented in numerous torrent clients. It works like a champ and it is a high efficiency and high performance protocol that can handle any realistic transfer speeds without problems, which implies that it is highly efficient.

The libutp library is one of the most important parts of the system if developers decide to go the route of utilizing the underlying torrent based transfer mechanism, which would provide the bulk of the engine code to the entire sync application.

There is a compelling reason to consider a torrent based approach because, first of all, there are open source versions available and secondly, and could be primary, it provides the bulk of the code for the entire program. All you need on the top of the torrent transfer engine is hooks that dynamically construct the torrents for one large or many small files in a single torrent and simply pass them to the torrent based transfer engine. So, the amount of development work reduces to bare bone minimum.

The basis idea is simple. Let us consider a model which includes the database of all the files in a share/top folder/repo/collection. Once some share is created, the program then goes through all the files in all the subfolders of the main folder and computes a simple hash for each file and stores it in the database.

Besides the file hash, you also need to carry some state information as to the state of the file, such as we describe in BTSync related transfer table logic. Specifically, once you refer to some file, you need to know whether that file physically existed in a file system the last time your referred to it. If what you see in the file system is not the same as you see in the database for that file that means that this file was either modified or deleted or restored after being deleted or renamed. These are the MAJOR states you need to carry in the database for any file.

So, once some share is created, the first step in this model would be to go through each file in the file system for this folder and calculate the file hash. Then you need to set the file state bits as "physically present in the file system", "propagatable", "updatable" (from other nodes). We have considered it in relation to BTSync logic problems here:

Current file update and propagation logic problems

Now, once you have gone through each file and indexed it, you can start the actual transfers if there are any other nodes present on this share. Actually, you can start transfers even before you finish indexing the entire share once you know the file hash and the state for a given file. But that is certainly something desirable, but not a necessary requirement. It translates into simple inability to start the actual transfers until you completely finish the indexing, which is nothing more that slight inconvenience.

So, once you index the share, then you can start constructing the torrents, in case you are the supplier of information, and tell the other nodes that you have "the latest and greatest" version of some/all the files.

Actually, the process is a bit more intricate as you can initiate transfers either by announcing, if, for example, you have detected a local modification of some file. Or it may be initiated by requesting the information about any given file from the other nodes on a periodic basis. So, the process of initiation of transfers needs to be considered as a separate issue and in quite a detail since the results may be slower than expected frequency of updates. For example, it is higly desirable to start updating other nodes as soon as the O/S level file system events are detected, such as file update or removal.

Torrent construction

For efficiency reasons, you may construct a single torrent for many files if those files are too small and would unnecessarily create too much of a network traffic activity for not much of a benefit. So, you may decide that "I am not going to start the transfers if I have more than one file to transfer and its size is less than say, 1 MB". Why would you want to create the unnecessary network traffic if you have tens of very small files that could be packed into a single torrent? Because a torrent may carry ANY number of files without any problems.

UTP transfer protocol

And now let us look at the UTP transfer protocol, which is available as open source.

Since UTP is a well established and tested with time high efficiency protocol, this definitely deserves a pretty serious look at. Again, this could provide to be the major part of the transfer engine, aside from torrent construction and update higher level logic.

The advantages of using the torrent based engine are multiple. First of all, there is no need to "invent the wheel". Basically, this would provide the core components of the transfer engine. You just need a higher level sync engine logic on the top of the torrent engine and "you're done", almost... Well, not exactly, because you also need at least GUI, which, if written in Java, would provide an excellent, very nice looking and as powerful UI as you can imagine, considering that it would be portable across the multitude of platforms.

It is our opinion at this time that it would be highly desirable to create a well defined sync logic engine which can communicate to all the major blocks or subsystems of the rest of the program via well designed programming interface. Basically, probably the most portable and clean interface between subsystems would be the use of the interfaces. This would clearly and cleanly separate the subsystems and reduce the poking of one subsystem into another to the utmost minimum. There should be no peeking and poking across the modules outside of the interface mechanisms.

The sync logic engine is the highest level logic of any sync operation, and that logic could get extremely complex in a hurry. Because there are myriads of cases that need to be well designed and handled.

This is one of the major problems with BTSync. They simply do not seem to have even a notion of a high level sync logic. As a result, the behavior of the program is highly inconsistent, unpredictable, and many logical cases are not handled properly which results in inconsistencies of data across the nodes and in respect to the master or reference nodes.

The sync logic table needs to account for any modifications, updates, deletions, restorations of files imaginable. Unless every cell of this table is filled with the exact, precise and consistent behavior, you are bound to end up with database inconsistency issues, which means disaster as far as data goes. Unless you assure and GUARANTEE the consistency of all the files in a share/folder/collection and across ALL the nodes, regardless of the fact that some of them may be on or off line at any given time, and for any conceivable length of time, your system will eventually and inevitably degrade as far as data consistency goes.

Some nodes will not be updated, or updated with incorrect version of the file, or they may even update the other nodes with older version of the file, which might look "newer" to the system if, for example, someone has decided to attack the share by modifying the files with any kind of garbage information and then setting the file update times to the later time than the original or reference versions of the files, in which case the garbled files might be, and are likely to be distributed to all other r/o nodes since these garbage files will have the exactly the same name but will look "newer" to the system. This can create havoc across the nodes. This case certainly has to be considered. Otherwise, any user can obliterate the entire share if you, for example, do not refer to the master or reference nodes to see it any file is actually a valid file as available on the master node(s).

And we had a chance to observe such a bizarre and highly damaging behavior with BTSync in too many cases to even mention here. Basically, we can observe all sorts of damages to the data or incorrect reporting in BTSync pretty much all the time. In fact, we can not point to a single share that was not damaged in one way or another and/or its reporting being partially or totally incorrect, and in the most profound ways. This is just to accentuate the overall significance of this issue.

There might be enough reasons to have not only a notion of a master node, but the reference node as well. The difference between the master nodes and the reference node is that the reference node is the ORIGINAL supplier of the information, and, if ANYTHING whatsoever does not reconcile between any nodes, then the only guarantee of an authentic version could be obtained from the reference node. The master node is a node that is guaranteed to have the same exact copy of a share, (including every single file), except it is not the original supplier.

Yes, there exists a tricky part of the notion of the reference node in cases where you have several master (r/w) nodes, each of which might also be an original supplier of some information. But, on per file basis, there could be only one and one only original supplier. So, in the database you may have some files where one node is the original supplier, but the other files were originally supplied by some other node. But this seems to be solvable issue, if, for example, we mark the reference node status on per file and not per node basis.

Why this difference between the reference and master nodes? The difference is that in case the reference node is off line, how do you determine that your share is not merely attacked with garbage files, or being damaged inadvertently? Any r/o node might become corrupted at some point, intentionally or not, and it may look like it has the "latest and greatest" version of some files. For example, if user touches some or all of the files with his current clock time, then all his files become "newer" and would have to be resynced.

The same happens if some node, discovering some error, decides to copy some files from the sync archive to the main folder, but he does not copy the files with preserve flag, in which case the new copy of the file will have an update time equal to the time when it was copied, which is now. Thus, the newly copied version will look like "the latest and greatest" version of the file despite the fact that the file is actually an old version.

Yes, if file hashes did not change than you do not actually transfer the files to the other nodes. All you have to do is to update their time stamps. Well, in this case, you can refer to any master node.

So, this is a little extra overhead related to consistency. Before you do any updates, you need to verify the file against the reference or master nodes. That might create some extra network traffic overhead. But it will dramatically increase the consistency across the shares.

There may be many master nodes, but there is only one reference node for a specific file at any given junction (again, the reference status is on per file bases, and not per node global).

But what if the reference node goes off line permanently and never comes back? Well, in that case you need to design a strategy of passing the reference node status to one and one only master node, and there are various ways of doing it. But let us skip it for now because these points are too fine at this junction.

Now, some files may never be updated if, for example, some user modified his local copy or deleted the files or restored the previously deleted file, after which they can not resync because the database "thinks" these files have been deleted and permanently marks them so, just like BTSync does it, which is probably one of its most critical design flaws as far as update logic goes.

There are many more pretty critical cases and issues to be considered and we will try to cover some of it if we find some spare time and we can see that this thing is going forward and is alive.

So, the sync engine higher level logic needs to be well designed and account for the most minute cases, "possible" or "impossible" - does not matter. In real life, those things that "should never happen" DO happen. Therefore, the sync logic must be as reliable as a tank and not miss or fail in any situation whatsoever. This is probably the core issue of a reliable sync system.

The MAIN rule in ANY system is STABILITY. There isn't anything more important in the system as stability, no matter how simple or complex that system is. Because if you can not guarantee the stability under ANY circumstances whatsoever, the system will crash, in one way or the other, or may perform in incorrect ways. And you can not possibly assure the stability if you can not guarantee the consistency of data across ANY nodes. Therefore, this rule is VITAL. Else the system will collapse or degrade, and inevitably so. It is just a matter of time...

So, the sync logic engine is the main subsystem that assures the consistency across all the nodes of any given share.

Finally, here's the code for libutp:

libutp
https://github.com/bittorrent/libutp/

Go language version:

https://github.com/andradeandrey/GoUTP/tree/master/libutp

About trackers

Any node should also have the ability to perform as a tracker. This means that any node which has been discovered by any other nodes by ANY means should pass the list of all the nodes it knows about for the particular share to the node sending a node discovery request.

For example, in BTSync, if you specify the "predefined host(s)" for some share, then those nodes will also send you the list of nodes they know about on this share, except a predefined host can behave like a tracker ONLY if it contains a physical copy of the data of the share, which is incorrect of a design.

(Actually, in BTSync ANY node behaves like a tracker in terms of passing the information about the nodes it knows about on this share.)

ANY predefined host should be able to behave like a tracker REGARDLESS of whether it has a physical copy of the data or not. You simply have to specify it in the configuration for the share. For example, in GUI edit box for this share you simply click on a check box "use as a tracker only", which means that this node does not have to have the actual data for the share, and yet, it can behave purely as a tracker node and pass the list of all the nodes it knows about to any requesting node, simple as that.

Furthermore, it should be able to maintain a list of previously seen nodes that are currently off line. Because now they are off line, but they have the static IP and the same port and may come on line ANY time. So, when they will come on line the next time around you should be able to get to them even if no other means of node discovery are available, such as DHT or "predefined hosts". Technically, it is no problem to remember ALL the nodes some node has seen on some share in the entire history. Because all you have to remember is the IP and the port, which means that you can remember hundreds of hosts without much problem.

About hashing

Well, hashing could be effectively used for identification of information purposes. For example, the base-32 SHA1 cryptographic signature is "good enough for the poor man like me" identification purposes.

You can identify even large amounts of information by simply calculating a hash of the entire file, block or a very long message. Yes, the guarantee of uniqueness of the hashed information is not 100%, but the statistical chance if misidentifying it could be easily 1 to a million cases.

You may calculate the hash not only to the entire file, but to individual blocks of that file. This may allow you to identify the partially transferred but incomplete blocks within the file, resume the file transfers of very large files by updating only the blocks with incorrect hashes and so on. File hash may be calculated out by going through the list of all individual block hashes, which could be useful for performance purposes.

Furthermore, to increase the efficiency of the entire transfer logic engine you may compute the folder hashes as well. This may significantly increase the performance of updates and minimize the amount of network traffic in order to figure out which files need to be transferred to other nodes or which files need to be requested from them.

So, in order to see what has changed in a share/top folder/collection, you don't really need to get their entire database or index and then compare it to your own in bulk. What you can do instead is to calculate the top folder hash by calculating the individual subfolder hashes by going through all the files in them, calculating the file hashes, and then calculating the folder hashes out of those file hashes. Then you calculate the top folder hash out of all subfolder hashes.

Basically, you would still have to periodically rescan your collections even if you do handle the file system events, such as delete or update. Because some nodes may be off line and during the time of event they won't be able to get information and do the appropriate thing. So, no matter how automatic or "great" is your mechanism from the standpoint of efficiency or performance, there are still countless cases where some things or events are going to be missed. This means that you can calculate all your hashes by default during the periodic scans. And, if you properly handle the file system events and recalculate the individual hashes as soon as you see a need for it, then you might obtain the maximum efficiency, performance and reliability of any file related operation.

As a result, to see if you are in sync with some node, all you need to do is to request the top folder hash first. If that hash is the same as yours, you have no business to do with this node and you don't even need to communicate with it any longer, except during the periodic global scans.

But if top folder hashes are different, then you need to obtain the individual folder hashes either by going through all the subfolders you know and requesting the individual hashes for those, or by asking the list of subfolders available to the node you are communicating with. And so you descend deeper and deeper into the folder hierarchy in the process of discovering which files were changed, deleted, added or renamed.

So, hashes allow you to dramatically reduce the amount of CPU load and network activity in the process of discovery what has been changed in all the nodes you know about. That is the overall idea.

About logging

Logging is one of those often overlooked facilities to assist in all sorts of reporting, including the operation status, transfer completion, various warnings and errors.

Well thought out logging system is a tremendous help from nearly any angle you can look at it and, quite often, is more effective then debugging techniques.

Basic principles of logging are relatively simple and do not have to be something "fancy" or too sophisticated. There is no need to overcomplicate things that are basic in nature.

First of all, logging needs to be done to a file, and not just to the screen, and that file should be created automatically as a default, on, for example, per session or daily basis. This way, in case you find some damages or errors, you can go back in the logs, possibly even from the previous sessions, and find out exactly when and why this problem had occurred. But the logging needs to be sufficiently precise and give you full paths to the files and things like that. You might have some files having the same file name, but either located in different places or belonging to different shares. So you need to be able to distinguish between them.

Furthermore, there may be several logs. One of them could be more or less detailed description of operations and the results. It is worth noting that the these things need to be logged in as complete way, as possible, providing full paths of file names and full identifying details of the source and destination.

Also, status information about success or failure of some operation is very helpful, including as complete details, as possible, so there would be no doubt what exactly happened, duing which exact operation, to which exact files and so on.

Then there may be a warnings/errors log. If you utilize the convention of categorizing errors and warnings by the degree of severity, then even looking at very large logs represents not much of a problem in order to find out what exactly was the issue.

Each log message needs to be date/time stamped and start with some easily searchable and identifiable string, such as

• "*** Operation successful ***", or
• ">>> WARNING: file complete_file_path could not be created/written to, operation skipped...", or
• ">>>>> ERROR: remote node full_ID_of_node stopped responding, operation could not be completed".

The number of "*" or ">" characters at the beginning of a message would correspond to severity or significance of the message. So, to quickly find out the most critical problems with possibly huge transfers, all you'd have to do is to search on ">>>>> ERROR: " string in the log file.

About portability and User Interface

Syncing as such, by implication, requires the highest degree of portability because it is desirable to be able to sync all sorts of devices including the popular NAS and mobile devices. The bigger the variety of devices the program supports the broader its applications.

One noteworthy application is using the sync program on a headless box, such as virtual hosts or other servers. That might place a set of additional requirements on the programming language from the standpoint of supporting GUI-less operation, or, alternatively, the browser based GUI where program is run without the need for the native GUI support, but with support of some other interface to the external programs, such as browsers, using some well established notation or data exchange format, such as JSON, for example.

Basically, the program must be able to run in a headless configuration.

Extensible architecture

Also, providing the well defined interface to the key program functionality and mechanisms will allow to hang on the top of the basic engine and program a GUI that fits the user's taste and needs the best. "There are no friends in tastes and colors" they say.

So, the program core is the same, but you can have several GUI packages, including the native mode O/S dependent high performance GUI or alternative and "universal" browser based GUI that could be accessed remotely.

Another thing here is that the users may create their own plugins for the core of the program to perform whatever crazy things they wish to do. So, if the program core is fully exposed from the standpoint of any logically possible and discrete and self-sufficient operation, then the program becomes not only merely an open source, but open and extensible ARCHITECTURE, and that is really something, which is entirely possible and not that complicated if you design your interface to the external programs really well.

About development strategy

One of the complicating factors of development strategy is the fact that one needs to have both ends functioning, at least to some degree, in order to be able to proceed with development in effective and productive manner. Because both ends of the sync need to work, at least to some extent.

What is a further complication is that many operations need to be developed symmetrically. You can't have just one end of the wire work and the other one not being able to cooperate.

This means that the very strategy needs to be stable and steady. You can't add some significant amount of code and/or functionality if you can't have both ends functioning and cooperating.

So, the very approach would be slow and steady, without creating too much of a radical change at any given point. Well thought out logging mechanism would help tremendously in this process. Again, because it might not be even feasible to rely on break points and single stepping. Else you might need to single step on both ends, and those could be drastically different, such as your PC based system and a headless server.

The main rule of reliability and stability

Well, this is one of the most common errors made by the programmers. They keep adding new things and all sorts of bells and whistles but never quite test all the conceivable and even "impossible" situations. So, what happens as a result is that they are being hit by those "impossible" situations nearly on unending and constant basis. But, instead of stopping and fixing the problem right then and there, once they see it, they just keep ignoring it because they are "busy" with other "great" things and features.

And, what is even more significant, is that at some point, if you do not fix things ASAP, as you first noticed them, then the amount of problems will simply accumulate. At some point, you might simply get overwhelmed by the innumerable number of problems, just as we see with BTSync, for example. It is amazing to see their main tech support guy on forums hasn't gone insane, dealing with piles upon piles of problems many of which are not even easy to replicate in a test lab setting.

Because once you get overwhelmed with problems, the whole project begins to give in and degrade, if not collapse. If that happens, any and all development has to be stopped entirely and every single effort and resource to be allocated to fixing the bugs and in a prioritized manner from the standpoint of stability, reliability, predictability and assurance of the consistency of data across all the nodes.

Otherwise, the end result in most cases is that the program works and/or performs well only under ideal conditions. But the "real life" has NOTHING to do with some "ideal conditions". The most unimaginable things happen ALL the time, whether you like it or not.

So, the "moral of this story" is quite simple: as soon as you see some bug or some malfunctioning or anything that represents ANY kind of problem, FIX it, right then and there, and you will never have to regret doing it. Because at any given junction your program will be stable and reliable and will work under any "crazy" conditions imaginable, which will only make you feel good, instead of forever feeling uncomfortable constantly seeing these seemingly small and insignificant things which you chose to "ignore for now" in the name of some "greater good" which is one of the grandest illusions there are.

Related useful open source repositories

The following repositories might contain the bulk of the necessary code:

• Vuze/azureus
  https://github.com/syj6516195/vuze/
  Vuze/azureus (original SVN version)
  https://svn.vuze.com/public
  Vuze/azureus is one of the more popular and revered torrent clients. It is written in Java, which is excellent for portability. Vuze/Azureus is not just "yet another torrent client". More proper description for it would probably be a large system, a "monster" of a program, that incorporates all the advanced torrent concepts, DHT node discovery, high efficiency UTP transport protocol, webUI support, and even plugins, which, upon closer look might turn out to be "the way to go" if one wants to incorporate the sync functionality into such a system. But this needs a detailed analysis.
  It also has a well defined interface that supports access to most important mechanism. Async notifications should represent no problem in this architecture.
  This is probably the prime choice for considering as potential sync client base code for the engine, GUI, the interface and other mechanisms.
• ezvu-torrenter
  https://github.com/kiwib1rd1337/ezvu-torrenter/
  ezvu-torrenter is clone of vuze without all the advertising and "copying is theft" B.S.
• qBittorrent
  https://github.com/qbittorrent/qBittorrent
  qBittorrent is one of the most popular and powerful BitTorrent clients and has the support for DHT, UTP and all sorts of very useful mechanisms that could be easily applied to the sync application. It is written in C++.
• Vuzetty
  https://github.com/alepar/vuzetty/
  Vuzetty is a remote torrent upload plugin for vuze torrent client and upnp control.
• alevy/comet
  https://github.com/alevy/comet
  Comet: An Extensible Distributed Key-Value Store. Version for Vuze/Azureus in Java.
• azdht
  https://github.com/arcusfelis/azdht/
  Azdht is Azureus Distributed Hash Table (DHT) in Erlang.
• Azhtmlwebui
  https://github.com/thecabinet/azhtmlwebui/
  Azhtmlwebui is Azureus HTML WebUI.
• Bootstrap-dht
  https://github.com/bittorrent/bootstrap-dht/
  Bootstrap-dht is DHT bootstrap server.
• TomP2P
  https://github.com/tomp2p/TomP2P
  TomP2P is a DHT implementation for node discovery.
• btapp
  https://github.com/bittorrent/btapp/
  Btapp.js is a backbone library that provides easy access to Torque/BitTorrent/uTorrent clients..
• btc
  https://github.com/bittorrent/btc/
  btc is command-line BitTorrent remote control (btc).
• jewel/clearskies
  https://github.com/jewel/clearskies/
  jewel/clearskies is open source btsync clone.
• clearskies_core
  https://github.com/larroy/clearskies_core/
  clearskies_core is distributed, secure data synchronization using the clearskies protocol.
• syncthing
  https://github.com/syncthing/syncthing/
  syncthing is continuous File Synchronization.
• syncthing/discosrv
  https://github.com/syncthing/discosrv/
  syncthing/discosrv is the Syncthing global discovery server.
• syncthing-cli
  https://github.com/syncthing/syncthing-cli/
  syncthing-cli is the Syncthing command line interface.
• libtorrent
  https://github.com/svn2github/libtorrent/
  libtorrent is a clone of SVN repo of libtorrent, one of the most popular torrent libraries.
• libutp
  https://github.com/bittorrent/libutp/
  libutp is uTorrent Transport Protocol library.
• GoUTP
  https://github.com/andradeandrey/GoUTP/
  GoUTP is a UTP reliable transport over UDP for Go, using libutp.
• Hive2Hive
  https://github.com/Hive2Hive/Hive2Hive/
  Hive2Hive is Java library for secure, distributed, P2P-based file synchronization and sharing.
• bittorrent/node-bittorrent-tracker
  https://github.com/bittorrent/node-bittorrent-tracker/
  node-bittorrent-tracker is a simple bittorrent tracker written ontop of node.js.
• bittorrent/poco
  https://github.com/bittorrent/poco/
  bittorrent/poco is the cross-platform C++ libraries with a network/internet focus..
• pulse-java
  https://github.com/dapperstout/pulse-java/
  pulse-java is Java port of the Pulse (formerly syncthing) protocol.
• ownCloud
  https://github.com/owncloud
  "ownCloud gives you freedom and control over your own data. A personal cloud which runs on your own server".
• OneSwarm
  https://github.com/CSEMike/OneSwarm
  "Privacy preserving peer-to-peer data sharing".
• Lsyncd -- Live Syncing (Mirror) Daemon
  https://github.com/axkibe/lsyncd
  "Lsyncd watches a local directory trees event monitor interface (inotify or fsevents). It aggregates and combines events for a few seconds and then spawns one (or more) process(es) to synchronize the changes".
• hollow/inosync
  https://github.com/hollow/inosync
  "The inosync daemon leverages the inotify service available in recent linux kernels to monitor and synchronize changes within directories to remote nodes".
• SparkleShare
  https://github.com/hbons/SparkleShare
  "SparkleShare is an Open Source collaboration and sharing tool that is designed to keep things simple and to stay out of your way. It allows you to instantly sync with Git repositories and is available for Linux distributions, Mac and Windows".
• DirSync Pro
  http://dirsyncpro.org/
  "DirSync Pro is a small, but powerful utility for file and folder synchronization. DirSync Pro can be used to synchronize the content of one or many folders recursively".
  Note: it only works locally and can not sync over the network. This is a pretty nice and quite configurable program. And, even though it does not do what we are after, it may turn out to be something to consider as a code base. Since it is written in Java, the portability would be one of the best.
  From the quick look at the sources it seems that the program needs some mechanisms that make it a really powerful sync engine. Because it seems to lack the more precise and more efficient mechanisms to determine the file updates and lacks the mechanisms for efficient and reliable determination of what has actually changed in the files, especially for the cases where files are modified only partially. Also, reliance on file update times is applicable only when you deal on the same local system and not across the net. The mechanisms for file/piece hashing are basically absent altogether. So, it does need some more work and thought put into it before you can consider it a general purpose sync engine for the network applications.
  Basically, in the most basic version, all you need to add to it is the network code, based on torrents, for example, which is available in Vuze/Azureus open source code base. But we did not have a chance to study their code and so we can not make any specific comments.

Other synchronization systems:

• Unison File Synchronizer
  http://www.cis.upenn.edu/~bcpierce/unison/
  "Unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.".
• Csync2
  http://oss.linbit.com/csync2/
  "Csync2 is a cluster synchronization tool. It can be used to keep files on multiple hosts in a cluster in sync. Csync2 can handle complex setups with much more than just 2 hosts, handle file deletions and can detect conflicts".
• Csync
  http://www.csync.org/
  "To be more precise csync is a client only bidirectional file synchronizer".
• SyncReplacement
  https://libreplanet.org/wiki/Group:SyncReplacement#csync
  "SyncReplacement is an FSF Priority Project".

Where to get BTSync?

Note: we do not recommend installing version higher than 1.3.109 because of all sorts of UI problems. You can get v.1.3.109 here:

You can get BTSync v. 1.3.109 right here:

Other platforms

But, if you like to install the "latest and greatest version", (or 1.3.109 does not support your device), you can get it here:

Download Resilio Sync

BTSync 1.4.72 disaster warning

And if you do decide to upgrade the version, make sure to back up your entire btsync configuration directory (C:\Users\username\AppData\Roaming\BitTorrent Sync directory on Windows). Specifically, make sure to back up the settings.dat and sync.dat files in that directory. Also, make a backup of all the database files (all the files with long names in capital letters and digits, *.db, *.db-shm, *.db-wal.

Because it 1.4.72 is going to override your previous installation and if you decide to go back to it you won't be able to.

Quite a few users wanted to go back to 1.3.109 which is easier said than done. We have spent two full days trying to recover and are still not recovered fully. It is hard to describe by anything other than a nightmare.

There are several very significant issues with 1.4.72. Probably the main one is a new GUI, which is IE (Internet Explorer) based, and is probably the most primitive UI you have ever seen in any stand-alone program. Because IE was not designed for creating a full fledged GUI to begin with, and, even if it can do some things that LOOK like something GUI-like, it is as far from the real GUI as you are from the planet Mars. We do not have time now to talk about it in specific details. Just a couple of points in passing:

1. There is a compatibility issue of 1.4.72 with some previous versions (do not recall which ones). This means that if you have some nodes with older versions of BTSync they won't sync. This issue is a killer in a general public setting where you don't have access to each node directly and can not even tell the users how to solve the problem. It simply won't work and they won't know why.
2. There is a compatibility issue with Windows XP/Vista/Server 2003 users.
3. With version up to 1.3.109 your main operating tab would be Devices where you could see all your shares and all the users for each share on one screen. That is the most comfortable way to work with the program. With 1.4.72 you would have to move your mouse, select some nearly invisible dots to the right and then click on some share to see its users. Now, while you are watching it, you can not see what is going on with your other shares or other things.
4. You can not resize the columns in a way that makes sense.
5. You can not rearrange the columns.
6. You can not copy any text from GUI to the clipboard which makes it impossible to copy some status and other messages, long strings, possibly written in foreign language, long hash strings with random letters you can not possibly remember and so on.
7. There is too much white space wasted on the screen and so, if you try to make it smaller, you won't be able to see anything of real value left. The classical example of a well designed GUI and screen space utilization is the Windows O/S itself. Everything matters down to a single pixel and no pixels on the screen are simply wasted. That way, you can pack a lot of information in a relatively small window or a dialog on the screen and then resize it if you need to, just like you do with Windows Explorer or some other file browser.
8. There is no File Transfer screen. This thing alone is nearly a disaster. Because when some transfers occur, and there could be several at the same time, in the previous versions you could see which exact files are being transferred and to which users, which is probably the easiest way to estimate the progress of things especially when it gets busy. With 1.4.72, you are virtually blind and can not see what is transferred and to whom and what is the progress of things.
9. Dialogs are modal, meaning that once it is open, you can not do anything else with the program until you close it.
10. There is no OK and Cancel buttons in the dialogs, which means that the values and parameters and settings are modified immediately, just as you type. So, if you make a mistake and close the dialog, it will have the wrong values left, because you can not cancel out, nor you can undo some possibly wrong values entered.
11. Dialogs can not be resized. They just sit there in a fixed place and with fixed size, just like dummies, and you can not even move them out of the way if you need to look at other things on the screen. This is simply pathetic.
12. Text contrast between active and inactive items is too little and makes them virtually indistinguishable. For people with poor eyesight or with poor screen lighting conditions you won't see any difference between active and inactive items.
13. Generally, there is too much of a mouse move and click to see something useful, and even there, you still can not see one useful screen that pretty much gives you the whole picture of all your shares at the same time. You have to constantly move your mouse and click on some things that are not even easy enough no notice and open up yet another dialog box that does not necessarily tell you much. Again, pathetic is probably the best word to describe such a primitive UI. It is definitely the worst this writer has ever seen in any stand-alone program UI design, even on the amateur level.

BTSync 1.4.72 has a totally redesigned GUI and it runs under IE "under the hood" and users are not even informed and/or not aware of it, or may not even suspect it, but what they see on their screen is IE running. For some users it is a matter of principle NOT to run anything from the Microsoft beyond the O/S itself. We have asked: WHY are users forced to use some 3rd party software? - No answer beyond ridicule.

There was a court case in EU against Microsoft shipping IE as a part of a Windows as a default, thus slanting the opinion of people in favor of running IE instead of installing some independent browser. Microsoft lost that case. But with BTSync, we not only have the same exact issue, but users a FORCED to run IE and thus the "message" from BT is essentially "get used to it". What do you call this?

First of all, it is not even clear what are your security risks when you run some program via IE. Why would you want to have another questionable issue with such programs as BTSync where you already have more than enough of questionable security or privacy related issues and some even allege that BTSync works or cooperates with the NSA and all these "encryption" and "security" claims are as fake as three dollar bills, just because it is a closed source program?

And why do you have to run a bloatware web browser program just to run the BTSync? What is the "message" here? - You can not exist without Microsoft? Any place you look there is Microsoft with backdoors to the NSA?

Also, in 1.4.72 some things have changed as far as special files are concerned. In particular, the .SyncArchive directory, where all the previous versions of files are stored, is now located in .sync subdirectory of the main directory for the share. Also, .SyncIgnore and .SyncId files are no longer located in the root directory for the share, but are also moved to the .sync subdirectory and their names are different now. Also, the file format for all those files was changed.

There are other things in 1.4.72 that have a profound effect and need a detailed analysis and understanding. But we do not have time to cover it right now. So..

Perpetual beta excuse for not fixing critical errors

To make the long story short, all you have to do is to ask yourself a question:

What would I do with my car which is broken forever, to the end of times?

If you see an answer to this question, that is all you have to read from this chapter. But, in case you are still curious, here it goes:

One of the main excuses for not fixing piles of bugs is called "beta excuse". It allows one to do anything they want in terms of adding new bells and whistles instead of doing those things that MUST be done in beta stage - BUG FIXING. This means that when you see people complaining about tons of most critical bugs, you simply blame it on "beta", like beta is like a rubber condom that could be stretched to the size of a zeppelin. So, let us first define a beta release.

The support staff, with exception of one guy, usually provides some scratchy and technically incomplete or imprecise answers, at best, and if you question any of it, they usually pull their standard "excuse": "But it is still in beta", and even try to ridicule you and make you feel guilty of something, even though it is THEIR problem. And if you keep trying to clarify the issue, the procedure is: ridicule, insult and then delete, kill and ban. Simple as that.

They even invented a new term for this pathetic excuse - "perpetual beta". So, the thing can be broken forever with this excuse and you can not even expect anything other than their excuses, including "but this is a free software". So what? Linux also is free software. Does it mean it has to be as flaky as BTSync, which is probably the flakiest piece of software you have ever seen in your life?

Like it means beta implies broken or half-working something. Well, BTSync has been in this "beta excuse" trip for a year now, and even today to call it beta is misrepresentation and entrapment, essentially, and luring more and more unsuspecting users into using a half-broken program, with hidden motive to corner the sync market. That is probably the closest interpretation of it all.

Why entrapment? Because most people would be willing to try out the beta version in understanding that this is something nearly as good as the final version, and people would never even suspect that this isn't even close to beta. Even to call it alpha is misrepresentation. And not that many people would be willing to try the alpha version because it is nearly assumed that there are likely to be problems with it.

It creates this eternal excuse for not really doing a good job on fixing tons of things that are broken, and instead, they forever keep adding all sorts of useless bells and whistles to entrap more and more customers and users with all sorts of "new features", while under the hood even old features do not work correctly or in a logically consistent manner.

If you just look at BTSync forums, you will find such a huge number of problems that it is hard to believe someone can even release such a thing as a product. At the rate they are going, it is probably going to take at least a year or two before this thing becomes real beta.

With such a huge list of problems, and we are talking about some vital issues of top priority grade in any self-respecting software outfit, is not acceptable in any professional product, or even in some open source project. And we have mentioned some of the key problems months ago, but they have not moved a finger to fix most of them, even the most critical ones. And we are talking about things that make the whole thing not to work in some situations, as bad as that.

In particular, the so called timediff problem, that totally prevents some nodes from syncing, have been brought up and we were even able to determine the exact technical issues related to it, and even proposed a solution to this nasty problem. They said that they will see what they can do to help the users in the future release, and since then there has been about 10 releases, but the problem is still there, and we see it every single day with several nodes on different shares. Not a thing has changed.

Furthermore, this problem is so nasty that it can screw up the shares on the nodes so badly that they won't be able to sync at all, or their stuff will get all messed up. To someone who understands the issue and its significance it should be a top priority item, to be fixed before anything else. And we are seeing its effects in the nastiest possible ways right now and have been struggling to fix things for days, LITERALLY, and having no luck, no matter what we did.

It looks like a market department driven organization with extremely poor management and technical oversight. And when companies start running according to the marketing department ideas, those companies quite often end out of business. Because they never fix things to make their product stable and reliable, and instead, keep adding more and more complications and all sorts of bells and whistles, thus loading the technical departments with things that take away their resources from things that do need to be done, and quite often, urgently. But there is no time for it left. Because marketing is always in a hurry.

Furthermore, some of the most obvious programming errors one can see nearly everywhere in BTSync, going down to such a basic level as GUI design, simply indicate that the level of technical competence of their coders is on the level of a three months class in some community college - total incompetence.

For example, let us take probably the simplest case of item selection in a list of some dialog. If you select some item, such a device name in Device tab, or items in the Transfers or History tab and the list is updated, then the item you have selected is no longer selected. What is selected instead is the item with the same index position from the top of the list, which is not even laughable and a sign of total incompetence, but it is simply pathetic. Any programmer with a few months of education could do it correctly within minutes. This author has never seen any program that can not handle this case correctly. All it takes is to select the item not by its index in the list, but by the key. That is all it takes.

And this simplest thing in the world creates a nightmare when you try to see what is happening with some device and devices come and go all the time. It is nothing short of a madhouse to try to keep up with it. It is simply a circus level.

We could provide here a list of the most obvious programming errors that could be fixed in minutes by someone who knows what he is doing. We have mentioned some of those things in their forums months ago. But they simply ignored it like something so insignificant that it does not even deserve an note or a response on their part. And if we look at the issues of much more fundamental and critical level, it might look like they simply have no clue of the most basic "no-nos" in multi-threaded programming.

The results of it, just as reported by users on numerous threads, is that the program starts eating the file handles and memory. After a while, it simply chokes your comp and it grinds to a halt and the program may even crash or grind to a stop. Because some program threads deadlock because of improper use of the synchronization objects and violations of the most basic rules of synchronization. So, they simply abandon those threads and keep creating more and more threads and/or file handles. It is too technical to discuss here and there is really no need. What will it change?

When we brought some of those things on the BTSync forums and asked some questions on how to recover a totally broken installation after trying to run 1.4.72, there was only one semi-intelligent response from one of the users. But all we got from the BTSync related personnel, such as "alpha testers", was nothing short of an insult and ridicule and by no means a precise and technically correct answer.

Eventually, they have started to simply delete the messages without addressing the issues and trying to cover it all up and make it look like those messages are not "on topic" for that thread. And this is not the first time they are doing it. It seems like their favorite practice to handle the most touchy issues. We don't even know how many other messages from other people they have removed trying to hush things down.

We include just a couple of those messages below. The first one is all that was left of them. All other messages were simply deleted. Not a single answer with sufficiently precise explanation, description of anything or instructions was provided by BTSync staff. They even went as far as to insult the writer and make him look stupid by saying things like "well, I do not know how to make it simpler for you", while they have not answered the question or addressed the issue in a competent enough way.

And it happens that some "alpha tester monkey" was insulting the most senior level developer trying to make him look stupid, instead of providing the precise answer and technically competent instructions, throwing around the lunacy terms like "you have to remove the configs" and things like that. Well, WHAT constitutes the configs? There are configuration files, there are configurations in the Windows registry, there are databases, history files and so on. So WHAT EXACTLY do you have to remove? Which key in the registry? Which exact files and located where? What kind of answer is this?

And, mind you, we are dealing with fatal level issues. The entire program no longer works. NOTHING works. But it worked perfectly well just before the installation of 1.4.72! And THAT is how you treat your users, you funky monkeys with three months of programming class in some community college, at best, while talking to some of the best developers of the architect's level there are with equivalent education and experience level of a Phd.?

I have a question for you: Are you REALLY insane, or just pretending?

I did ask one of them a question: are you a sadist? But he did not have guts to reply. Because he certainly is. Just look at his/her favicon and try to figure out his/her gender and "preferences". It is not even clear if we are dealing merely with the transvestites or outright mutants and hybrids from the "dark side" directly. Looks like Martians are here already.

GreatMarko. Alpha tester of BTsync. Hey, what planet are you from?

Hello... from "the dark side"!!!

To psychologists, this is a typical picture of a suppressed degenerate sadist. Stalin has ordered a study of physical indications of degeneracy to be conducted by the top medical experts. They have identified about 40 typical indications of degeneracy expressed in birth defects. Several of those indications are clearly present in this picture. But this is a totally different subject. So, we'll skip it.

So, here we go with the last message that survived this butcher-mutant-sadist-wannabe something big. How many of other on the point messages has he/she deleted? How many people did he/she insult and how many accounts did he/she/it close? Guess...

Messages on BTSync forums about 1.4.72 upgrade disaster mutilated and eventually deleted by the "moderators" and alpha test monkeys

So, basically, what you see on BTSync forums is a highly censored and mutilated version of reality. Yes, once in a while they do allow some pointed questions or comments, probably to make it look like a balanced view on something. But we know for fact that there are so many most critical issues in BTSync and there are so few posts addressing them that this can not possibly happen in a world not filled with lunatics and complete zombies.

Basically, what this predetermined commitment to stay with IE, no matter who says what, means is that this product has no future. Even without this IE "improvement" the product has quite a list some of the most critical issues that do not seem to be even scheduled for resolution. It seems that as long as there are new bells and whistles offered every few days, that is all that counts, pathetic as it gets. The most important thing to be done seems like producing as much hype as possible.

Existing open source sync alternatives

Syncthing (Pulse)

But the good news is that there begin to appear similar products. For example, there is Syncthing which already works for private networks. That means that if you only want to use it to sync between several devices to which you have access, everything should work for you as good as BTSync. Syncthing is also quite portable and supports a variety of operating systems and devices even though some people complain about absence of support for some esoteric devices. You'd have to look at it yourself to determine if it supports your situation.

Unfortunately, they do not allow just any user to join the shares automatically. It all depends on manual approval by other nodes, which means that you have to sit there by the screen 24 hrs./day and watch the requests to join showing up. But from the standpoint of a new node that wants to join some share but can not until he is approved, the whole thing might look like "well, this thing isn't working as advertised" to the new users and they are likely to abandon the whole share or even the program. After all, how many hours or days would you be willing to sit and wait and see nothing happening?

For general public applications, where you use the program to distribute a collection of information, joining the share or repository should be fully automatic, just like with torrents. Once you have the torrent file, you can start downloading stuff immediately assuming there are other nodes present on that torrent at the moment.

That is the MAJOR problem with Syncthing, even though there are other desirable things to be implemented, but none of them are critically important for the most part.

So, the program seems to be reasonably well suited for the private networks, but not for the general public use. But those guys have begun to discuss the easier ways to join the shares and the main developer has promised that version 1.0 will have that feature working. We'll see.

Links to their forums and release page.

Syncthing/Pulse Releases

And here is support forums:

Documentation

Support

Development

Using Syncthing for public distribution of information

First of all, our PRIMARY interest in Syncthing/Pulse project is the public distribution of information applications, just like with torrents, for example. Yes, the program works, more or less, with purely private applications, such as syncing your home box with your camera, laptop and so on. And yes, these applications are useful. But that is not interesting to us in the context of the most urgent work of information distribution that has profound significance and affects ALL the people, regardless of who or where they are.

So, this is the primary context for any information about Syncthing you will find in this section.

Update: Unfortunately, we did not consider one major design flaw, and that is ANY node may configure itself to be a master node, which means a disaster for the public distribution applications. Because the master node has full and unconditional access to a share/collection and may do with it anything imaginable, and it will be unconditionally delivered to all the other nodes on a share. Any actions on the master node, such as file deletions, modifications, additions and so on will be automatically replicated on all the other nodes, including all the master nodes.

This may damage or totally destroy the entire share/folder, either as a result of misunderstanding of how the program works, or for deliberate reasons to attack the share.

Basically, the problem is that the master node may simply replace the files in a share with any kind of garbage and it will be propagated to all the other nodes, including the other master nodes, which is a disaster. Or, if some user on some master node decides to delete some files because they are not useful to him, then these files will also be deleted on all the other nodes, and he may not even realize the consequences of his actions, possibly thinking that it only affects his own node and his own collection.

Furthermore, in the current design, which utilizes the Lampert clock for the archival file version, they can make their version anything they want by merely renaming the files to any version number they want, in case the program also considers the file version to make the decision on whether to perform the file update or not.

In case the file update time is used to make a decision on updates, it may be set at the attacking node to anything they wish.

As a result, all the nodes will be updated with their changes, no matter what all other nodes, including the master nodes, think or do about it.

Update: As of version of Syncthing (0.9.18), there is a concept of an "Introducer" node, which might look like something that could be used for automatic joining the shares in general public application. Unfortunately, joining the share isn't really automatic, even with the introducer nodes. Effectively, this concept does not help much in terms of joining the share. The new node still has to be manually approved by all the other nodes that wish to share a particular folder/share with it. In order for the sync process to start between the nodes, both nodes have to enable the node(s) with which they are willing or are interested to exchange the files.

So, for now, the entire issue of using Syncthing/Pulse for general public applications is basically totally "no go" situation. Full stop.

The good news about it is that it is an open source product, which is exactly what is needed. Unfortunately, it is written in Go language, that was created by Google and has some built-in mechanisms for "global, transcontinental, massively parallel" blah, blah, blah and so on. But what has to be kept in mind when you deal with Google, no matter what issue you are dealing with, is that Google = Evil. Even the CIA and NSA pale in comparison to Google, which, at this particular junction, is probably the most powerful and influential force on this planet, and in many respects.

Long story, but if you study it, you will see it in quite a colorful picture, even though not everything is quite as apparent, as it might look. And the evil starts with the language syntax. Everything is turned upside down. So, reading that code is like reading something standing on your head.

Proposal to developers to implement the public shares functionality

This post was made on syncthing/pulse forums.

How to make Syncthing/Pulse BTSync comparable by implementing the public share/folder functionality

This post was made on syncthing/pulse forums.

Public share/folder functionality
GUI design

Disastrous consequences for public shares

Well, persistent denials by the main developer to even consider the issues of public applications of this program with an excuse that "it was not designed for that" are not entirely groundless, even though they are in principle, if one looks into it on a deeper level.

The problem number one, and probably the most significant one of all, as far as public applications go, is the fact that the very concept of a share/folder/collection ID is entirely absent. Each share is identified by the share NAME, which, first of all, can not be guaranteed to be unique in public applications. Yes, it may work just fine for small, private uses where one controls the share names, but in public applications you can not be guaranteed that your name will not collide with others "out there".

Secondly, since there is no separate share ID for the master (r/w) and r/o nodes, ANY node may configure itself to be a master node, which will eventually and inevitably lead to devastations to the entire share in public applications, which we observe nearly all the time with BTSync.

If you have a separate ID for the r/o nodes, then they can not possibly damage the share. They can only GET the information from the master nodes, or other r/o nodes, but they can not update them in any way, shape or form. And you can also grant the master status to other nodes, if you are certain that their intent and understanding of the program workings are on the par with what it takes.

So, since you join the share by merely specifying its name, then it becomes impossible to distinguish between the master nodes and r/o nodes. The net effect is that ANY user may configure itself as a master node and simply fill the entire share with garbage or destroy any files merely by replacing them with any kind of garbage, which will be immediately propagated to all the other nodes and they will be effectively destroyed. Here's an "explanation" of one of developers:

(Note: we will comment the most significant points in-place in italics.)

First of all, the arguments are profoundly incorrect. EVERY ONE of them. Basically, these are not the arguments, but the concoctions, trying to defend the conceptually incorrect design that has a fundamental flaw and a built-in potential for damaging or even destroying the shares.

This entire line of "reasoning", which is nothing more than denials of the REAL issue, as real as it gets, simply translates into a dire warning to all who might think of using Syncthing for public applications without even suspecting that their share may be eventually converted into a garbage dump if someone decides to attack it.

This particular developer is simply not being honest and is trying to defend an essentially bankrupt and clearly incorrect position and a critical design flaw. Unfortunately, these are the kind of developers that will be eventually left on the project after all the "undesirables", who vocalize their opposition to some design flaws, will be eliminated by the main dictator, as funny as it is.

Because in public applications you have no control over what other people may and will do to it. Sooner or later, some violent person is bound to join this share and will decide to fill it up with garbage, merely as a joke, for example, or as a result of clearly evil motivations.

The "moral" of this story is simple and so is the solution:

Implement the concept of a share/folder/collection keys, and the separate ones for the master nodes (r/w) and the r/o nodes, and distribute the master keys ONLY to the people you can trust like you can trust yourself. All others are considered to be the leachers and are not interested in contributing anything to the collection, or may not even be allowed to do so because any arbitrary changes may damage the integrity of author's intent and a purpose. Else, no matter what you do, this thing will never work. In principle. Because anyone can destroy the collection.

Yes, one can argue: well, as soon as I notice the share is being damaged I can disconnect that particular user from MY node and restore the collection, if I am smart enough to keep the original copy stashed in some other place. Except you do not even know which particular user destroyed the collection. Secondly, the collection is still destroyed on ALL the nodes, and so every node has to go through the same pain as you did and disable that node. But how does he restore the collection? He might not even have the archive flag turned on, so the entire collection may not even exist on his disk except in damaged version.

That means that he has to resync everything again, and so is everyone else, except that he can not control the collection even though he might have the master node status set on his node. Because the source of the collection is on some other (originating) node and only that particular node may restore the original version of the collection, and even then, all the nodes have to resync again, from the start. Can you imagine what happens to large collections having humongous number of files and total size of terabytes, as some people report their collections?

Average Joe and public distribution of information

Following is the arguments of one of the Syncthing/Pules developers regarding the need for public shares/folders/collections. Basically, his argument is simple: Average Joe does not need this because he only wants to sync his pictures and "does not care" about anything else, just as his mind was programmed...

Basically, the whole argumentation looks like a "twilight zone" talk.

How is Pulse different to BitTorrent Sync?

This post was made on syncthing/pulse forums.

Choice of programming language and GUI design

This is a little technical, so, people without technical background and programming experience may skip this chapter or its parts even though a quick discussion on program user interface may be helpful in understanding some general issues.

Anyways, the main argument of main Syncthing developer to use Go is that it is highly portable language, which is great, because it allows for support of wide variety of platforms. But its GUI aspects are basically browser based and are quite primitive compared to natively designed GUI.

And GUI design and its power and flexibility could be easily underestimated. Basically, a native GUI design, meaning it is compiled to run as a binary executable code vs. web based design differ drastically in potential power. In native design you can have the features of sophisticated manipulation of massive amounts of data. For example, you have a huge list of files that need to be displayed in GUI in some box or a list. With native design, manipulating such a list is relatively easy and natural. Take for example the multiple item selection and then deletion or dragging and dropping aspects. Web browsers are not designed for these things.

In many cases, you have some dynamic language underneath, such as Javascript or PHP. But with all due respect, those are not as powerful as native code, neither in terms of performance, nor in terms of power and flexibility. If you observe, any time you go see some dynamic web page the response to user actions is sluggish in most cases. You push some button and you have to wait, and not necessarily because of the network speed.

Another aspect of GUI is packing as much useful information on the screen or a dialog box as possible and being able to drag and drop things even to some other applications. With web based GUI such things are practically impossible.

Basically, we could go into GUI aspects deeper, but no matter how you cut it, browser based GUI is simply not something natural to the browser, and for quite a few reasons.

Therefore, the native GUI, as provided by the native mode or inherently portable languages, such as Java, is far more superior, far more flexible and allows one to pack much more of significant information into the smallest possible screen and window size.

One of the most important characteristics and properties of GUI is to pack as much of most significant and useful information per square inch of the screen, as possible, which isn't quite the case with browser based GUI, even though one might argue that with more extensive and more precise use of the CSS styles you should be able to pack things to the same degree as with native GUI design. But this is mostly a wishful thinking.

The better choice, in our opinion, would be using Java language, which is also about as portable as it gets. It is INHERENTLY portable, because it uses the JVM (Java Virtual Machine). One other plus for Java is that it uses the familiar C/C++ syntax and could be easily read and understood by most programmers.

So, why turn everything upside down? What does it REALLY "buy you", in principle? Why do I have to stand upside down just to be able to read this thing? In my personal opinion, a well designed language and a well designed program should read like a morning paper with a cup of coffee and not like the grand unifying equation of creation of the Universe. Why create the stress and strain? Is it REALLY necessary? For what?

Java is a well designed language and you don't have to stand upside down to read it. So, if there was some automated conversion tool that could convert at least the bulk of the code from Go to Java, that would give Syncthing a significant boost and create plenty of interest for other developers to join the project. Several people expressed pretty much the same opinion as far as Go language goes. But this is too technical to discuss here.

Interface to the 3rd party GUI subsystem and extending the functionality

What is highly desirable at this point as far as synthing/pulse goes is providing the full functionality access to the core of the program via interface to external 3rd party GUI subsystem.

Basically, the way it is designed right now, the GUI, even though it looks quite clean and functional, still does not show all the dynamics of transfers in the ways the users like to see it. One of examples of that might be BTSync. In versions before 1.4.nn GUI had a Transfer tab which shows you exactly which files are being transfered currently and to/from which nodes and their current transfer speed. So, just switching to this screen in a single mouse click you see exactly what is happening.

Also, a browser based GUI subsystems are not really powerful or effective. Because web browsers are not designed to behave like a well designed native GUI subsystem. Such things, for example, as drag and drop, dialog box resizing, multiple item selection in some list box and some other things are not quite possible in a browser.

Another unpleasant aspect of a browser based GUI is that it does not allow you to size and move the dialog boxes and pack as much useful information in each of them as, for example, you can see in the O/S file browsers. Moreover, it might contain some visual elements of the browser window, such as button and command bars and so on and those things occupy the extra space on the screen and are neither necessary, nor desirable in the context of some GUI screen.

Each pixel of the screen needs to be used effectively in order to pack as much of useful information per square inch of the screen, as possible. This is particularly applicable to all sorts of mobile devices with limited size screens.

This means that if you provide a fully functional GUI interface to the program core, you can hang on it your own GUI subsystem, going as far as developing a native mode version that could or might be O/S dependent.

For example, you can design the sophisticated and high performance GUI subsystem in Java, which will provide you the support for most platforms out there since Java runs on the top of their own virtual machine (JVM). And it is relatively easy to write the Java code for the GUI and it can be done in a matter of days.

So, if you have such an interface and it supports the async mode of operation via event notification messages, then your GUI becomes responsive and it does not consume much of the CPU resources. Plus it does anything the way you want it, and not merely provide you the basic functionality of a browser based design.

High performance program core interface to the external programs or program controlling mechanisms.

Also, providing a well designed, language independent, interface to the core of the program will allow to control it and via 3rd party mechanisms. This could be done for all sorts of reasons, starting from adding the new functionality, not available as is, performing a much more informative logging facility which allows to see much more details as to dynamic aspects of operation, such as a progress of file transfers, and for other reasons.

This interface needs to be able to support the asynchronous operations and information exchange between the core of the program and the external controlling programs or functionality extensions.

Ego politics

The ugly part of the "good, bad and the ugly"

Well, the following section might be suitable to those who like the excitement of a battle and have courage to look at the raw reality of it, without any "nice" embellishments of the phony world.

Basically, what you will find in this section is how all sorts of arrogance, perversions and denials of some of the key issues with Syncthing/Pulse have been accumulated until it finally literally blew up in utter insanity of "suspending" the messenger from the forums for 10 YEARS, if you can even imagine such a perversion and the utmost blood-boiling hate.

And suspending for what specifically, one might ask? Well, if you have guts, then you might be able to figure it out and make up your mind by reading this section, even though it is not for the faint-hearted. Basically, "suspending" for things that have absolutely nothing to do with either Syncthing, or sync software as such, or even with things that have anything to do even remotely with software and technology as such.

WHAT? How could that be? - Well, who knows, ANYTHING is possible in this world some say. See if you can figure it out if you have any interest in intense impressions.

Syncthing problems and issues

Eventually we were banned ("suspended") on Syncthing forums, and for the most bizarre reasons, having nothing to do with either Syncthing/pulse project, or with being "off topic", or doing anything "bad" or "illegal" whatsoever, and for 10 YEARS, if you can imagine such an insanity. Here's some explanation of the that ban and its possible implications from the standpoint of this takeover of Syncthing project by the ind'ie operation, which seem to lead to Amazon:

Mutilate, delete, deny, kill and ban all those who disagree with your self-image of grandness!!!

It is nauseating to even talk about that ban, even though it is profoundly significant event in itself, even for the entire project, and not to us personally. But it has to be done, so that the general public and other developers have a chance to take a hard look at what is happening to the project and think well before they decide to continue to contribute their energy and efforts to it. To us, this foolish ban means nothing more than this project will no longer be supported by the forces beyond visible and no further ideas are likely to be forthcoming to take it to the next level, even though we will try to outline the most significant ideas here, in this document.

Because that ban indicates that the main developer is essentially a totalitarian dictator with hyper-inflated ego, who only understands the principles of domination, which eventually and inevitably leads to violence and elimination of the "undesirables". Because in HIS twisted mind, he is the only one that "counts". It is like because of HIM the Sun is rising in the morning, if you can even conceive of such a thing.

That means that this individual will not tolerate anything that does not align with his own ideas and his self-image of grandness, and will go as far as to mutilate, delete, deny, kill and ban. ALL the dictators do. In fact, from our very first message on the Syncthing forums regarding some ideas to implement the public share functionality his response was an ugly form of intellectual insult and an attack on the "messenger". This was quite a strange response in the context of purely technical issues that could be dealt with by simply stating ones own position, point by point, instead of resorting to the level of outright insults.

But who controls the Syncthing/Pulse project now?

That means that this project might be in jeopardy as it stands right now. In particular, this "deal" of converting Syncthing to "Pulse" essentially means that the whole project has been taken over by the commercial interests.

This is what WHOIS records show for this ind.ie operation, Noteworthy is the Amazon.com.

Dates 	Created on 2007-04-02 - Expires on 2015-04-02 	
  
IP Address 	54.230.68.76 - 14 other sites hosted on this server 	
  
IP Location 	United States - Virginia - Ashburn - Amazon.com Inc.
ASN 	United States AS16509 AMAZON-02 - Amazon.com, Inc.,US (registered May 04, 2000)
Whois History 	114 records have been archived since 2006-10-03 	
  
Whois Server 	whois.domainregistry.ie
Website
Website Title 	  Kieran Masterton 	
  
Response Code 	200
SEO Score 	86% 	
Terms 	211 (Unique: 142, Linked: 150)
Images 	7 (Alt tags missing: 6)
Links 	102   (Internal: 74, Outbound: 27)
Whois Record ( last updated on 2014-10-28 )
domain:       ind.ie
descr:        Indietech Domains Limited
descr:        Body Corporate (Ltd,PLC,Company)
descr:        Corporate Name
admin-c:      ADD376-IEDR
tech-c:       AAM456-IEDR
registration: 02-April-2007
renewal:      02-April-2015
holder-type:  Billable
wipo-status:  N
ren-status:   Active
in-zone:      1
nserver:      ns-526.awsdns-01.net  
nserver:      ns-1987.awsdns-56.co.uk  
nserver:      ns-258.awsdns-32.com  
nserver:      ns-1234.awsdns-26.org  
source:       IEDR

person:       Aral Balkan
nic-hdl:      ADD376-IEDR
source:       IEDR

person:       Blacknight.ie Hostmaster
nic-hdl:      AAM456-IEDR
source:       IEDR

Note: well, we just saw one message from ind'ie guy on syncthing/pulse forums which says:

Well, may be, may be... But why not just use some VPS solution for $20/mo. and install anything you want on it? Does $20 represent such an issue for the "sponsors" that promise all sorts of "support" to this project, that they are compelled to use the companies that are not known as the hosting or VPS providers?

What is the benefit of using the Amazon or even coming close to them? What business they are in? - Hosting? Or buy/sell anything under the sun?

For one thing, accessing the syncthing forums now shows us the spinning wheel in a browser nearly half the time. What kind of "improvement" is it and of what?

Change of license issue

Now, the first indication of the effects of this takeover of Syncthing project was the change of license from MIT to GPL-2. For what specific reasons was it done? That change has some profound consequences. See the discussion here:

https://pulse-forum.ind.ie/t/license-question/1066
(Support forum: License question thread)

GPL is probably one of the most convoluted and complex licenses one can find and one particular person, who claims to have spent years on dealing with various licenses warns "if anything, stay away from GPL like from a plague" (not a literal quote).

Secondly, if you are a honest man and are not merely an ego maniac or a totalitarian dictator, you don't do things like these without any preliminary discussions with other developers or even with users, bouncing things back and forth. Because things like license conversion have some profound effects on the entire project and all the developers that contributed their time and energy to it. Simply doing it silently and putting every other developer before the fact that the massive code changes related to license conversion AFTER it has been already submitted, could be interpreted as what kind of thing and according to what kind of motivation and intent?

Well, to put it bluntly, it was a spit in everyone's face and a "screw you all" attitude of some dictator-in-chief, or an arrogant fool, at best, that has no respect to anyone else. No honest, half-decent individual would be able to even conceive such a thing, most likely. Because it is UTTER disrespect to all other developers and even the users that have given this project their trust and might have even involved some other people and started doing things that depend on it in the understanding that this is truly an open source and non-commercial project.

One has to be profoundly sick in his very soul to even conceive doing such a thing in the way it was done. Because it is well beyond any reason. Basically, it is a mental department level, just to be blunt.

Furthermore, as it was "explained", this license conversion was done without explicit agreement with other developers, or even a discussion of it, since no one "objected", which is a perversion. Yes, those who are currently actively involved did not vocally object, and it could be because they know that they are dealing with a dictator and in his language and understanding objecting to anything means you are considered from then on as "an enemy of the people" with all the consequences thereof. This is a standard trick of all the dictators.

But those developers that are not currently active on the project or could be temporarily busy at the moment with other things in their lives, did not even have a chance to know or realize what this change of license means for the project and all the work they have done for it, and it does mean quite a few things. Furthermore, all the developers, currently contributing to this project, may not fully comprehend the real motives for this license change.

Because the issues with licensing are some of the most complex there are, and to fully understand all their implications or hidden meanings and/or motives you'd have to be a pretty competent copyright lawyer, and the copyright law is one of the most complex areas of the law according to the competent lawyers.

I am the king, screw everyone else

Then, the main developer begins to behave like a CEO of some sort. He is invited to the meetings with these "sponsors" that promise the project "support" and all sorts of "bright future" to butter his hypersized ego and to make him feel important and grand in the scheme of things. And he does not even ask the other developers whether they all agree that he should go there or not.

Moreover, what is the need to be physically present and travel some place? ANY kind of information, questions or issues could be simply discussed, even in the conference setting, directly on line. So... What was the REAL reason to physically travel anywhere?

Nor does he tell them the purpose of those meetings and what is intended to be discussed there and asks other developers of their opinion on whether they are even interested in such discussions.

Nor does he report the results of it when he comes back, and so no one knows what was actually discussed there and what was agreed upon behind the curtains.

Essentially, he behaves like a "king" of some kind and does not seem to even consider that the opinions of others matter in the scheme of things. Because the only one that counts is he himself.

Yes, if this was his own private project and no other developers have given it some of their energy and life in the understanding that they are doing something for the common good, then it is another matter. "My toy, I play with it in any way I please".

But what he did instead is to open up a project as an open source, hung an MIT license on it, which would be acceptable to anyone, more or less. Then people stated joining and contributing to it. And then - boom. It turns out they are just "nobodies" and mean nothing. At best, they can fart or even piss against the wind, if they don't like something, and that is the extent of their significance "in the scheme of things". WHAT?

What this all means is that developers might have some naive expectations and understanding that they are still working on an open source project, whereas they are working on/for something entirely different from now on, and the sorriest thing about it is that they do not really know what is it.

He is probably given a grand reception, like a king of some kind, invited to some meetings with the "important" people, and it is whispered into his ear how "important" the particular individuals are. He is probably placed in a very nice hotel and could be even driven on a limo, just to butter his ego and to buy his corrupt soul of an ego maniac.

"Bright future" for the project - hell

During the discussions of this "conversion" on Syncthing forums it was quite something to see all the praises and promises of the "bright future" offered by the new "sponsors". It looked like some kind of Herbalife or Scientology promotion rap, full of the same kind of hype and the same sleazy ego buttressing templates and techniques.

Furthermore, these "sponsors" and their agent, giving a rap about the "bright future" for syncthing and investments of resources into this project, did not really disclose the REASONS why are they so interested in this syncthing thing. When we looked at their site the last time around, they had some beep, beep, beep type of projects and were offering to build the "communities" that can exchange files, pictures and other things. But, according to the main developer, the program is not designed for that.

If any user can configure his node as r/w (master), then the whole "community" will pretty quickly turn into a madhouse, "ruled" by chaos, as various people begin to add/delete/modify whatever they think belongs to them and their own collection, but, actually, mirrors what everybody else has or does, and everybody can mess with the stuff of others. It is like: you look at the screen the next time and notice that all your pictures or even videos are gone, and now you have tons of other crap, that magically appeared, which you don't even want to see. So, what do you do then but to delete it all, which will delete it all on all other nodes. And some other people might start adding it all back because they happen to like that particular zombie stuff. How long will it take before all of you will simply go mad, not even realizing what is going on?

Well, some say: "do not try to make something grand out of something that may turn out to be mere stupidity". But some also say: "better be safe, than sorry". Yes, to look for some sanity in a madhouse could be not something beneficial to one's own health and well-being.

But we choose to play this game to its fullest potential, until it exhausts itself like a balloon. All you need to do with a balloon "to see its nature" is to pinch it with a small needle. So, join the trip if you wish. Else, go find something "better" to do with your life and time.

Who are these ind'ie clowns promising a "paradise" and what does Amazon have to do with it?

And who are these ind'ie clowns? Where did they come from? Where do they get the money to "sponsor" this thing and what kind of financial benefit do they expect to gain as a result of taking over this syncthing? What was the purpose for that 4 day trip of main developer to visit these clowns and who paid the expenses for it? If they are mere nobodies, then who is going to throw away thousands of dollars on such trips? And what was the need to see the main developer in person? All the issues can be discussed in all sorts of ways on line. About the only reasons there is a need to meet in person is either to sign some papers of impress the ego maniac in order to buy his soul cheaper. What other alternatives are there?

The whole thing looks like a puppet theater and nothing more than that. But the REAL motives are not disclosed or not even discussed in a way that makes any sense. About ALL you hear is grand level hype and promises of some "bright future".

What seems to be strange here is that it is not clear for what exact purposes the Amazon, if it is them who stands behind all this, is interested in this project. Amazon is in the business of public distribution of products and public distribution requires the public functionality in Syncthing, if that is what they intend to use it for. But, even after the meeting with the ind'ie "sponsors", the main developer still refuses to even consider the public share and have bluntly stated "I do not need it and I am not going to implement it". Like it is his private toy and not an open source project to be used by all sorts of people with all sorts of needs and applications.

So, what would the Amazon need this thing for if it does not implement the public share functionality and, moreover, the main developer flatly denies to even consider such functionality? Do they intend to use it exclusively for the internal corporate network and do not even consider it as a tool for public distribution of information?

So, there are plenty of questions and warning signs of all kinds related to this "conversion" and you need to investigate the issues for yourself and in the in-depth manner and come to your own conclusions. Because we can only guess here with the information that is publicly available at this junction (as of October 29, 2014).

Now... As a result of this utterly foolish, utterly perverted and evil-intended ban, which does not help anything whatsoever as far as this project goes, we can not contribute anything to the project via forums. Nor we can express our opinion or view on some matters and the issues. But, nevertheless, the issues still remain, just as Truth itself. So, we will cover some of it here for everyone to have a chance to look at the whole thing from a different perspective, and especially those developers, that might consider joining this project, to be forewarned of some pretty nasty effects of some of design flaws and product limitations, shortcomings or some general project issues.

Time to start thinking?

Now, considering the utter stubbornness of the developer "in chief" as to implementation of a concept of a public share, which is a must in our opinion, and in combination with some issues as to the very nature of this project takeover, it might be not such a bad idea for the developers interested in the progress of this project in genuinely open source spirit to start considering the alternative paths to further development, since from now on, whatever they contribute may not bring the results they hoped for when they joined the project.

And "the bottom line" is that either this project grows as truly an open source and non-commercialized and not controlled by some individuals with their hyper-inflated egos, dominating over it, or some other parties, or it will fade away into the lands of obscurity, where it essentially belongs from now on.

In our opinion, either the program supports the public information sharing functionality, or it has no future, at least from the standpoint of information distribution to the general public. Simple as that.

Because the information distribution needs of the modern world are orders of magnitude more significant than any individual needs to sync one's laptop to the mobile phone and his desktop and a server, or a chip in his shoe. That is all nothing more than "spinning the wheels getting nowhere real fast", or, in other words, the "karmic convulsions" of the unconscious and unaware.

Because the world is going through the most profound changes and every moment now counts more than centuries in the past, and everyone will have a chance to see it in quite graphical terms. It is just a matter of time, maturity and willingness to truly investigate the issues.

Witch hunting:
Mutilate, delete, deny, kill and ban all those who disagree with your self-image of grandness!!!

And here is the kind of things these cunning cowards aka dictators do:

Ban for 10 YEARS? For doing what kind of thing specifically?

Disagree with dictator? - Guilty of "hate speech"!!!

Let us start looking at it starting with probably the most significant "reason", that might have some far reaching consequence: "author of website of hate speech".

Well, WHICH "hate speech" site is he talking about? HOW did he learn of such a site if we never published any site links to ANY site that even remotely has to do with any kind of "hate speech"?

And if you look at the very definition of "hate speech", it is quite something indeed. First of all, there is a notion of a "protected group" and "hate speech" is saying something that will create problems of all kinds to those "protected groups". So, some "groups" are EXPLICITLY protected via this perverted trick. But others are not. Why is that? Do we live in a world of "democracy"? Why certain "groups" are "protected" better than others? What is democracy then? And if you look at the list of those "protected groups", it might blow your mind. You will find sexual perverts among them, various Nazi kind conspiracies, hiding behind the backs of plain ordinary people based, for example, on nationality and so on.

But if you are a mere "mortal", then you are not offered the same "protection". Only the "elite" of some sort is protected by it. Then the question arises: but what is fascism in that case? Isn't the separation on "special" and not so "special" classes of people is in fact the very root of fascism and Nazism in particular? Why some groups are protected and others are not?

Basically, this "hate speech" trick is pretty much the same trick as to classify someone as "extremist" or even a "terrorist". There is a system of zombification of human consciousness utilizing the so called "slide response" technique developed by the CIA. The essence of it is really simple: you just repeat the same lies ad nausea and label them with some negatively charged label. The more you repeat, the better. Adolph Hitler was told this trick by his psychologist.

So, the time comes, that when some of such labels are merely mentioned, the mind simply "slides" to the next subject, agreeing with the zombie program in ones mind. For example, you see a sentence starting with the words "terrorist" or "extremist". What are you going to do but skip it, subconsciously accepting and agreeing with whatever garbage will follow that label? Or are you going to waste your life reading about some "bad" guys, like "extremists" and "terrorists", again and again and again? How many times are you going to read the same crap that tells you the same thing, and that is that "extremism" is "bad for you"? Or, are they going to tell you something different in that sentence or even an article?

The same thing is with this "hate speech" concoction. Once someone says: "an AUTHOR of a hate speech site", who is going to object or even bother to find out what does he mean and how did he come to such a conclusion and based on what arguments or facts?

Because "hate speech" = "evil". PERIOD. Everyone "knows" it, don't they? WHO is going to even consider the implications? Interestingly enough, these tricks are invented and propagated by the most evil ones there are, the criminals against the mankind, who, like George Bush, Sr. said: "should be chased down the street and hung on the nearest light pole".

And how would this pathetic dictator know who is the author of the entire site? There are no authorship claims on any of our sites, at least for any works that do not explicitly identify the author. With all other works that do not mention the author specifically, the authorship can not be determined, unless it is claimed. And there are no such claims on any of our sites.

And why did he use precisely the "AUTHOR" label? Well, because in THAT case there can be no doubt that this is HIM who is "evil". Otherwise, if you say "the one who runs the site" or "serving the site", then it isn't as convincing, and many people may simply laugh at such pathetic accusations. But the "AUTHOR"? Well, it is "serious business" now, even though he did not explicitly state "the author of hate speech", published on the "hate speech" site.

Well, thanks for THAT much. We really "appreciate" your style and foresight. Otherwise you could be sued, couldn't you? But even saying the "author of a hate speech site" is by itself deserving to be sued for. Because it is derogatory in the most profound sense of the word, at the very least. In its very essence, this is the very nature of defamation and slander.

Now... Yes, understood, if he would state that some site has some false information then, who knows, may be he does have some point. But then the question arises: what does ANY site in the world have to do with the Syncthing/Pulse forums? Is this a court decision? Was it determined as such by any competent specialists in a proper legal proceeding, as required by the Universal Declaration of Basic Human Rights in articles 10, 11, 18, 19 and 30?

And what is "hate speech" in essence? What are the EXACT examples of such a speech on any sites we have?

Basically, to even conceive of carrying the "hate speech" on your site is nothing more than lunacy. Because mere hate does not resolve any issues, does not solve any problems and does not clarify anything. Hate is just that, hate.

First of all, the sites we run are some of the most prestigious sites in the world, just to be blunt. And they are visited on a regular basis by thousands of ALL sorts of people every single day, including the organizations, educational institutions, local and federal governments, military and even the intelligence agencies, as strange as it might look, not even mentioning the world's foremost researchers and some of the most well known writers and bloggers of the world fame caliber. And the kicker is that they are visited on a regular basis even by the employees of the major search engines worldwide, and for those guys to visit some site, that site must be "way up there", in the sky, as far as quality of information goes.

Basically, our sites are considered by many to be the REFERENCE grade sites, and they contain the works of the outstanding and world famous authors, researchers, historians, scientists and so on, and there are so many tens of thousands of links and all sorts of praises to our sites that it would be nearly impossible to even begin to enumerate them all, even though we could, if we had nothing better to do with our time but to brag about how "great" we are. What does it matter to others or to the Truth of it?

Nearly everyone thinks he is "great", this way or that, doesn't really change anything of substance. Telling someone "I am great" would, most likely, be taken as an offense and a reason for some kind of fight, just to "prove" that MY ego is "better" or "greater" then yours, even though any ego stinks in pretty much the same way and merely offends others. What is so new or "special" about your "greatness" and what does it worth or "buy" you beyond pleasing your pathetic egos, forever craving for attention as a result of the inferiority complex, hidden in the depths of your being, well below the surface of your mask of some "persona grande"?

Secondly, these sites stand at the very top positions of the search engines worldwide, and to get into the top slots on Google, for example, you have to be quite a site. People are willing to pay some serious money to get into the top slots, and we did not pay a penny to anyone. Because Google search engine has over 500 THOUSANDS rules and variables that analyze your pages unlike you could even dream about, including the AI technology to actually analyze your content for meaning and significance, even though the very idea of an automated content analysis isn't really worth much to begin with.

Thirdly, these sites are some of the oldest sites on the web. Not exactly the very oldest, but old enough to be considered the stable and reliable sites to be used as a reference. And we have never received ANY kinds of complaints from anyone regarding anything, with one exception of some cunning dictator coming to tell us that we should not refer to one of our sites from our Osho site.

Well, we did cover his bitching, and on such a level of detail and precision that it became a book, and that book is regularly read in numbers and it became one of the best books on those specific subjects, and it is being read on a regular basis even after 5 years since its original publication.

Since our public answer to his totalitarian bitching, we have not heard from him ever since in the last 5 years. Because he was disgraced to the degree that he probably lost his "client base" in his horseshit peddling business of a "group leader" in the "spiritual growth" business, zombifying the minds of his victims with the ideas that he KNOWS something they do not know, which is not even pathetic, but simply laughable. What a fool? To waste ones own life on trying "to make a buck" on people's confusion as to their own worth, dignity and creative abilities and potential.

And in fact, we are grateful to him for his bitching in totalitarian style. Because it gave us a chance to cover not only the concoctions he brought up, but the entire "spiritual business", and to go into it on a much deeper level and expose his entire utterly dishonest manipulation of the facts in order to achieve his rotten agenda of a plain, ordinary conman with totalitarian tendencies, a parasite essentially, sucking the life energy of blind people and leading them astray, and not towards some kind of "enlightenment", as he claims. As it was said: "Blind, leading the blind, shalt fall into a ditch". Simple and clear as a bell.

Actually, this ban is pretty similar to that guy's accusations and fabrications. The same kind of people, doing the same kind of things.

So, which "hate speech" site this pathetic dictator is talking about?

But the "thousand dollar question" is: but what does ANY site in the world have to do with the Syncthing project, especially if it does not even mention it? Yes, sure, if he would claim this "hate speech" argument on his FORUMS, then it is another matter. But then he would have to provide the specific evidence of "hate speech" and explain the mechanism and principles that allowed him to come to such a conclusion. Otherwise, how could such an ugly accusation stand and what does it mean in reality and in relationship to this project?

The "pattern of negativity" trick

As to the "pattern of negativity" argument, yes, indeed, that blanket and a highly negative notion is definitely a nice trick in itself. To control and dominate others, putting up the mask of "positivity", which is as false, as a three dollar bill. It makes one SHRINK in guilt! Works like a champ. But not in our department. We shred this kind of stuff to dust, to fine powder, every bit of it.

But there is a "but" here, at least with this particular dictator, aka "developer in-chief". The thing is that if we post nearly every single one of his own responses to purely technical ideas, proposals or pointing out the shortcomings, that were not meant to do any harm to the project in any way, shape or form, and in fact, just the other way around, were meant as a help and a contribution to it, then he should be simply "chased down the street and hung on the nearest light pole", just like George Bush, Sr. said:

Nearly every single response of this dictator to purely technical proposals was utter denials and the most poisonous intellectual insults of despicable grade, camouflaged in some candy wrapper of "grandness" and "know it all" attitude. And the most significant and critical issues were simply ignored or denied in bulk without any valid argument or consideration of its meaning or significance, which in itself is an insult to a senior developer. Because it implies that the arguments are so dumb and incompetent, that they do not deserve even a response! Here is just one example:

Virtues and blessing of "revolutionary" Go language

First of all, what is noteworthy here is that he did not use a SINGLE "bad" word in his highly offensive response. Everything looks perfectly "reasonable", at least on the surface. But deep down, it is such a poisonous hate, full of such profoundly negative connotation, that one could hardly come up with something more disgusting than this arrogant, self-boasting concoction of utterly unsubstantiated crap, and of the LOWEST grade at that.

First of all, there isn't and can not possibly be even a shred of anything "revolutionary" in sync as technology, syncthing or BTSync or anything else. Nearly the whole thing is intuitively obvious, and there is simply no need for any miraculous mechanisms for it to work. Any more or less experienced developer has seen the permutations of what it does many times over in his experience. And he knew at the moment of that post that he is dealing with an experience developer.

Why not reinvent the wheel?

What he is saying here is "if you are so smart then write your own program". Well, fair enough. But the thing is that there is no need to reinvent the wheel and start writing something from scratch when it was already written and is available.

How much of your life are you willing to waste spinning wheels for nothing, just to achieve pretty much the same result, as has been already achieved in some open source project? For what? The base is ALREADY there. All you need it to make sure it all reconciles and has powerful and flexible enough architecture and is stable, has a predictable behavior and assures the consistency of data as far as sync operation goes, and has a fair level of performance. That is ALL you need with this particular project.

Yes, this is the key to understand all this: "all the critical code is already there in open source version." What are you wasting your creative energy on, even though, certainly, you are free to spend it on ANYTHING you like? Reinventing the perfectly working wheel?

And what did you reinvent that makes it even half as good as what BTSync has or does? Actually, BTSync has one major and the most significant problem: it is not an open source. All the mountains of bugs and problems could have been fixed a long time ago, if it was an open source project, and the program would probably be much more powerful than it is right now, if you could have the external development resources of the open source community contributing to the project.

But they want to dominate the "indie promotion distribution market", just as this Pulse thing is trying to do as of now. Strangely enough, they seem to be after exactly the same market as BTSync. It was pretty interesting to see. Just do a search on http://ind.ie and you'll get the idea, unless they have removed some information which we have seen on their sites.

Now, could HIS own response be interpreted as some kind of a "positive" or constructive opinion or a feedback on quite a reasonable opinion of the language?

Portability argument

The thing about Go is that one of the MAIN arguments of this particular developer-in-chief, when asked why did he choose Go, was that it is so portable that you don't even have a "single dependency".

But that argument does not work with Java. Because Java is probably the most portable thing there is. Simply because it operates on the top of JVM (Java Virtual Machine) and it runs on so many platforms that they claim that it currently runs on 3 BILLION devices, down to your fancy super duper bread toaster, not even talking about your "smart" refrigerator.

Secondly, does one have a "RIGHT" to like one language over another? Or, does he think that unless you write in Go, you are "outdated" or "out of fashion"? But does not he know that fashions come and go, like a wind and rain? Is it "OK" for me to like the vanilla ice cream and not to like coca cola?

Dictators know better... Always!

And what kind of an ugly concoction is "I haven't actually looked at how syncthing works or what it does, but here's what I think you should do instead" Yes, "shuda wuda" is certainly one of the favorite words of dictators of all kinds and this one is not an exception.

But this is nothing more than a fabrication, and an ugly and highly insulting one at that. Like there is some miraculous mechanism in syncthing that does things no developer has seen in his life.

Basically, we could present the overview of the entire sync process, regardless of whether it is done in Syncthing or anything else, without even looking at the source code. Because all the basic mechanisms and subsystems will eventually resort to doing some very specific things, that are logically implied by the mechanisms and higher level logic design. There isn't anything "special" in Syncthing, unless the author simply went "nuts" making things more complicated than necessary.

Sure, any program may utilize its own tricks to do the same things that other programs do, only slightly differently. But that does not invalidate the entire logical operations of the high level logic.

And yes, there are some quite tricky issues specific to sync operation as such. Most of it has to do with node update notifications or requests, maintenance of the modified file lists and ways to verify the correctness and originality of a file version, ways to inquire or inform about file updates, block locking in case multiple nodes have the same block that some node needs updating, node discovery, various hashing and encryption algorithms and methods and things like that.

Yes, there are plenty of things to be done. No arguments about that. But NONE of it is something so esoteric that a fair level developer will not be able to conceive or comprehend it even without looking at the source code, unless one deals with some specific and tricky issue that could be done in different ways to either increase performance, efficiency, reliability the soundness of the very model and so on.

Moreover, the question arises: how could such a great developer as our developer-in-chief even conceive of designing the sync program with vital design flaws that allow any user to obliterate the whole share/collection, not even having a bad intent, but merely not realizing the consequences of some actions on the files he intends to perform, just as we see in Syncthing? It seems to be so basic, that it would be interesting to know what pushed him to allow for such a fundamental oversight as to crush the entire share? Such things should not have been allowed under ANY circumstances, no matter what is the intent or a purpose. How could it be allowed for the user to damage the share without even realizing what he is in fact doing?

We did propose quite a number of things to improve the program, its functionality, use, logging and dynamics reporting abilities and performance. But nearly all of it was simply flatly denied, and in the most insulting and arrogant manner, with one exception - a suggestion to comment the source code, which has already started to happen, at least with the latest updates to the sources. So, we did achieve at least some progress with this thing.

And that thing alone, when and if is fully completed, will make this project much more manageable and will help any other developers to understand the program workings and things to watch out for, and probably unlike anything else. So, for new developers that decide to join the project this would be of great help in clearing up some misunderstanding of the program operation and internals.

So, instead of being grateful, this arrogant, self-boasting fool starts barking at us, denying nearly every single point, idea or proposal we have brought up. For what? What could it possibly achieve at the end? To show to the whole world that he is "smarter" than us? OK, fine. But will it help the PROJECT? In WHICH way?

Does he at least have the slightest clue about who is he barking at with all his futile and childish "arguments", if you can even call them such? Because one can hardly find a single real argument in his wholesale insults addressing the issues in ANY of his replies. Nearly all of them essentially say: there is no such a problem, screw you, or these issues are so insignificant that there isn't even a thing to even look at or consider as something real, something that needs to be addressed.

One other fool from his team even went as far as to ask for an example of "use cases" for the public sync applications! Question arises: does he actually have a functioning brain? How could one even conceive such a question? It does not take more than a few seconds to come up with a pretty fat list of public applications. The whole net is public applications for the most part. Interestingly enough, he asked this question even after we posted a number of the most detailed articles addressing all sorts of issues and aspects of the whole thing.

Java could not be possibly a better choice because it is heresy!

And where did it say you "SHOULD" do this or that? The post merely said:

"Actually, Java is probably a better choice", and the reasons were provided: "from the standpoint of code clarity, synchronization and portability issues". Is it "OK" to have ones own opinion? Are these arguments valid, by any chance, or is it all merely a bluff? Can you PROVE it, Mr. Smart?

Does he at least know Java to any degree of competence to make a comparative judgment? At least we did look at the Go main documents that explain what it is and cover some of these issues and its design philosophy. Well, like anything else, there are some points that seem significant, at least from the preliminary look. Yes, certainly, we are not the experts on Go, by ANY means, and do not have the slightest desire to become such. Simply because we have an urge to puke when we are looking at that code. It is simply nauseating, so upside down and so childish some things look to us.

Go language revolution - turn everything upside down!

What these language designers seem to do is to begin to "revolutionize" that, which is already accepted and has a tremendous base of developers behind it. And if you look at some of those "revolutionary" language design ideas, about the first thing that jumps at you is that, yes, there IS something in it, but why should I scramble my entire knowledge base, which works fine and has been validated with time, to follow yet another way of doing the same essential thing?

What is the need to turn everything upside down and create all sorts of notions that fundamentally do not "revolutionize" anything of substance?

But hey, understood, to YOU it may look like some kind of Bible to be worshiped or make you feel like a "superman" of some kind, who is capable of comprehending a new "breakthrough" in languages that will rid the world of all evil, and that is how it looks like.

But if you look at it on a deeper than skin-deep level, then what you might find is that, yes, some things do make sense. Except the same exact results could be achieved in other languages by merely writing a wrapper or a library that does exactly the same thing, if it "buys" you anything to begin with. Why does one need to waste energy on learning something that does not change anything in any fundamental way?

There isn't a shred of anything radically new or "revolutionary" in all this Go trip. The same thing happened with Lisp, Forth and even Python or Ruby at some point when they just came out. And in fact, any of the above languages have many more "revolutionary" features or concepts than this entire Go trip, at least as far as doing things in utterly different ways. Just Forth alone, or even Lisp is quite a trip in itself. And even Python was considered at some point as something that will make the life of a programmer much easier. But, for some strange reasons, none of those languages were able to gain the wide popularity and support, even though some are using them even to this day. You won't find more than a few full-fledged applications written in them. But what did it change fundamentally?

My language - RULES! And yours - sucks!!!

The same thing happened with Java a while back. It was such a hype that at some point nearly all the bookstore shelves were filled with Java this and Java that books, like you can not exist without Java. Where have all those books gone now? Fair level Java programmers began to behave like some kind of "gods" or "super-programmers", and would talk to others like to "infidels" and "inferior" beings, exactly as we see it here with this Go trip. The same exact crap, only the labels have changed.

And his post was not merely "the curt answer". It was an ugly insult, WELL "below the belt". One can hardly concoct anything uglier and more insulting under the circumstances. And it was the language of a bully, an arrogant dictator, by whom everyone is to measure, at least in HIS twisted mind. He was basically just plowing you out of the way or running over you like a tank. Did he, by ANY chance, expect that one would bow down to him, like before a king of some sort, and ask for his blessing?

Yes, we hold our opinion: Go is profoundly ugly as a language. There are several major violations of accepted and highly effective lexical principles, at the very least. The ideas of dynamic variable typing and scoping have been beaten to death for generations, and, one way or the other, you will pay for it. Because the data types are inherent, down to the level of a CPU.

You can't store a floating point number into an integer register and vice versa. Nor you can store a string into an integer register or vice versa. Sooner or later, at some level or another, there is an inherent need to do the data type conversions, and could be many more times than is necessary. All the dynamic typing, scoping and instantiation of variables does is to introduce the processing overhead and brings in the performance degradation into the process, at the very least.

But... No matter what they have invented with data types, they still have them. Why, one wonders? If they are as "automatic" as they seem to claim, then just forget about the data types and pump any kind of garbage through your smart typeless pipes or "channels" for processing and let that "smart" programming language detect it in real time. Why not? Why even keep the concept of data types on a syntax level?

Yes, the idea is great, just like the ideas of some paradise or utopia. But this idea has not gained any acceptance for some strange reason. Why? If it is so "great", why would vast majority of programmers "suffer" unnecessarily dealing with data type issues when they could have an instant orgasm with dynamic typing and scoping? Are they merely stupid? Or backwards? Or do not really comprehend the benefits of some genuine "progress" in the programming languages?

And, interestingly enough, in Java they did address these very issues and did achieve some improvements, at least compared to C/C++ idiom. But they were smart enough not to go so radical. As a result, it all simply makes sense, and without much fuss about it or some radical "revolution" about data types as such. They were able to eliminate quite some nasty issues, such as, for example, the whole notion of pointers and pointer arithmetic, and you can easily see the benefits of it in significant simplification of the source code and increased clarity as to what is what and behaves in which ways under certain circumstances. In Java, you won't find any of this pointer funk or even mentioning of references with their most bizarre and utterly unintuitive behavior and syntax.

One of the very few "improvements" of Go is that they got away from the concept of generics of Java or templates in C++, or inheritance and some other "object oriented" hype, which is a great idea. Because all those ideas are mostly designed for those programmers that want to LOOK "smart" and do not deliver on the alleged "benefits" they promise. And that is precisely why plain, ordinary, "vanilla" C has at least 10 times more stable following than any other language out there.

But I'd rather do it in Python

Interestingly enough, we have seen somewhere on forums the message from one guy who said something like: well, I'd rather do it in Python. And that is really an insult to such a "great" language, if even a Python looks more appealing to some people. What is it worth then?

Otherwise, even starting with Java, they'd get away with variable types. Java is probably one of the cleanest and best designed languages out there, and for so many reasons, that it would take a pretty fat paragraph to even enumerate it all. And, interestingly enough, they did not radically redesign the language notation and syntax and tried to keep the language as close the the familiar idioms, as possible. So, reading Java isn't much more difficult than reading C++, for example.

Any programmer can understand the most of it without any effort. And they did get away from the concept of pointers and pointer arithmetic and quite a few other pretty obscure and "esoteric" aspects of C++. In fact, Go seems to have borrowed some ideas from Java in respect to the syntax and notation, except they decided to "revolutionize" the language notation, which is nothing more than bluff, creating nothing more but pain on the neck to millions of professional programmers, unless you spend months or even years getting used to living upside down.

In Go, everything is upside down compared to what the experienced developers are used to. In that respect, reading Go to them is like reading Chinese or standing upside down. For WHAT? What does it buy you, but merely confusing things even for the people that might have spent their entire lives as developers? Why do you need to invalidate the whole world and human experience that has been gradually growing and developing and was tested by time? Does it radically change something? Does it make the language more efficient? Increases code clarity? Makes your code more robust? What does it actually achieve at the end?

There is quite a bit to be said about the language syntax, except we'd have to write a few paragraphs to clarify it.

Screw accepted language notations

Now, the "punchline" in Go syntax is that they refused to use the very concept of a look-ahead token on a lexical level, and, therefore, they can not properly parse an opening curly brace in a block of code, unless it continues on the same line as the conditional statement. This is simply bizarre. First of all, it implies that half of the programmers in the world that use the left curly brace at the beginning of a line and not at the end of the line of the statement itself, will feel nausea when looking at such a code. The reason people like the left brace at the beginning of the next line is that it allows them to very quickly match the braces visually, by merely scanning the code block at the same horizontal position, and, possibly, find some otherwise hidden errors in case of braces misplacements.

But with the brace at the end of the line of the statement itself, it can not be easily noticed. Your eyes would have to jump all over the screen if you wish to have a rough idea of your braces matching correctly while looking at some block of code. And the end of statement could be on virtually any horizontal position on the screen. And this is done on a subconscious level as a "background job". So, you do not actually waste any additional energy on scanning the entire screen like a madman or a monkey. And mismatched or incorrectly placed braces may and are likely to cause the most profound logical misbehavior of your code.

The argument for it is that you know about it by the sheer fact of code indentation on the next line, is utterly invalid. Because code may be indented, but the brace is missing on the previous line and you may think that you are in a new block of code, while you are in fact are in the same old level. How do you make sure it is not the case but by explicitly looking for the left brace in addition to looking for the indentation level, doing twice as much more work than is necessary?

This "breakthrough" alone is enough to turn a half of the world's programmers away from this language and they are not likely to touch it "with a six foot pole". Simply because they have no interest to go insane, scanning their entire screen all day long.

Monkey business: Let your head spin when you look at the source code

Another "breakthrough" in Go is that they got away with a concept of an explicit end of statement like it is done in C, Java and a bunch of other languages. The end of statement is implicit, by the sheer fact that you have started a new line, which is simply bizarre. Strangely enough, they say that the end of statement semicolon is still there, but it is "automatically inserted" for the compiler!!! WHAT? Why?

Well, because, first of all, you create more work for your parser if you don't have an explicit end of statement token. Because it is not inconceivable that you may hit the wrong key in your source editor and both statements will end up on the same line, which you might not even noticed, and, in combination with the nearby comment, this might create a nightmare for your parser.

Because it won't be able to correctly report the exact location of the error since there is no explicit end of statement token, such as semicolon in many languages, at least as a programmer supplied it. So, where is the exact point of error, even in purely logical sense of the word, when you have two statements on a single line? Is it the first statement, which the compiler may not even be able to identify in some cases? Or is it in the 2nd statement, which, and likely, the compiler will not even be able to parse correctly? Because it did not even finish processing the fist statement, as a result of some other things present on the same line.

Then why do they use dots at the end of a sentence in the human languages? Well, to explicitly identify the end of a sentence, which is the same exact thing as the end of a statement in computer languages. Just remove the dots from a human language and try to read it. How long will it take before you have a grand headache?

Question arises: WHAT did this "breakthrough" in languages "BUY" you in terms of language properties, power or anything else whatsoever? Did it increase the efficiency and/or performance? Did it make the work of your parser EASIER? What did it achieve as a net result? What did it "save" you? A single key stroke explicitly identifying the end of the statement, so that, for one thing, the compiler could report the exact place of an error correctly?

What are these "great" and "breakthrough" language designers feed you but a nightmare for every single subcomponent of a compiler and to load your brain with an additional and utterly unnecessary work and make you go "mad" running around the screen with your eyes while trying to match some things or to see where exactly each statement ends? Why does your head need to spin just by looking at some code? Why does it have to be even more complicated than generations ago, while the language designers are trying hard to convince you that it is just the other way around?

Yes, in Go, everything is upside down, starting from notation, lexical analysis, syntax and even semantics. You don't have to have a PhD in CS to see that. Their "explanation" of exchanging the variable/argument names with data types is really quite some piece of horseshit, and of the royal grade. And sure, they do have some point, especially for the "great" programmers that have a tendency to write the most complicated looking code, just to show their "grandness" to everyone else.

It is like: if I can write some code that nobody will be able to understand, but myself, then those, who would have to read this pile and won't be able to understand what it means or does, are the ones to be fired from a job before me, in case there are job cuts, but not me. Well, at least according to the upside down logic. Otherwise, the whole project will collapse if there is no one left to understand my great code. This trip is called "job security", at least in the Silicon Valley.

Screw libraries!

One other thing, which is as bizarre, as it gets, is that it looks like you can not even build the libraries in Go. Just now, we have noticed a message from one guy proposing to separate the Syncthing engine itself from the rest of the code, such as GUI or the controlling logic, for example, which is an excellent idea. Because, if you have an engine library, like libtorrent or libutp, then you can hang on it any controlling mechanism and any native mode GUI you wish and it will work and behave exactly as you desire, which would be a tremendous help to the very project.

In that case, you can implement the most sophisticated higher level controlling logic which will allow Syncthing to behave even in the ways the original version was not designed for. Because most of the problems of Syncthing are purely logical. The engine, or the "worker" part does not change, at least for the most part or in some major ways. The underlying mechanisms are the same, regardless of a specific application and implementation. And THAT is what modular programming allows you to do.

But our great "developer-in-chief" says: sorry, Go is "very ill suited" for that. WHAT? Is this a joke?

It would be interesting to see someone, who claims his language is real, except you can not build libraries for/with it!

"Lord, SAVE me"!!!

Actually, when you begin to think about it, it does make some sense. Why? - Because Go and the entire concept of "net computing", pursued by Google and their threats to slaughter Microsoft, and all their promises of "trans-continental" and fully network computing makes the very concept of a library utterly unclear. What if you have a browser "plugin" that might require some library?

Because with Go you don't even know what and where you run. How do you know what you run and where in case your GUI is browser based? So, the program itself may be on one continent and the GUI is on the other one. So, on WHICH end would you say you need to use some library or a plugin?

Well, this mental exercise certainly looks quite funny and unreal, but we would not be surprised that at least a part of this argument is applicable to Go, at least to some degree.

"The bottom line" is that the arguments as to Java vs. Go are not invalid, by ANY means. And if we look at more complex issues of the language, such as synchronization, exceptions and a few others, then Go simply looks childish with its blanket ideology and methods. And if you merely want to replace one thing with another that is doing essentially the same thing, then you need to consider the "bang for the buck" issue. Just making a lot of bang isn't necessarily what makes the real tangible difference.

Revolution in synchronization

What they have with "channels", as a synchronization mechanism, can be implemented in pretty much ANY language by simply wrapping up the existing synchronization primitives into a wrapper to make it behave like a channel. There isn't any "miracle" in that mechanism.

A while back, there was one computer scientist in France who argued that the very idea of some miraculous language that will solve all the problems and will be able to even write some programs in an automated fashion, is nothing more than utopia. Because every system is unique and it has to be painstakingly crafted out of subcomponents and plain, ordinary logic. There exists no general purpose "do it all" mechanism, in principle. And they have been battling this argument for generations and invented all sorts of languages since then, but to this very day, they were unable to disprove his argument.

Every language has its flavors and its own ways of doing the same things, and if you look at any of them, then, sure, it can do some things better than others. But what is interesting here is that the most active Usenet group among all the languages happens to be C group, not even C++ or Java or even Python, as popular as it was and is, at least for some things. So, it turns out that plain, ordinary "vanilla" C is orders of magnitude more usable than any "advanced" language out there. And it does not even have threads, synchronization, or any other mechanisms and bells and whistles? Why would that be?

Sure, if you study their mechanisms in more detail then who knows, might be there IS something to it, that is not as obvious as it might look from the first glance at it. Except the issues of synchronization do not necessarily reduce to the issues of serialization. Those are not quite the same animals. No matter what "channel" do you use to serialize things, it does not mean that you do not have the synchronization problems on a higher logical level, or even on the parallel level. It all depends on what you synchronize with what and in respect to what. There isn't a general purpose mechanism that covers all the cases conceivable. Otherwise, it would have been implemented generations ago.

Synchronization is something "natural" to Go

So, to claim that synchronization is something "natural" to Go, and is way more powerful and universal than anything out there, as this developer-in-chief claims, isn't quite the case. Yes, certainly they have come up with some tricks that make certain things easier, hopefully, but so is with nearly every other language.

But synchronization is rather a subsystem or a resource access protocol in itself, and it is highly contextual and needs to be crafted and well thought out as a whole, system-wide. Else, you might be unnecessarily blocked most of the time, if not getting into the blocked states for prolonged periods of time, while you could actually do some other things. When you have multiple threads running, that could be on different levels of execution, and you need to synchronize access to a number of different resources, depending on a context, regardless, then it is doubtful that there exists some magical trick that will cover all the specific cases conceivable, even though, sure, you may be able to simplify some things.

That is why Java, for example, did not even attempt to take some radical steps in language design and introduced only some additional mechanisms while retaining the synchronization primitives, familiar to all, without any fundamental changes.

Exceptions issues and error recovery and reporting

Let us look at the exceptions mechanism in a much more detailed way. Because this is one of the keys to reliable programs and some aspects of it might not be apparent and the benefits of exceptions could be easily underestimated, especially from the standpoint of error recovery, meaningful error reporting and the overall stability of a program.

The authors of Go go as far as to claim that the code with exceptions is actually BIGGER and MORE complicated than without them, which is simply bizarre and utterly invalid. How could the code without conditional logic be bigger than with it? Even if you consider the catch clauses, it does not necessarily mean that all of them need to be handled with any additional code, and if they have to, then all the code in catch clauses is meant to either do some cleanup work for resources release, or to provide some informative error message, which has to be handled, no matter what, exceptions or not. Otherwise, what is that code for?

Exceptions are nothing more than "long jumps" essentially, that simply redirect the flow of execution. Nothing more than that. Except that there might be a stack pointer adjustment that puts you in the correct place in a routine that catches the exception. Exceptions are like a fancy branch instructions for the most part. There isn't any additional complications in your code because of them. NONE!

Claiming that the code with exceptions is bigger or more complicated than without them is essentially claiming that checking for return codes to detect the errors takes an additional work. Yes it does. But does it meant that you do not handle the error conditions? HOW? And in fact, that is exactly what exceptions are for. Because they allow you to avoid checking the insignificant errors in the context of what is happening and bulk it all in the exception block. But you are guaranteed to handle the errors, while with return codes it is much easier to get lazy and not handle some seemingly insignificant error conditions, and that is precisely what breaks the programs in most cases. Lazy or sloppy error checking or handling.

Furthermore, it is precisely because of exceptions the mainframes are known to be orders of magnitudes more stable than even Linux, which is probably the most stable thing the "mortals" have a chance to deal with. I have heard from one guy who was a system programmer for the IBM, that the exception mechanisms in the IBM mainframes are so sophisticated that they are even more complex than the program itself. They are effectively chained and re-handled on the way back to the main working routines. And that is precisely their power in our opinion.

Very often, if not in vast majority of all the cases, the error reporting value is highly underestimated, if not ignored entirely. As a result, you can see it in nearly every program out there. Something did not work, but what you see as an error message is totally uninformative. It does not tell you a single thing of real value that helps to identify the problem. In most cases, they do not or even can not report the correct file names. Because as soon as they encounter any error on a deeper level, they simply dump any crap that they know about on that particular level of execution, which, in vast majority of the cases, can not possibly tell you which exact file, being transferred from which node to which other one, had a problem. About ALL they can tell you is that some block read have failed. But what does it tell you of any value? How can you identify what did it have to do with? In relationship to which file? During which higher level operation? What was the source and destination in the logical operation sense and not on the level of O/S nuts and bolts?

If you do NOT use the exceptions mechanisms, then you'd have to check the return codes from nearly every function call. Because anything can go wrong, down to seemingly simplest operations. That is why nearly a half a well written and reliable code does the tests of nearly all the return codes, and only proceeds if the results are OK. But the thing is that from the standpoint of errors and program operations those intermediate things are not necessarily significant overall.

So, if you do not use the exceptions, the program logic related to constant checking the return codes easily gets so complicated and so confusing that sometimes you can not read that code, at least easily. Because it is full of if/else and unnecessary while or for loops that may be trying to redo the operation, and, possibly, even go to sleep doing that. Because the resources might not be available, at least temporarily, or they may be blocked by some other thread.

If fact, one may easily claim that the code without exceptions, as a rule, is more complicated than necessary, more convoluted and less efficient, simply because there are unending tests all over the place, and every test eats the machine cycles away. Every jump is the most expensive CPU operation there is outside of function calls and associated argument pushes/pops and expensive arithmetic instructions.

But what IS significant is that a particular routine did execute correctly, and if it did not, it is not an absolute necessity to report which exact called function did not work or even take any further action. The caller may implicitly understand that things did not work as advertised and will reschedule it if necessary. For example, if you want to transfer some file in the torrent approach, you have to transfer each block/piece, which has its own hash and its own algorithms to do various sub-things.

But, from the standpoint of overall program operation, you do not necessarily report, or even care to know, that "block number 447 was not able to complete because the connection was lost", or even worse than that, "for unknown reasons". You might not even know to which file that block belongs when you hit that error condition in a low level code, so all your error reporting at that level becomes meaningless.

But about ALL you need to report is that file such and such was not completely transferred from node A to node B because the network or the remote node is down, but it is rescheduled to complete as soon as the net is back. That is about ALL you need to know about that error. Anything else might be nothing more than a meaningless noise that simply saps your energy while delivering no benefits of any kind.

Furthermore, one node may work and perform the transfers happily, while, at the same time, the other node may not work. But if error is not reported descriptively, you might have an impression that the entire net is down!

So, what you do with exceptions is to place some block of code into the exception handled block. But, no matter what this block of code calls or does, it might be utterly irrelevant from the standpoint of the overall operation. So, you may have 10 things called, each of which may result in an error, but the fine-grained analysis is not really necessary, or even helpful to the overall operation, or could be even distracting from more important things. So, you don't need to check on every return code in that block. You just execute any statements in it or any routines it calls, and if ANY of them fail, you hit the exception for the entire block, even though the called routines might have thrown the exception themselves that took you into your exception catch clauses.

But you do not necessarily care which particular routine and at which depth of stack and while what exact activity have caused it. Thus, the exceptions also behave as information gathering gateways. They are like the interceptor nodes through which the traffic flows regardless of what is happening under the hood and in which particular place. So, this way you can test for various logical consequences of all the significant operations.

Since in the code with exceptions you don't need to test the return codes, there is no if/then/else logic, filling up the program sources, at least for the most part. About ALL you have is statements that actually do things. And, if ANYTHING goes wrong somewhere in the block of code, the corresponding exception is hit, which can be handled locally, to log some things or free some resources, and then re-thrown to the higher levels with probably a different and more precise exception, that has a higher level logical meaning, and will allow you to handle the situation in the most appropriate way and log the error in a much more meaningful manner, with all the pertinent details as to the arguments to the routine that caused some error.

Because the routine itself is not likely to know the higher level context of execution, and, therefore, its error message will either miss the really important conditions, arguments, or parameters to the main worker function, or will display quite a meaningless error message that will give you some irrelevant low level details that will not allow you to figure out the source or relative significance of the problem in the clearest and fastest way possible.

To achieve the same results without using the exceptions, your code would probably be twice as big overall. Realistically speaking it is simply impossible. Exceptions are probably the only way to "control" the higher level logic from the low level code. Not exactly "control", but at least to communicate something far more meaningful then simply return some error code and then let it guess what was the exact reason for this error condition.

Furthermore, different exceptions may be caught in different places of the higher level code even if they were hit in the same lower level block of code, which means that you effectively have a number of different paths "back home" - to the higher levels, depending on the exactly appropriate logical meaning of those. This way, you can "staircase" the exit path to higher level, where each successive level logs or displays the more general level information about the problem, and at each level you may log the pertinent conditions or arguments, starting from the most fine grained and up to the most upper level of general operation.

Furthermore, from the same exception block you may hit the higher level catch clauses in different higher level subsystems, such as the file system interface, or networking or any other subsystem, at least for the most part, at least if you run in the same thread. In order for you to do the same thing with return codes the number of source statements would probably triple, and at the end you might simply get confused and would loose the control of the whole thing as far as being able to follow the code conditional logic or trying to figure out what exactly has happened by merely looking at a single error message, which could be insufficient for clear identification of the problem.

Go design summary

And we can go like this for a while, combing this Java vs. Go arguments in the most precise and specific manner, and after we're done with that, there will hardly be anything left of this entire Go vs. Java argumentation trip, which is peddled by Google like something that will help to "save" the world form all the evil.

Go, essentially, is nothing more than a salad of languages, such as Pascal, Python and so on and some ideas implemented in all sorts of external libraries, such as synchronization grade issues. And it took probably the WORST of those languages, strange as it may sound. Except in Python they have gone even further in madness and removed even braces around the code blocks and replaced them with mere white space indentation on the next line(s), following the conditional statements.

And yes, they also removed the end of statement tokens. That is probably where Go had borrowed this sickness from. And the kicker is, if in one case you have a line beginning with 4 blank chars and the next line is intended to be on the same indentation level, but starts with the tab character, which is also "legal", then depending on the tab expansion setting in your favorite editor, these 2 lines may look like they belong to the different indentation levels, whereas they are not. And these are not just some "wild" cases that never happen. In fact, they happen all the time, by merely copy/pasting some code from a totally different program, or even a browser window and then thinking that it all looks right, whereas you are now executing a totally different block of code than what you thought you are.

And we stand by these arguments, whether he likes it or not. It does not matter, even an iota, and it does not change anything, at least to the better.

Therefore, the very argument of portability, at least as compared to Java, is utterly invalid.

Sure, you can fall in love with Go. But it does not mean that your blood needs to boil when someone mentions Java, or even Python for that matter, and tells you that he would prefer in was done in Java.

Yes, certainly, it is easy to fall in love with some new language because it is fascinating, at least for a while. But eventually, when the freshness and all the hype around it rubs off, you might not believe how naive were your expectations of that "bright future of mankind" and a "miraculous" solution this new language claims to offer.

Finally, "There are no friends on likes and tastes" they say.

But would you be willing to KILL someone or even BAN them for 10 years, which is probably the sickest joke in the poorest of tastes one can imagine?

Interestingly enough, that ban was put in place after we posted a single message about Java port of Syncthing, where we said that we support that development with both hands, only if it was of any help to that work. Well, was THAT the reason for this "suspension" for 10 years, classifying something as "hate speech"?

:)

Will Syncthing (Pulse) remain open source?

The following post was made on the Synching forums in regard to Syncthing being rebranded to Pulse in relation to new sponsor, which, how we could say it without offending anyone, is "taking over" the former Syncthing project and gives all sorts of promises of the "bright future" to the project including the resource and other forms of "support" to the project.

Messages to Syncthing main developer about optimized architecture proposal

The following message was posted on Syncthing (now Pulse) forums in response to the post of the main developer, which was his followup on the 18 points of optimized architecture proposal as described here:

Update logic engine design overview

calmh Jakob Borg Developer said:

alberto101 said:

Otherwise, it might look like a "job security" strategy. "I am the only one who really knows how this thing works and, therefore, I am irreplaceable and can not be fired otherwise the whole product flaps".

Actually, I was surprised how popular is this trick among top developers in the Silicon Valley for example.

Well, no, not really. There's no job security angle here. The code is out there; the more contributors, the happier I am!

Well, I certainly hope so. Except "job security" is a much broader notion than it might look on paper. Only time will tell what is real and what is illusion. :)

calmh Jakob Borg Developer said:

You probably haven't seen it yet, but v0.10.2 is going to be the most *amazing* release ever (hello Apple!) in terms of contributors. We've added five contributors to the list since v0.10.1 so far. That's people who may or may not be new to the project, but who have their first contribution added just now. I think it's absolutely awesome how people are picking it up, finding things to fix, and *fixing them*.

Well, that sounds interesting. It certainly does not hurt to have more contributors. But how did they appear in numbers all of a sudden? Does it have anything to do with the change of brand name? I have a strange feeling about it, but it is "good news" for you and the project. Except I would not like to discuss it, at least right now.

So, if that is the case, then I have a proposal:

Since you are quickly becoming someone like a director of development, or could be even the VIP kind of guy, then some additional things could be done...

For example, you do not seem to be under pressure now to "carry the world on your shoulders". So, why not optimize the resources?

So, the next few days you might spend on documenting the code really well, so it begins to look like reading the morning paper. That thing alone would help all other devs tremendously. Because then they'd be able to understand the code, and even could be some subtleties, much faster and much better.

Then you suggest they start studying the code that you have already commented to a sufficient degree.

Meanwhile, you clear your name, in terms of the notion of "scientific truth" by going through those 18 points of mine and addressing the issues. There is no hurry as to when and to what degree of detail it will be done.

Because right now, it looks like your great reputation has been tarnished a bit. Because you kept throwing around all sorts of blanket and quite wild formulations without actually addressing any issued therein, some of which looked pretty close to the point of intellectual insult.

Secondly, you'd provide the contribution to the document, I am working on, with your insights, which could, and probably would, be helpful to quite a lot of people that started to get interested in this work, which I know for fact.

Thirdly, it might help you to get some courage back and take a little step back from the aggressive position and clarify some things for yourself, especially in regard to such things as a need to download the entire index and looking at what it actually helps. Who knows what may come out of it?

Because by merely downloading the index, it still does not assure the data integrity. Even by the time you actually start downloading the actual files from that node, those files might be either different, or there might appear a bunch of new files and even subfolders with a bunch of new files. Which means, that even if you downloaded the entire index, it might become invalid, at least partially, almost immediately. So, what was the point of that exercise then?

So, if you do this, then you do not look like you are just trying to spit into my face and make a fool out of me, who is "running his mouth" without knowing what is he talking about. Not that it matters much to me because I have no need to carry the image of "greatness" around. But you would fulfill one of the major requirements for discovery of That Which Is, which is a definition of Truth, if you care about such things.

Otherwise, it might look not in your favor and might mean that you are simply frightened of the consequences of such a "radical" change in the point of view in the entire project's architecture and design. Yes, I see some things that might cause some serious reconsideration of some things, which, at the end, may turn out a major effort.

But... The good news is that now you no longer have to look like a "coder slave" and can afford some luxury of a more distant view and supervisory abilities.

Plus, you will be able to PROVE that WE are so wrong with those 18 points that there is just nothing in it worth even discussing. Pick the ones you "hate" the most and go at it and get as "critical" as you wish. The more, the better. But the arguments have to be precise and sufficiently complete to comprehend what you mean and why. Those things do not hurt us. Because we have no need to carry the big egos around. They are meaningless to us "in the scheme of things".

You will also be able to prove to me, the others and to yourself, which is probably the most critical aspect, that "lots of words" I produce are merely "full of hot air" and do not signify anything of substance, and are in fact, "tales told by an idiot, full of fury, signifying nothing in the end".

Then you could really start looking at a good place in the world to put up a monument to yourself. Because to "beat" the guy like me and those who stand behind me would be "something else" indeed and would deserve an applause, no matter how you cut it. And also, all those who might be following these things in various places would be able to rejoice in their heart with the idea of "the justice has been done, finally".

Makes sense?

So, you either scrap those 18 points as a hole and provide the "killer" argument, or you invalidate any of them, which you find the most "ridiculous" or unrealistic or are "full of holes". That would help me, personally, and other people that are watching this show, and who knows whom else? The world is big! And once things are born they become alive forever, some say.

calmh Jakob Borg Developer said:

Because god knows there's enough stuff to fix, and all of us who work on it do so on our free time, without compensation, taking time away from family and friends etc.

Well, I already gave you quite a compliment and encouragement in the very first post of mine, if I recall correctly.

And I can repeat: your work is certainly appreciated by many, including those who stand behind me, and that is probably why your project and you were chosen "to take this thing to the next level". You know what we are looking for, don't you?

calmh Jakob Borg Developer said:

Edit: Yes, you have a point - the current code doesn't read like the morning newspaper. If you look at the last few months of commits though, you'll see a conscious effort to both clean up dark corners and document *why* changes are made, in comments.

Well, actually, the last two times I have looked at your code, especially the last time, precisely this aspect just jumped into my eyes. I wanted to rub them, asking: "Do I believe what is see? This guy actually started adding some comments, and some of them are pretty good, and some of them are even in line?", (which is great idea to warn about tricky places and issues).

What to do if it takes so many words to make that happen? But it DID start to happen and the value of it is hard to even comprehend. Because with well commented code you open up the gates of Heaven to your project. From then on, the development could start progressing in such a rapid manner, that it is hard to even believe it could. And so the bug fixes might start pouring on your head like a spring rain.

Who knows?

calmh Jakob Borg Developer said:

I also think some credit should be given to the fact that we've created something that fills a niche and mostly works

There is no doubt in my mind this is actually the case. Except I do not like the very notion of some "niche", which is like a hole to me. Who are you trying to hide in those "niches?". Not from YOURSELF, by ANY chance? Why niche, when you can have that, which I can not even begin to describe to you?

I can tell you this, and I hope I am not too "pushing" here:

Once you implement the concept of a public share/folders, no matter how "difficult" it might look on paper, you'd be doing such a great contribution to the whole of mankind, that I dare not to even start imagining it here. That much of responsibility I am willing to take for making such "grand" statements. Primarily because I see nearly "exactly" what will happen to this project and in which ways it will contribute to some vital things in this world, and these are not just "big words".

calmh Jakob Borg Developer said:

(although with bugs etc). So we're not ISO 9001 standardized in the process, but at least we've *made* something.

Yes, indeed.

calmh Jakob Borg Developer said:

You have to realize that there are very few people working on this project and we have a finite amount of time to dedicate to it. To be blunt about it, arguing about stuff here takes time away from actually taking care of the almost one hundred to-do items we have in the bug tracker at any given time. So at some point, doing stuff is worth more than talking about it.

Well, may be it is time for you to "step back" a little and take a second look at the bigger picture?

Things are rolling now. You are getting new people coming. So, you no longer have to carry "the weight of the world on your shoulders". It seems to me that your contribution in terms of well commenting the code would be many times more effective than the actual coding work you might be able to do in that time.

You can just take a position of "senior consultant" and propose some ideas to other developers and make sure that people do not start fixing things or bringing some new ideas that might break the other things because they do not quite comprehend the consequences.

And if you hate the very idea of a public share/folder/collection, just because it is "too much of a pain on the neck", then why not ask one of the new devs to consider those 18 points and see if they can find a way to make syncthing work with fully automatic joining of the public shares? Who knows what might come out of it? What if someone really gets interested?

Another great thing I see is: what if you spend a few days on writing a design document where you cover the basic ideas about the architecture, its "philosophy" and get into some details on why, what and how?

Combining this with code comments, could mean that the MAJOR work is done for you, personally. And now you can afford to "lay back an relax a bit", unless, of course, you are a restless kind of guy and might even begin to start having the heart problems if someone pulls you by the ear from the screen and coding, which you seem to enjoy quite a bit.

calmh Jakob Borg Developer said:

Killer zombie mutant/hybrid: "Deeds not words!"
KILL 'em!!!

Well, I am not going to comment on this picture, but if you think that "deeds" could be done with a gun pointing upwards by the hybrid "superman" kind, who has a brain control box on his chest and a brain controlling transmitters on his head band, then sorry, "I am from a different joke".

No, to me this is a shame and disgust that a human being could be reduced to the level of an obedient biorobot, if not an outright killer zombie.

These kinds of images of violence, war and destruction are used extensively by the NWO machine to zombify the human minds and reduce humans to the level of a mechanical machine, killing machine in this particular case. Just ask yourself a question: "what is the need to promote the ideas of death and destructions anywhere you look"? Is it something NATURAL and necessary for Life as such?

Hope this helps.

Are there any "use cases" for public applications and a public itself?

This post was made on syncthing/pulse forums.

In fact, you might even formulate it like this:

"Use cases for what? - For general public? For libraries? For newspapers? For books? For TV and media? For education as such? And even for zombification of a human mind and biorobotization of the human beings?

Because ALL of those are examples of public applications and sync is probably the best way of delivery of ANY kind of information, and the information is DYNAMIC in its very nature. So, how could one be so dumb as to even comprehend asking such a disgusting and highly derogatory question?

There are dozens of torrent clients regularly present on line. The torrent traffic is probably the major part of the traffic on the entire net.

But try to prove to such perverts that it all makes sense and there are indeed "use cases" for it.

And so, if THAT is what constitutes their "development team", then it seems pretty obvious, even by implication on its face value, that most of the best of developers, that participated on this project were simply eventually banned and you can't see even a trace of them on these "forums". Because it seems that ANYTHING, and quite literally, that challenges the image of "greatness" of the dictator-in-chief, has been either mutilated, moved to some dump, nobody reads, or simply deleted under all sorts of pretenses, such as "not on topic".

Because if you have to "prove" even such obvious things as public applications as such, then what would happen with more creative proposals, suggestions or some critical analysis of the issues of performance, functionality and futures, if any and all of it is flatly denied at the very root?

Then the questions arise: but what IS "on topic" with this project? What is its FUTURE? In which way, shape of form can you develop it any further? What kind of things you might consider "on topic" with this project? What kinds of things and features would you implement next?

Take, for example, one single case. In version 0.10.3 or so one of the things added was an ugly biorobotic face in variations of shapes made out of little square boxes filled with color, next to the node name in GUI. What a breakthrough in development! What was the development effort and time wasted on? Yes, WASTED, LITERALLY! To make one feel like he lives in the world of robots and the ugly pakimons or Simpsons "rule"? Just to remind him on subconscious level that he is not different than a biorobot, a functioning machine? How can one even look at such idiotic pictures without getting utterly disgusted?

And THESE kinds of ugliest things are literally jammed into ones subconscious mind, forcing him to "get used" to the fact that he is not much different than a mere mechanical robot, only a biologic version of it. Because every time you look as Syncthing GUI you see this disgust, even next to your own node name!!! What does ones subconscious mind is to make of it? Is one expected to feel "good" with this kind of "breakthrough" in development? Well, but what happened to Windows 8, but utter disgust with it worldwide and a significant drop in sales, and PRECISELY because of this very ugly model of reducing the world to mere ugly pictures with zombie-like images all over your screen?

And then, starts the usual thing: violence

And then, the last message on that thread was from the main developer:

Mind you, this individual has moved several perfectly "on topic" posts to some hole that does not even show up in the forums list. And some of those posts he moved were by developers, not counting the author of the proposal, who is the most senior level programmer and a systems architect.

That raises a question: how many other ideas, proposal or attempts to discuss the program problems and shortcomings were simply destroyed and how many developers have just disappeared from this project, dominated by the ego maniac with hypertrophied sense of self-significance in the scheme of things?

So, instead of being grateful for the contributions and proposals of a solution to probably the main and most fundamental limitation of the program, these people would sacrifice even Truth itself, just to protect their pathetic, rotten, arrogant and obnoxious egos, full of crap and hot air.

There is no point to quote the other of his ugly and highly insulting posts, full of hot air claims and denials of the most critical issues of the program. But this is something noteworthy in itself.

Well, what can one say but this is the reflection of the mind of the same man who posted the ugly picture glorifying violence under slogan: "Deeds, not words".

Deeds, not words

Basically, the same procedure as on BTSync forums: deny, insult, pervert, humiliate, followed by mutilate, kill, and finally, the most pleasant thing: ban!

About the only question remains about this is:

How many other posts have disappeared from that thread?

And probably the most significant question would be: what do you think is the future of this project and what may possibly come out of it?

Final message on Syncthing/Pulse forums about public shares and general public applications

How to insult without saying a word of insult

The following message is the response of the main Syncthing (Pulse) developer on the above message with point by point analysis of the blanket statement he has made, and not for the first time, by ANY means.

Well, in order to have some "point" you need to express it. Just to say "it's tricky" tells you nothing. WHAT SPECIFICALLY is "tricky" and, if that is the case in fact, then why one of the main developers said:

"We'll have to disagree" on what specifically? Where is your great enlightening insights into reality of how it is? Are you God - All Knowing by ANY chance?

Well, what can we say? We know this position already. But the point of that post and a proposal was to discuss the issues raised by calmh, point by point, and, using them as a starting point, to consider the related issues involved, including the crude model of downloading and even keeping around the entire index from EVERY SINGLE node, and not even talking about the alleged need for the permanent TCP connections. We claim that these things in combination will simply choke this program and will bring it to a grinding halt even at relatively small loads. Are these issues significant enough for yerr royal highness, Mr. Smart?

Yes, certainly, you can refuse even to discuss the issues. But in that case, every point you have brought up, is thereby invalidated. Simple as that.

Else, you'd have to argue your case. Pick any point you like and EXPLAIN the specific details of WHY you think such and such point of yours stands.

Join the herd and attack the messenger

Yes, SHOULD. That is the only language the dictators and their rear end lickers seem to understand better than anything else.

Well, first of all, regarding the "long articles", this is actually precisely one of the major problems with documentation in general. Because it is written in "one liner" style, that is cryptic, insufficiently described, not covering all the cases and so on.

Well written documentation tells you nearly everything and every single aspect, error condition, various uses and so on. Sure, there is a limit of detail, otherwise you might have to write a pretty fact chapter on nearly every issue out there.

But the idea is that once you read a particular section in a doc, ideally, it should describe anything and everything about some particular case. Otherwise, it is nothing more than waste of time and you have to go find some other information somewhere, waste yet more time and so on.

But you can hardly find a single reference that you can use and expect it to answer at least most of your questions fully, in detail, so the issue becomes clear, even to to not so technically inclined readers.

If you look at most documentation out there, it is simply horrible. Instead of answering questions it merely creates MORE questions.

The same thing happens on forums of all kinds. People write a single sentence in response to some issue, but they do not sufficiently describe things, and their argumentation is usually absent altogether. So, either the issue remains unanswered, or is answered only partly. As a result, there there are followup questions, which lead to yet another one liners that do not necessarily answer it in full.

The "long articles" is precisely the difference between caring and "I don't care"-ing, respecting Life or worshiping and promoting the ideas of death. Because in essence, those "one liner" articles and documentation is essentially sucking the energy of the victims, yes, victims, that are to read those one liners. Because they essentially tell you nothing and clarify nothing, and so they merely sap one's energy.

And so you have to spend days until you finally get an answer to your question or some issue finally gets covered in full. And that is simply a royal waste of time. But if you describe the situation fully, in a single post, then that particular issue does not even need to be discussed further, unless some new aspects are revealed.

Basically, if your TOC is detailed enough and you can find all the issues described in a doc directly in the TOC, then all you have to do is to click that link and go directly to exact explanation of an issue. You really do not need to read things that are of no interest to you at the moment. For what?

Secondly, as far as writing a book, that book is ALREADY written, or, more precisely, is being written as we speak, and you are reading it right here and now, and, btw, this book is considered to be a "Great reference document", even on superuser forums.

BitTorrent Sync - Exclude All Specific Subfolders
http://superuser.com/questions/749957/bittorrent-sync-exclude-all-specific-subfolders

And this is not meant as something to brag about, but just to put things in proper perspective...

Interestingly enough, it is the only article on these kinds of forums that contains a single comment or answer, and that is:
"Great reference document", and here it is:

Sync and BTSync Detailed User Manual: Using SyncIgnore file

Or here:
Sync and BTSync Detailed User Manual: Using SyncIgnore file

And this is the "original, factory" User Guide with detailed installation instructions for different platforms:

BitTorrent Sync User Guide

And, since that reply, there is not need to discuss it any further. That means that once one looks at that link, there are no more questions left in his mind as to how it works. And that is the best compliment to any documentation that the writer could even hope for, and that is precisely the difference between well written documentation and some cryptic one liners that raise more questions than they actually answer.

There are quite a few people watching it as it develops, strangely enough, at least according to the "smart ones" of your kind. Interestingly enough, it is being watched even by the Google employees, strange as it may sound.

Do you even begin to realize how many people have already read it, and are reading it every single day? Well, probably more than Syncthing/Pulse forums.

"Every single of your posts is so long it's agonizing to read them"?

First of all, it depends on reader as well. If he is just an impatient psychotic, then he won't be able to read anything longer than one sentence. Because the span of attention of most people nowadays is between 3 to 5 seconds.

Sure, if you claim that the posts are just too long and do not cover the issues and do not present any useful or important information and are nothing more than some blabber, then yes, you do have a point and there are plenty of such posts, even if they are not too long.

Question: how do you cover a number of aspects and issues if you do not present your detailed argument that covers the issues from all the relevant points of view? And the issues discussed in every one of those "long" posts are multiple and the consequences are multiple and profound, basically every one of them. Just try to find some irrelevant argumentation in any of those posts? Even if we mention this satanic NWO thing, it could turn out even more relevant than any purely technical arguments. Because the significance of those issue spreads on everything, the very Life as such. So, how can you possibly ignore it?

- Well, then what is the need to read them? Just skip them and go to those posts that make some sense to you. What is the need to make a fuss out of it and in such an augly and insulting matter? Do you, lil rear end licking cockroach, even realize how disgraceful your idiotic and utterly devoid on ANY content post is?

Are you merely trying to make a fool out of yourself? Or show what a big and "significant" prick you are, getting into someone's face with your idiotic insults?

Well, what to do, if you don't seem to be "getting it" even if it is described in the most detailed way, giving you the specific and most precise technical arguments conceivable, and all the reasoning behind them and even their connotations?

The way we write is really simple: once something is written, there should be no need to further discuss or explain anything contained in the writing, simply because everything is described in such a way as to allow even not very competent people to understand any of it with exceptions of some technical details that depend on ones background.

So, once something is written, it then becomes a reference of some kind and once you read something, the picture is clear in your mind to the utmost degree. We have no need to write or even discuss the same things, unless we feel compelled to "drive some point in" since it does not seem to be "getting through".

And how do you describe the volumes of issues and design shortcomings which syncthing has at the moment, and seemingly total absence of a good willed intent to even look at probably some of the most important issues with this thing, and that is public shares, performance, stability, predictability of an outcome, consistency and so on?

"@calmh says implement a feature if you really want it"?

How many times is it necessary to repeat that, first of all, currently we are not doing the development work, even if you pay for it, whether you like it or not, makes no difference, and secondly we can not look at Go for more than a few minutes at a time because after that there is a strong urge to vomit?

Secondly, we do not need this feature. YOU do, or more precisely, syncthing needs. Otherwise it has no chance to become something real, at least for the public level applications. Simply because it does not even have such a concept as public applications, and, furthermore, has some fatal grade issues even in some group applications, simply because ANY member of any group can obliterate the entire share/collection/folder if he so desires, and ANY time he so desires.

Basically, the way it is at the moment, it is good only for "play with yourself" applications unless you can trust others in your group as much as you can trust yourself. Tell me, oh master of disaster, how much do you trust even yourself?

Advice message to developers

The following message was posted on Pulse (Syncthing) forums in reply to the main developer.

Basically, this is an advice and a warning message to the developers.

Basically, the primary reason for this message was to at least help their project to become more manageable as far as code clarity goes.

In essence, such a flat denial by the top developer of the entire point by point analysis of specific issues raised by him in one of his blanket posts, was beginning to look like an insult of a subtle kind: "well, you OBVIOSLY do not know what you are talking about and so there is no point for me to even bother". This his another subtle insult "well, this thing is getting too long" [for me to bother with]. Implying "I am such a significant guy that I do not have time to talk to the clueless who do not know what they are talking about.

And we have seen this again and again with nearly every "great" developer, at least on sync related projects.

Well, most of these developers are so "great" that it would be appropriate to mention here what Diogenes have said to Alexander "The Great":

That was his answer to Alexander's query just before one of the biggest battles of his life, when he came to Diogenes and asked him the question:

"Tell me, why everyone in the land says your are such a great wise man"? So, I came to get come wisdom from you before this big battle I am going to.

Yes, and that is the message to all the "great" developers "out there".

(Diogenes, one of the greatest and wisest men that ever walked the face of this planet, lived in an empty barrel on the bank of the river, away from the crowds, their "great ones" and their madness of a herd. And Alexanders of all kind were willing to give him a half of their kingdom if he would agree to teach wisdom.)

Tahoe-LAFS

There is also Tahoe-LAFS that is past beta phase and is a fully functioning product. It allows for creation of distributed storage, which has its own pluses and minuses as far as pure syncing is concerned. It is a bit more complicated than merely a sync program, but it seems to do its job, except we did not verify any of it and so can not make any specific statements about it.

https://code.google.com/p/nilestore/wiki/TahoeLAFSBasics

Hive2Hive

Hive2Hive is an open source library for sync written in Java. This is a good choice of a language because Java is as portable as it gets and runs on a variety of platforms.

Unfortunately, these guys are pretty slow on making a prototype app that can demonstrate what their library can do. Considering it is written in Java, it should take a few hours to create a sample prototype app for the others to see how and if it all works. But for someone who does not know anything about the code it may take days, if not weeks before one can see the results, which is totally unacceptable at this junction.

We wrote them an email at least 6 months ago asking for the same exact thing. But even to this day, Sept 6, 2014, we do not see a working demo app, which is a shame. All we see is another version of some theory, telling you things, but not putting it all together and hooking it all up to some GUI so that one could see how it actually works, instead of wasting days and weeks of his time trying to figure it out just by looking at the code.

What's wrong with those guys?

http://Hive2Hive.org/

github:

https://github.com/Hive2Hive/Hive2Hive

picosync

Picosync was an attempt to create something compatible to BTSync. The author tried to look at the traffic and packets and did a considerable amount of work trying to reconstruct some of the things BTSync does and how it works, and he was able to get some preliminary results.

Unfortunately, that is not quite a correct approach. Just by looking at the traffic may take many times longer to figure out how the program works, and even then some of the weird algorithms or BTSync can not be decoded by simply looking at the traffic. That is practically impossible. From purely technical standpoint, even program de-compilation might have produced more effective results.

For example, there is a pretty good decompiler called REC and it produces a decently looking C code. Sure, decompilation is not a simple thing to do, but still, you could see quite a bit of the inner workings with this decimpiler. Yes, most of decompilers you will be able to find will not even run. They'll crash on you as soon as you start it. But this one doesn't.

Download REC Reverse Engineering Compiler
http://www.backerstreet.com/rec/recdload.htm

As a result, there seems to have been no work on this project since mid 2013. But the author was able to figure out some significant enough aspects of how BTSync works, which deserves an applause.

Actually, considering a pretty weak methodology, the author was able to get quite a few things written in actual code.

Here is his code:

https://github.com/picosync/workingDraft/tree/master/picosync-qt/src

https://github.com/picosync/workingDraft/tree/master/picosync-qt/src https://github.com/picosync/workingDraft

ClearSkies

ClearSkies is not yet a functioning program. But it seems there have been a fair amount of thought and consideration to it, and you can probably find some interesting and valuable ideas in what is available at this point. And there is some code to look at. It is written in C++.

We haven't looked at it to the degree that we can make a comment worth the paper it is written on. So...

Here is the code:

https://github.com/jewel/clearskies

Protocol:

https://github.com/jewel/clearskies/tree/master/protocol

TomP2P

TomP2P is not a complete sync program or specification. It is a DHT implementation for node discovery, which is one of the more important, if not the key aspects or establishing P2P communication. We have just glanced over it and can not make any comments of any value. But one of the most important aspects we have noticed is that it works asynchronously and the author also claims a high performance. These are two of the most important keys to node discover over DHT.

Also, it seems that this DHT implementation is fairly complete and utilizes the distributed nature of the DHT node discovery, which significantly reduces the program load on resources and a memory footprint.

So, for the node discovery, this looks like something worth looking at.

One more good point going for it is that it is written in Java, which means it is as portable as it gets and is written is a well designed language that proved itself with time.

https://github.com/tomp2p/TomP2P

libutp - The uTorrent Transport Protocol library

This is one of the most critical subcomponents of the sync system. For more details see:

Transfer and higher level logic engines

So, here's the code:

https://github.com/andradeandrey/GoUTP/tree/master/libutp

Vuze open source torrent client as a base to build the sync app

We have made some attempts to contact Vuze with a proposal and asking for a feedback on using the Vuze as a code base for the sync app.

Vuze is written in Java, which, for one thing, means a very high degree of portability. Another advantage is that with Java you are no longer forced to use the WebUI approach to GUI, which is not the best way to provide a powerful GUI. Because browsers are not designed for that. Java has a very powerful GUI subsystem, which is orders of magnitude more powerful and flexible than any browser based design.

Here is the messages we have posted on Vuze forums:

Vuze Sync plugin approach query

What is the difference between torrents and sync?

Messages on BTSync forums about 1.4.72 upgrade disaster mutilated and eventually deleted by the "moderator" and alpha test monkeys

Finally, these are the messages posted to BTSync forums. All, but the 1st one were removed for "violations" of something.

One funny thing about the "thought control police" on BTSync forums is that one of the most violent, obnoxious and arrogant "moderators", about whom several users expressed displeasure as a result of his "moderation techniques", has a title of "alpha-tester". But BTSync is in "beta" stage. So, he seems to be from a wrong joke.

Secondly, why are people like these allowed to edit and delete the posts and even shut down the user accounts on BTSync official forums? Are they at least employees of BT and do they represent the position of BT in all these matters?

Update as of Sept 24, 2014.

Nearly a month has passed since we tried to install the 1.4.72 disaster update. After days spent trying to recover from nearly all the nodes not syncing we can report the following:

• Our most important shares do not sync for nearly all the nodes. Those nodes still show as empty even though we know for fact that they were fully synced before we tried the upgrade. Since these shares are general public situation, we can not possibly contact the nodes and give them advice on how to recover. One or two of those nodes did recover, probably after reading either this manual or the chat page.
• We have lost a number of nodes on the most important shares. Those nodes were the regular nodes that would show up periodically to sync with updates. But we do not see some of them showing up any longer. How many of them have actually gone forever is hard to impossible to say because it is a matter of statistics of nodes showing up on a share.
• We did receive one email where some user asked what to do to recover. We gave him instructions and he did successfully recover by following our procedure on going back to 1.3.109.
• It seems that something has changed on all the nodes that do not sync after our attempt to update to 1.4.72. It could be file permissions or file update times. But we have no way of knowing it for sure unless we spend a good few hours trying to retest the whole upgrade situation, create an r/o node and fully sync it and then try to recreate a disaster. The whole thing is likely to take the entire day, which we have no interest in spending on this thing and which still won't help with all the nodes that still do not sync.

All the following messages were removed by GreatMarko without even saying anything or commenting or explaining anything.

First of all, it is b0rman who provided a semi-legible suggestion. Secondly, yes, indeed, these are not technically correct and complete instructions. Because what does it mean "uninstall Sync and remove all settings"?

One of the problems is the database format has changed in 1.4.72, so even if you remove ALL the settings, even if you have any idea what it actually means, all the files and registry entries created after you run the program for the first time will not be removed even if you uninstall the program from the O/S. Because install removes only those things that it created DURING the installation, but not after.

So, if you do not also remove the database files for all the shares so that you would start from the equivalent of the fresh install, during the installation BTSync will crash on you because of the bug that does not properly handle the database exceptions it seems. And if you try to restart it, it will crash in few seconds. It simply won't run from then on, no matter what you do. And if you try to reinstall BTSync again, hoping to recover, it won't help a bit. Unless you completely remove all database related stuff, you won't be able to run the program. That took us a day or two to figure out on our own since these alpha-tester monkeys and "thought control police", called "moderators", or better yet, dominators, simply started deleting the messages posted to BTSync forums expressing the issues as they were coming up.

Then, there are several ways of doing the uninstall and it depends on the internals of the install itself, which may behave differently, depending on a program. The most complete and thorough way to do it is via Windows control panel. But it may have a nasty side effect of removing the registry entries and so when you reinstall the program later, you may loose some of your important setting that are kept in the registry. Furthermore, it may screw things up so badly, that the program will not behave correctly if some of the key settings in the registry are deleted and you may never be able to recover some things, just as we see after being able to finally reinstall the 1.3.109 version, after which every single node that is in fact synced started showing that it is empty.

And, no matter what you do from then on, they will show empty forever. That means that we are seeing something of a virus-like behavior and it has in fact propagated to the nodes themselves, and, no matter what you do from then on, you will never be able to restore the correct sync state unless you have access to those nodes physically, which, in general public situation, where you have no idea who are those users and have no way of contacting them, is simply impossible.

It's been more than a week since we tried to upgrade to 1.4.72 and we still have no idea as to what can be done to fix several shares so that all the nodes show the correct sync completion status. On one of them, ALL the nodes but one show as empty while they are in sync. So, at this point, we do not know which of those nodes do sync and in what state they are. And it is likely the users running those nodes also have no clue of what is going on and some of them may decide to abandon either a share or the whole program, interpreting it as "hey, this thing does not work, screw it". And these are the users on some of the most important shares we have.

Which means that you will see those nodes, and in our case the entire share, as UTTERLY OUT OF SYNC! FOREVER!!! You can just jump from the Golden Gate bridge and even that won't help. Furthermore, those nodes will never show as synced in the Transfer tab (releases up to 1.3.109), if they will ever sync at all!!! How do you like THAT kind of issue and what do you call it but a total disaster?

That is why computers are PRECISE tools and no sloppy factor of any kind is forgiven or allowed. Else, you might spend days and weeks trying to recover everything to the proper operating level. So, in such a nasty situation, where BT support staff admits that going back to previous version is not supported, they need to provide the EXACT, step by step instructions to revert to the previous version instead of merely saying "remove all configs" which is meaningless of a statement. Many people may have not a slightest clue of what it means.

The second way to uninstall the program is to simply remove the program executable by deleting the file. In that case, your registry settings are not affected and you may be able to recover at least some of your previous settings the next time you reinstall it.

The way the BTSync install/update works is that you do not actually extract/install some files out of the installation package. The program, as distributed, is simply copied to the final run directory (C:\Program Files\BTsync on Windows) and, depending on where are you running it from - from the download directory or from the final run directory, it simply behaves differently. The code executed is exactly the same, only some conditions change. That means that by simply removing BTSync from the the directory from which the program runs may and should provide it all the clues it needs if you try to reinstall the program. If it detects that there are no executable in the final run directory, then it should properly perform ALL the actions necessary, including the registry settings and config or database file removal, if necessary, to properly reinitialize the program so that it runs 100% correctly when you run it. This is another one of those weird things about BTSync - the installation is done not in a commonly accepted standard way, which is strange by itself.

Secondly, what does it mean "remove all settings" and what is IMPLIED by "settings"? There are configuration files that are kept in appdata BTSync directory, such as, settings.dat and sync.dat. They do have most of the program settings. Then there are numerous settings in the Window's registry and there are quite a few of those thrown around the registry in totally different places as we have noticed. So, if there is one or more of SPECIFIC setting that need to be removed, then they have to provide you the exact key for those settings and their location in the registry.

And what other "settings" might be there, only the competent people know. So, what should one remove and where in order remove those settings? Is this a COMPETENT answer to the precise and specific technical questions or just a sloppy way of saying "if you have a problem, get lost and 'read the manual'"?

Is this something that constitutes a competent and to the point tech support on a dedicated forum, especially in a situation with fatal consequences where you will be banging your head against the wall for days to finally restore it to a working state?

First of all, the 1.4.NN upgrade should have provided you some message during the installation with warning in big letters that once you start installing the new version, all your configurations and your previous installation will be screwed up and you will not be able to recover it.

Secondly, the installation should have made the backups of your previous installation and save it in a way and in a place that you can easily recover and go back to the previous version. Because the new version is a radical change to quite a few things and it totally breaks the existing installation in irreversible way.

Thirdly, quite a few people may and will not like the very idea of being forced to run IE for all sorts of reasons, and may find this new GUI simply pathetic and totally unacceptable, which is in fact happening right now as more and more people say that they are going to try to go back to the previous version and will no longer hope that this whole thing has any future.

This "new GUI" approach of running under IE appears to be the management decision and they have made the statement that this is the way it is going to be no matter who thinks what. There is no way back and old style proper GUI will no longer be supported in the future and versions prior to 1.4 are not going to have the bugs fixed and no new updates of 1.3.NNN version are likely to appear.

Anyway, let us see more of this whole "customer disservice" trip.

Following message is a response to GreatMarko's excuse and insult post which said something of a kind "but it would be better not to complain about problems".

The "response" from BTSync was: Both of those messages were deleted and then this:

And so, "to add insult to the injury, they send you these emails:

"Moderated" or abused? Yes, most "moderators" you see "out there" are some of the most immoderate people, craving for power, control and domination. In other words, sick people. And that is precisely why they licked some rear ends at the beginning of their "career" to get their hand on this red "moderator" power trip button labeled "Kill", so that they could insult, ridicule, delete messages and even ban the users.

That is their "joy". And to make sure they have it, they pretend to do a "good job", helping people. But most of it is not authentic. It is nothing but endless self-promotion, so that everyone would get an IMPRESSION that these are some of the most helpful people there are. And to make sure they do have access to "kill", "delete" "deny" and ban button, they keep promoting those on behalf of whom they are doing all this trip, those who own the whole thing. And the harder they lick, more "credit" they get. There comes the point where they look nearly irreplaceable, and that is the idea. It is called "job security" trip. Weird stuff, I tellya! :)

Because it is essentially a result of self-denial. As a result, one denies oneself the very possibility of being creative and, as of necessity, creativity as such. That's "the whole point of it". As a result, they claim there is no such a thing as creativity as such. And the inevitable outcome is that you become destructive. If you are not creative, you are BOUND to become destructive. Because the emptiness has to be filled with something, some substitute, some surrogate, even if it is something unnatural, that void needs to be filled.

And a result of that is that NATURAL joy, that comes as a result of creative acts, is absent and a false notion of "joy" comes to replace it - a perverted "joy" of destruction. Just like in sadists. Thus, once you deny yourself the creativity, you are bound to become a pervert and sadist. If you study these issues in-depth, it will become "clear as a bell". As they say: "there is no ifs and buts about it".

Just ask yourself a question: why would anyone waste his life and spend most of his time on being some "moderator"? For what, when you can enjoy the creative aspects of Life and explore things? What do these "moderators" do but performing the functions of the "thought control police", the most pleasing aspects of which, for the perverts, is the acts of mutilating the messages of others, ridiculing and threatening them, telling them how they "should" think and all sorts of other destructive activity?

And the final "joy" of moderation is kill-ban-deny, shut them down, make them dead, which is nothing but psychosis and about the deepest sickness of ones own being!!! What a waste of Life and creative potential! These sadists do not even realize that the biggest harm they do is to themselves by denying the validity of THEIR own being.

These destructive entities do not even realize that they merely perform the rituals of worshiping death, which is the biggest illusion of all. Because Life IS creativity, no matter how "small" or "insignificant" it might look to others and no matter in what area of human activity it is carried out. It does not mean you are some great artist, poet and so on. You can be a cleaner or a cook and even a grave digger, and still creativity is possible. It is the very nature of Life. Without creativity no Life is possible, even in principle.

And that is "the bottom line" and a core reason for propagation of sickness and violence of all kinds, which you can see nearly any place you look, and it all happens on subconscious level. Most people do not even recognize and realize their inner tendencies. HUGE subject to work with...

The sickest of them all get "awarded" a "moderator" status. And this sad thing is the reality of the "modern world" in the NWO style. Impose "democracy" on them with sword and fire!!! This is the very bone and marrow essence of this ugly totalitarian domination trip of "moderators" and "thought control police" of all kinds nearly every place you look.

These people, if you can call them people, are so sick and so filled with "power", domination and destruction trip that they should have NEVER, under ANY circumstances whatsoever, be allowed even close to not only the "moderation" red button, but to the customer service job of ANY kind in any self-respecting company. Simple as that.

And, for some strange reason, the sickest ones of all are precisely those who are "moderators". Strange, isn't it? And the most interesting thing about it is that on BTSync forums quite a few people told these "moderators" about their rudeness, insulting and even offensive behavior. And even if they give you some "advice", quite often it is either superficial or incomplete or simply said by someone who TRIES to get his face into everything and stuff his ugly favicon into everyones face, as often as they can. This "me, me, me" number. Who are they trying to convince of their validity? Not themselves, by ANY chance?

Which means it is not something nobody knows about. The issue is right up in your face. And they still remain as "moderators". In that case, the question arises: is it done by design, by ANY chance?

"Content moderated for 180 days" means that you can not post to the group. Because your posts go to the "moderator" first, and after he "approves" it, he MAY let that post through. MAY. If he is in a mood and you don't bring up or insist on resolution of the issues that they seem not to know how to resolve or wouldn't even want to bother to look at them. And it seems at least conceivable that they do in fact know those issues and are deliberately trying to hush them down and remove all the "undesirables" from even being able to discuss it.

In that case, it may imply the intentionally built-in destructive behavior of BTSync BY DESIGN, as wild as it may sound. For example, this "unusual" behavior may be used to devastate the "undesirable" information. But to users it may look like a bug or something does not work right kind of thing. It is like a back door to your comp or BTSync share in this case. Because some of those issues have a profoundly destructive effect on the nodes and shares and allow one to literally devastate your shares under certain conditions, which we do not wish to discuss at this time, and for several reasons. But if we smell something rotten or destructive, that information will be released.

We have certainly observed this behavior in specific cases while either experimenting with the program behavior or were trying to assist the other nodes to come to a correct state of sync. And we have seen it on more than one occasion and found it to be predictable to the point of certainty.

Several months ago, we have brought up the issue with timediff problem (some nodes do not sync at all if their clock time is not set as BTSync expects, and it looks like what BTSync "expects" is totally wrong. Also, nodes that have a timediff error begin to behave in totally unexpected ways in respect to the other nodes. But these issues are too technical to discuss here).

They simply flatly refused to even LOOK at the timediff issue and this sadist "GreatMarco" simply stated that this issue was not significant enough because not that many people post to that thread. But when we posted a message saying that that particular thread is read by over 80 people every day, and there are all sorts of reasons why people do not post, even if they have the same issue, they simply started attacking the author, insulting, ridiculing and provoking him in the most disgusting and even offensive ways, and finally deleting his messages and going as far as placing him on this "quarantine" trip, insane as it gets. So, they simply suppressed the whole discussion of one of the nastiest problems with BTSync. For what? What is the result?

But did the problem go away? Well, several months have passed and the problem is still there. Not a thing was done to fix it. So, who is "guilty" of what here? And who is "wrong" about what in this situation? Are these issues REAL, by any chance?

Finally, after a couple of follow up attempts to address the issue, they did the same number and issued a "warning for abusive behavior" and also placed the author under this "moderator" quarantine trip, which is simply nothing but insanity.

For what? For bringing up valid issues and trying to get their attention so they even look at it? So, instead of being thankful for the contribution and identification of one of the nastiest bugs and some of the most vital issues, what do they do? "Attack the messenger"?

Eventually, one of the customer reps admitted that there IS such an issue and promised that it will be taken care of "in the future release". That was about 6 months ago as of this writing, and there have been at least 10 releases since then, and that critical level issue still exists to this date, even on this "latest and greatest" 1.4.72 disaster release.

And here's the kicker in this whole trip of insanity. A while back, the author of this manual has received an email from BT saying something like: we really appreciate your work on this manual and it is a great help to BT. May we, please, use it for our needs? And so they were told: sure, that is exactly why it was created in the first place. But then, after all this insanity started with all their bizarre denials, attacks and ridicule, we have informed them: NO, you can not use this manual in no way, shape or form. Why should we support the butchers?

So... Does it look like a customer service more than a butcher department and utter denials of That Which IS? Mind you, not a single competent and sufficiently precise answer to several different questions that address the issues of a fatal grade was provided. Does this look as something promising? Promising of what and when?

How to get this manual via BTSync?

Note: This manual is also available via the following btsync keys ("secrets" or hashes):

Note: If you edit your local copy on r/o node, you will no longer receive its updates. So, it is not a good idea. Better to create a copy of it, even in the same directory, and work with that one.

But, if you want to keep receiving it again, then in Folder Preferences -> Advanced tab check the "Restore modified files to original version" checkbox. This should restore the original sync status of all the files in the database for this hash (share) and re-download the latest versions to your r/o node.

Program futures and comparison with torrents

BTSync operates on the same principles as regular torrent programs except it does not create a single torrent file for the entire collection, but creates a separate torrent for each file when it needs to transfer it to others.

Advantages:

• BTSync is capable of updating your version of information and synchronize it with the master copy of a site or collection.
• After initial download, you automatically receive only new files from the site or modifications of existing files without downloading the entire site or collection.
• Information updates happen within seconds if there are any changes on the site and a typical update takes seconds to complete.
• If you have access to web server, you can create a perfect mirror of the master site and let it to be automatically updatable without any intervention. All you have to do is to install the BTSync on your server and add the site hash. That is all you have to do. From then on, you do not have to worry about any site updates. Everything is fully automatic.
• Once the site is fully downloaded, you access the information directly from the files on your comp and open up the browser tab for the pages that are of interest to you. If there are any updates of those changes, all you have to do is to hit a refresh button on your browser and see what new topic appeared since the last time you looked at it.

  You can open up a separate tab for the TOC page in your browser and simply look at the category of your interests. If new pages were added to this category on the master side, you will see the new links in your TOC page by simply refreshing this page.

• You can create your own private and secure networks to exchange some private information with as many nodes as you wish and have them all automatically updated from the master nodes. You can exchange the information on that network without leaving any footprints of your traffic on the net, so that no one can interfere and/or snoop on it. Because they have no secret keys to your shares and traffic is encrypted. Yes, the issues are bit more involved than this simple description, but, nevertheless, there is at last some truth to it.
• You can create your own private collections of information and automatically update and sync a number of computers to the master version.
• BTSync encrypts your traffic, at least as they claim. So, to snoop on it becomes much more problematic even if some party gets a copy of all your traffic. They would have to crack it via "brute force" techniques, if they want to find out what exactly is being transferred. Yes, some people claim that all the claims of encryption security made by BT are suspect. Because, as some of them claim, they have observed some things that evidenced the cooperation of BT with NSA. But this subject needs closer examination.

  Furthermore, it is not certain how reliable is the encryption. It is a known fact that one of the major encryption code providers released the code that could be easily cracked by the 3rd party. Because its random number generator algorithm is not as random as one might think, which makes it relatively easy to crack it.

  For example: Are claims of security and encryption real?

Disadvantages:

• In some cases you may not get the freshest and complete updates if the master site(s) are not on line and the r/o (read-only) nodes do not have the latest version of the site.
• Some r/o nodes may exclude any files from propagation to other nodes by either including them in their .SyncIgnore file or simply modifying their copies of the files or physically deleting any files they are not interested in. In that case, their version could be incomplete. So, if there is only one such r/o node on line, then you may not get the "latest and greatest" versions of the files. But if there are several nodes on line, such problem is unlikely.
• Theoretically, you are not guaranteed to get the complete version of the site, if the master version ceased to exist for whatever reason and all the r/o nodes that get at some point on line do not have the complete version as a result of eventual site degradation if all those nodes are missing the same files and master version never comes back. With regular torrent programs you are guaranteed to get the full contents if there is at least on seeder on line. This is probably the major problem with sync approach as you are not guarranteed the integrity of a collection.

Existing BTSync collections (English)

The selected BTSync collections are described in the following page:

Existing BTSync collections (English / Русский)
(Opens in new browser tab)

Useful links

http://www.getsync.com/

Files are not being downloaded or synced

There are several reasons why btsync does not start downloading files after you created a new btsync folder. Generally, following are the major reasons:

• You have entered the incorrect "Folder secret" while creating a new folder in btsync from the "Folders" tab. Make sure there are no surrounding delimiter characters, including the blanks, before or after the secret word.
• File permissions do not allow btsync to create a folder or write into the existing folder that you have created manually, not via btsync when creating a new btsync folder.
• Your firewall blocks the port number which is set in btsync via "Preferences" tab -> "Listening port:". If that port number is set to 0, then random port number will be used, which might create a problem with your firewall.
• The time or timezone on your computer is not set correctly and when btsync queries the other nodes, their time differs for more than 10 minutes (default that can be changed from "Preferences" -> "Advanced" -> sync_max_time_diff parameter (in seconds).
• There are few more cases, at least in complicated setups when you have a LAN, routers and so on.

It is highly recommended to create a folder (directory) from btsync program when you are creating a new share by clicking on "Add a Sync Folder" and are adding a secret word for this share. Because if btsync has problems creating that folder (file permissions do not allow it for the directory where you are trying to create this folder), then it will display you an error message and you will be able to fix the permissions and then click again. Furthermore, btsync will create the directory with correct file permissions and ownership to allow it to save files in this directory.

If you have manually created the folder where you want to download and entered a secret and selected that folder, but (some) files do not download for some reason, you need to make sure that the folder you have created is readable and writable by btsync and the user account which runs the BTSync program (btsync.exe on Windows Task manager -> Processes).

In Windows explorer, right mouse click on that folder and in Properties -> Security, make sure that your user account has Write permissions for this folder.

The same thing is applicable if you work with Linux. That folder should belong to account btsync and group btsync should have the write permissions for it. It is best to create the folder from the BTSync GUI while initially creating this folder in your btsync collection. Simply copy paste the folder name from Linux, enter secret word and hit Create folder button. If btsync could not create that folder because of permissions, it will tell you so. In that case, you need to change the permissions in the parent folder, change its group to btsync and make it writable by the btsync group. Also, set the SGID bit (file mode 2775 - r/w/x for user, and group).

If you create that folder by hand from Linux terminal or from explorer, then make sure its permissions and ownership are set as above. That folder should belong to btsync group. Otherwise, btsync may not be able to start downloading it because it does not have the write permissions for it.

The detailed instructions for Linux could be found here:

How To Use BitTorrent Sync
https://www.digitalocean.com/community/articles/how-to-use-bittorrent-sync-to-synchronize-directories-in-ubuntu-12-04

If you have any problems, send the email to here: preciseinfo [@] mail [.] ru.

If time is set incorrectly on your comp

Time has to be set correctly on your computer, including the time zone. Otherwise, if it differs from the time set on the seeder computers by more than 10 minutes, then btsync can not sync your computer, and you will be informed about it next to device name of the seeder with the message:

Sync stopped: device time time difference more than 600 seconds

Most likely, your time zone is not set correctly or may be your time is not set correctly. Could be any one of those. So that your local time in combination with time zone set you have chosen, does not correspond to global GMT/UTC time standard.

In order to find out how much is your time off from other nodes in a swarm, you need to turn on the logging. See: Enabling logging.

Then, open your log file in your text editor and search for a string "Merge: timediff with remote peer" (NNNN is more than allowed 600, aborting). It will tell you how much your time is off with each separate node in the swarm, which you are trying to contact (+/-). And so, there would be necessary for you to change your time zone (most likely) or the time by that amount in order to decrease that difference to within 600 seconds.

For example, you find a message in your log that says:

"Merge: timediff with remote peer -3909".

This means that YOUR time is LESS than time of the remote by 1:05:09 [HH:MM:SS]. So, to be able to sync, you need to INCREASE your time by 1 hr. You can ignore the 5 mins 9 secs. Because it is within the allowable range.

Here is the log records for this situation:

[2014-03-20 22:36:23.542] Got id message from peer Mac mini (00657FB386A75D415E5684E3734368751E47D336) 1.2.91
[2014-03-20 22:36:23.542] Merge: timediff with remote peer -3909 is more than allowed 600, aborting

If your computer or system still makes the automatic and periodic requests to synchronize its clock with some external source, and, as soon as you change it manually, it is reset to that time, then you have to disable that automatic update.

So, you need to synchronize your time with one of the sources on the net and to make sure your time zone is set correctly. If, for some strange reason, your "normal" time zone is set in such a way that your UTC time differs by a number of hours, just try to change the time zone until the error message goes away and sync starts. Once you are synced, you can reset it back if you have to. But every time you want to update your sync files, set it back temporarily until sync completes.

I can't connect to other devices - I think it's a Router/Firewall issue?

... Also, if you're having trouble connecting to other devices over a VPN, try increasing the MTU size to at least 1500.

Finally, please make sure that all your devices are running the same version of BitTorrent Sync. For example, 1.0.x versions will not connect to 1.1.x versions, and 1.1.15 won't connect to 1.1.22, due to changes made to the protocol between these versions.

Sync isn't completing - not all my files are synced / file counts don't match!

There are a number of reasons why this may happen:

1. Sync won't sync files whilst they are currently open/locked/in use by other applications. Closing any open files in these applications should then allow Sync to complete the file transfer.
2. Some files may be excluded from syncing (check the exclusion rules in your .SyncIgnore files)
3. Files may have invalid timestamps. If a file has an invalid timestamp, for example, it appears to have been created in the future!, Sync may ignore the file.
4. Sync won't include a count/the size of files in .SyncArchive folders, where as your Operating System will - this often explains why Sync reports one file count/size, and your os reports a different set of values.
5. You modified or renamed some file (applicable to r/o nodes only) [not sure about this one though].

Restoring the original sync status of some files that won't sync any more

If you made some modifications or deletions of some files and, as a result, it does not seem that some files are updated when they are changed on the master machine, and you are running Linux, you can try to do following:

• From web GUI click on the gear icon below "Secret / QR" button.
• In Folder Preferences -> Advanced tab check the "Restore modified files to original version" checkbox. This should restore the original sync status of all the files in the database for this hash (share) and redownload the latest versions to your r/o node.
• It will then start resyncing all the files in the DB and, if there were some .SyncTemp files on a master machine (r/w), then it will go into an endless loop trying to resync all the files that had .SyncTemp version of them, which may never complete in some cases. You need to delete those files and then it will fully sync.
• After the files had resynced, you can unckeck that checkbox.

After this procedure your folder should show as fully synced and it should be exactly in the same state as the master (r/w) node.

Installing BTSync on mobile devices

Installing BTSync on mobile devices, such as tablets or phones is described here: Установка BTSync на мобильных устройствах и планшетах

BitTorrent Sync User Guide
(Opens in a new tab of your browser)

General notes

This document contains some text from the sources on the net that may provide the instructions that mention the older versions of BTSync.

Where to get BTSync?

This is the official download location with the latest version, if you decide to install any version above 1.3.109:

Downloads
http://www.getsync.com/download

Also, even "hotter" version may be available via "Sync General Discussion" forum on Bittorent forums. The thread with the latest version information is usually named "Latest Desktop Build 1.3.93" (for version 1.3.93).

So, in any instructions here that deal with installation, use the above locations and skip any instructions that tell you to download BTSync from elsewhere. There is one exception - Linux tuxpoldo version that uses the ppa. That version, strictly speaking is not "authorized" by BTSync but widely accepted as some kind of a standard. But read BTSync forums for more information.

Setting up Folder Properties

Folder Preferences in Folders -> right mouse click -> Show folder preferences choice -> Properties tab defines the parameters for a folder in terms of how it finds the other nodes, connects to them, what kind of tracker does it use and other things that are pretty important in specialized applications, to manage your security and configure other connection related issue.

Even though the Properties tab has only few parameters, but it presents you with many ways to configure your folders connection and security-wise. Some issues and settings are quite complicated in their effects to describe in few sentences, but we can still try.

Let us go through each individual parameter, as presented by RomanZ of BT Sync Support.

Search LAN

Search DHT network

Well, this one is a tough cracker. The DHT protocol is pretty complicated and is based on principles that may not be easy to understand to an ordinary user.

Basically, it has to do with the issues of how you find other nodes on some share. There are different ways to do it. One of them is when you know at least one node with static IP and predefined, fixed port number and every node on some share connects to it as they would do to a regular tracker. In effect, that node becomes a private tracker.

For example, when you click on "Use tracker server", it tells BTSync to use predefined and hardwired BT trackers with known IP:port addresses. As of March 22, 2014, there are 3 BT trackers (for redundancy purposes) that will be contacted to to get the list of all peers for a given share (sync folder or a "key").

In that case, BTSync is guaranteed to provide the connections between ALL the nodes that have contacted the BT tracker. When some node contacts the tracker with a peer request, tracker returns a list of ALL peers seen for this share. That means that they do not have to contact or search for each other, but simply contact a centralized tracker list. We'll talk about it more in the "Use tracker server" parameter description.

The other way to discover a list of all the nodes for some share is to use the tracker-less DHT network. Again, the DHT protocol is pretty complex to describe here, but the basic idea is that you send out the UDP broadcast packets (to the whole world), or, more precisely, to virtual DHT network, and every node running the BTSync and has "Search DHT network" parameter enabled, becomes a member of global DHT network.

But not all the members of the DHT network necessarily have a list of peer nodes for a given share. That would be too much load on them and would slow down the network considerably. So, DHT has a concept of the "closest address", and each node that has this "closest address" will respond to the DHT requests. Eventually, the node that has the EXACT peer list for a given share will respond and send out the list of all the nodes that are either peers on some share, or have been participating on it recently (which times out eventually).

In this arrangement, you are not dependent on the CENTRALIZED tracker and, therefore, that tracker can not possibly interfere in your ability to discover all other peers if tracker decides NOT to respond and provide the list of peers, possibly on selective basis, to the specific host, being blocked, for whatever reasons. If tracker does not provide the list to your node, then you won't see the other peers. There are many aspects to this, but the basic idea is that the tracker HAS the ability to control which nodes receive which lists. Whether they will actually start interfering this way or not is irrelevant.

What does it mean the DHT NETWORK?

Comment: This means that if you enable the DHT, you will store SOME node lists (distribution tables) on your own node. Those lists will be only the "closest" nodes to this share (closest does NOT mean closest physically. It means closest in terms of hash code, or "the name of share" and has nothing to do with physical location of nodes. DHT protocol scrambles the hashes in order to randomize the names and then simply uses numerical comparison to find the nodes that are "closest". This is done for the purpose of distributing DHT traffic more or less equally across the DHT network.).

This is also an optimization of the protocol to make sure all participating members of DHT network do not store the whole world' on their nodes, but store only those lists that at least have a chance to be "close enough" in terms of share hash. But all their stored lists are not necessarily guaranteed to be the EXACT matches for some share. And then it gets even more esoteric then this :).

There is an interesting aspect to peer discovery. Once you know at least one peer for some share, that peer ALSO provides you with the list of ALL the peers it knows about. The net effect is that as soon as you discover the 1st node, the process of discovery explodes and you very quickly find all the other nodes for this share by directly contacting them, without even DHT network involvement, and they will provide you with the lists of peers that THEY know about. So, the process of further node discovery after the initial one, goes really fast and is very efficient. Every single node very quickly discovers every single other node on this share and every single one of them knows all others.

So, when new nodes join the DHT network and it provides them the address of at least one peer, from then on, they can very quickly get the list of all other participating nodes without any further contacts to the DHT servers.

It should be noted that DHT is applicable and works for ANY node, regardless of whether it is r/w or r/o. Every node that has the "Search DHT network" checkbox enabled becomes a participant in the network and it may make the requests to the DHT network and also becomes a DHT server that provides the list of peers to other nodes that also have this option enabled. And it also provides the lists of the "closest" members of DHT network, and not only the list of nodes involved in this particular share. So, there is certain overhead associated with DHT in terms of traffic. About 5 to 10% of your bandwidth may go exclusively on servicing the DHT requests. Again, the DHT protocol is pretty complicated, even for the professionals to understand all the nasty details of how it all works, but this is a basic operational idea about it.

So, "the bottom line" is: if you want to make sure you will discover the maximum number of peers, (in case you have enabled the BT tracker and it decides to interfere and not to provide you with the list of peers), simply enable the "Search DHT network" feature. It won't hurt anything besides a little wasted bandwidth.

It has to be noted though that if you enable DHT, security-wise, you expose yourself more than if you only had your own trackers (see "Use predefined hosts" description). Because you broadcast your requests and send out the responses to the whole world, basically to anyone who is participating in the DHT network. But security related issues are even more complicated if you look at it from different angles. So, we are not going to talk about it here.

But in that case, you will be able to discover other peers only if at least one more node on this share also has his DHT turned on. In other words, if you are the only one in the swarm that has its DHT turned on, then it won't help it to discover any other nodes unless it also has the "Use tracker server" option enabled.

See also: Peer discovery principles

Use predefined hosts

This parameter is basically the same thing as "Use tracker server". Except you can disable the BT tracker for whatever reason, such as security, and, instead, use your own private tracker for this folder. So, as far as tracking, it behaves like one. But there is a difference: trackers do not actually have to have the data for the share, but predefined hosts do.

Basically, predefined hosts are no different than any other node in the swarm and they behave in a semi-tracker way not because of some particular behavior, but by the sheer fact that ANY node shares the list of all nodes it is connected to, even though it does not behave like a tracker in terms of discovery of other nodes. It simply shares the list of nodes it knows about, just like any other node.

So, the difference between the tracker and any other node is that you know its IP/name and port, and so any node that knows about it, can contact it as to a "well known address".

The only thing to remember is that you need at least one other node in this swarm that also enables this option and have at least one "predefined host" on their list. If you are the only one that has a predefined host and other nodes can not "see" you for whatever reason, then your server becomes virtually useless. But, if at least one of them has "Use tracker server" enabled, then the result will be joined in terms of total number of nodes known in the swarm to all its members.

Therefore, even if BT tracker does not distribute the full list of all the nodes in a swarm to some nodes for whatever reason, but still distributes at least some of them to at least some nodes, then your "predefined host" becomes your guarantee that all the other nodes will be discovered by all other nodes.

For creating the secure private networks, every single node in your network has to have "Use tracker server" disabled and "Use predefined hosts" enabled. In that case, all their predefined hosts will also serve as private trackers. Also, you need to turn off "Search DHT network", so your "footprint" on the net is minimized and you are not visible to others on the net (unless, of course, BTSync will transmit some information from your host even without your knowledge. Whether they do it in fact or not is not significant in terms of security. But you can prevent them by number of ways, such as a firewall blocking ALL the BT trackers or via other means, which is a separate subject of discussion).

So, let us see how it works. First of all, this is what BTSync team member has to say about it:

What it means to me is that a predefined host HAS TO HAVE the data for a share, unlike the DHT discovery process. So, it can not serve as purely a tracker. Pure trackers do not need the actual data for the share. But predefined host does act like a tracker ALSO, in a sense that through it you can discover all other nodes in the swarm that are connected to it.

Secondly, what it means is that every host in a swarm has a list of peers it is connected to and can share that list with any other node in the swarm that are known to it. That is the meaning of "Every host is a little bit tracker".

Chaining effect

Let us assume that we have a private network with 5 nodes that are interested in participating on the same share. How would they find each other?

If NONE of them had a single predefined host, then they would not be able to find each other (assuming they do not use either BT tracker or participate in DHT network).

Let us say that host 1 adds a predefined host with IP/name of the host 2. Now, both of these hosts can connect. But all other hosts still can not find them. Then, host 3 adds either host 1 or host 2 to his predefined host list, no matter which one. Now we have 3 nodes participating. The node 4 can add any one of these 3 to his predefined host list, no matter which one. The same thing with node 5.

This means that it is not necessary to add every single node to ones predefined host list. As long as there is connection from at least one of its predefined hosts to the 3rd one, every single of these hosts become visible to each other. Sure, the most reliable way is to add every single host you know in the swarm with this share to your predefined host list, but it is not an absolute requirement.

It also means that in order for you to create a private swarm, you don't need to enter every single host as predefined host. You may simply use the same single host to serve as connector between all the nodes in the swarm. So, all that every node needs to know is a single predefined host that is ideally the same for all of them. But it does not necessarily have to be the same host for all of them. As long as it can chain to the 3rd host, that is enough.

But it is advisable to have more than one predefined host for redundancy purpose. The more predefined hosts you have on your list, the more reliable your network becomes. But they do not have to be exactly the same hosts on all the nodes.

Use relay server when required

Well, in few words, relay server is basically a proxy server.

Here's what RomanZ of BT Sync Support has to say about it answering various questions:

Use tracker server

Here's what RomanZ of BT Sync Support has to say about it answering various questions:

Setting up private secure networks

Well, the questions of security is a huge issue to discuss. It turns out that even if you encrypt your traffic, the encryption algorithm may not be as secure as you might think. In particular, it has been recently revealed that one of the major players in the SSL business, allegedly as a result of their cooperation or funding from the NSA, has released an encryption scheme that can be easily cracked. Because their random generator is in fact not as random as it should be. This, in turn, implies that such an algorithm may be cracked relatively easily by brute force approach. But this is a long subject to discuss.

Nevertheless, here is an interesting quote with plenty of ramifications, which leave little doubt that BT is indeed keeping this issue silent and does not comment on it instead of making a public statement:

So, some questions arise naturally.

1) Why BT never commented on that thread?

2) Why have they never made a public statement on this issue?

3) Why did they delete that thread? And this one is probably the most revealing one of all, even though it seems to say nothing. But, sometimes nothing may turn into everything...

If you "put two and two together", the answer becomes simply inevitable...

And about the simplest way for the NSA to snoop on what you transfer is not necessarily by decrypting the traffic, but merely knowing the "secret words". At that point, they can simply look at your data directly, without even wasting the resources on decrypting it, unless the data itself is encrypted with a reliable algorithm. And that is the easiest thing in the world.

This, in turn, simply means that BT itself does not even have bother to look at your traffic. The very fact they know your hashes is enough. And if BT knows them, then how much of a difficulty would it be for the NSA and other agencies to also know it?

Nevertheless, besides the issues of encryption and cracking in general, regardless of how secure BTSync claims to be, there are some things you can do if you are interested in setting up a secure private network that would leave no footprints of itself on the net beyond the fact that it exchanges the information on the net.

If you have a number of hosts on the net with fixed IP and you are interested in exchanging the information only between them, which is not available to general public, then it is possible to do it with BTSync. It has several configuration parameters that allow you to enable or disable some features.

For example, even though BT claims that they do not collect any information on the transfers that are managed through their tracker(s), still, they have enough information and are capable of not propagating the information between the hosts. Whether they would actually do it or not is irrelevant and quite arguable from several angles. But the fact is: if you rely exclusively on BT trackers, then there is no guarantee that you will see any or all hosts that actually try to connect to your share if BT does not propagate some traffic.

This problem can be solved on private networks, especially if you know the IP or domain names of your hosts. But for dynamic DNS access things become more difficult and in some cases impossible.

So, how do you create a private and secure network that is not going to inform and rely on BTSync trackers for transfers?

Well, there is a concept of "predefined hosts" in BTSync in Folders -> Folder Preferences -> Properties. If you check the "Use predefined hosts" checkbox, then you can enter a list of host IPs or host names and ports to be used to communicate between the hosts directly. In that case, you don't need the BT tracker. And you do not need to do the DHT broadcasts to discover other interested hosts. And you don't need the relays to communicate with each other.

This means that you virtually leave no footprint on the net while transferring the information between those hosts.

In this case, you simply enter the names or IPs of all the hosts on your private network and they all will be able to transfer the information between each other without leaving any footprint on the net, and BT tracker won't even know you exist. Because you do not do any DHT broadcasts and all your hashes and shares are known only to your hosts.

As to "Use relay server when required", all the ramifications are not quite clear. Logically, the relay means that all the traffic flows through them, not only the connection information, in which case, if the encryption algorithms in BTSync are not that trustworthy, then, technically, your traffic can be relatively easily decrypted by the NSA super-computer, for example, via brute force method.

On Windows

So, on Windows, your settings for various checkboxes would be:

• "Use relay server when required" - unchecked.
• "Use tracker server" - unchecked.
• "Search DHT network" - unchecked.

So, the tracker would not know about you (but who knows what really happens "under the hood") unless you enable the DHT network, so the tracker will not be able to read your broadcasts. Otherwise, if you rely on tracker, it might decide not to present the lists of connected nodes and so they won't see each other.

On Linux

On Linux, things are configured via configuration files. The default location on Ubuntu is /etc/btsync/debconf-default.conf

Its location is specified on the command line to btsync process:

~/btsync/install/btsync --nodaemon --log file --config /etc/btsync/debconf-default.conf

Apparently I can't see anything wrong in your configuration file. But you must make sure, that the firewall of the server does not block the required connections. Since you have no chance to see anything, I would suggest to make a backup copy of your configuration file, remove completely the shared_folders section and activate the web UI in order to get a better overview of what is happening (your shared folder will still exist, since btsync has stored it into its internal database).

After you solved your problem, you can deactivate the web UI and write the shared_folders section back.

[Because if you have the "shared_folders" section defined in your config, you won't see those shares in the web UI.]

Here an example of one of my config files: (lines with leading // means a comment and can be used to comment out certain parameter settings.)

//!/usr/lib/btsync/btsync-daemon --nodaemon --config
// or, if you use the btsync version provided in xxx.gz file, then use this instead:
// ~/location_of_btsync_binary/btsync

// This btsync instance provides sync services for
// all common files in the YeaSoft Server Grid
{
	"device_name" : "yeasoft-gate2 - Server Sync",

// "storage_path" is location of btsync configuration files, logs and databases
// for all currently running shares, default on Ubuntu: /var/lib/btsync
	"storage_path" : "/var/lib/btsync-serversync",

// If "listening_port" is set to 0, then every time you start btsync it will use a random port,
// which you do not want if you want to specify the hosts directly
// and avoid using btsync tracker
	"listening_port" : 22144,
	"check_for_updates" : false,
	"use_upnp" : false,
	"download_limit" : 0,
	"upload_limit" : 0,
	"lan_encrypt_data": true,
	"lan_use_tcp" : true,
	"rate_limit_local_peers" : false,
	"webui" : {
//		"listen" : "10.65.0.16:8889",
//		"login" : "Administrator",
//		"password" : "MyTestPassword"
	}
	,

	"shared_folders" : [ {
		"secret" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
// dir is a top level directory of your data folder
		"dir" : "/mnt/data/tftproot/yeaboot",
		"use_relay_server" : false,
// If "use_tracker" : false and "use_dht" : false, then BT tracker
// won't even see your node (if you are concerned with security issues) 
		"use_tracker" : false,
// DHT allows a discovery of other nodes without btsync tracker
		"use_dht" : false,
		"search_lan" : true,
		"use_sync_trash" : true,
		"known_hosts" : [
//			"10.65.0.16:22144",
			"10.65.4.1:22144",
			"10.65.6.1:22144",
			"10.65.8.1:22144",
			"10.65.16.1:22144",
			"10.65.18.1:22144"
		]
	}
	]
}

As you can see, I made the same test: the web UI is disabled with comments... If you activate it, you have to comment out the shared folder section. In my special case, I disabled also all external helpers/relays and work only with predefined hosts.

(tuxpoldo, Advanced Member)

Installing BitTorrent Sync on the Raspberry Pi (Linux)

[Note: these instructions are generally applicable to any Linux system.]

NOTE: I am using Raspbian Wheezy, but I suspect these ideas will translate to other operating systems.

Unless you have a large SD in your Pi you will probably want to use an external drive for your sync location. I will be using a USB thumb drive. You may need to format your thumb drive as EXT4 if you are having issues. (WARNING! This will erase all data on your drive)

df -h                               # find your drive here, e.g. `sda1`
sudo umount /dev/sda1               # replace sda1 with your drive name !
sudo mkfs.ext4 /dev/sda1 -L BTSync  # replace sda1 with your drive name !

Now all you have to do is launch the btsync application and you will be up and running!

cd ~/.btsync
sudo ./btsync  # can be killed with `sudo killall btsync`

If you see the following output btsync is properly running.

BitTorrent Sync forked to background. pid = 3003

Navigate your browser to Your-Pi-IP-Address:8888/gui to manage the btsync process. To add the thumb drive select "Add Folder" and navigate to /media/BTSync. You will need to generate a secret as this is the first time you are adding the folder.

Other Devices

Now go download the Android app and/or the desktop app and connect them using the secret you just generated It's that simple! Any change on any device will be synced across all online devices. If you keep your Pi online it will store and push the most up to date content as your other devices go on and off line.

Start at Boot

You may want to set btsync to start when you boot your Raspberry Pi. To do that we will place a script in /etc/init.d/ and then register it with update-rc.d.

sudo nano /etc/init.d/btsync

Paste the following code in the script

#! /bin/sh
# /etc/init.d/btsync
#

# Carry out specific functions when asked to by the system
case "$1" in
start)
    /home/pi/.btsync/btsync
    ;;
stop)
    killall btsync
    ;;
*)
    echo "Usage: /etc/init.d/btsync {start|stop}"
    exit 1
    ;;
esac

exit 0

Then change the permissions, test, and register it to run at boot:

sudo chmod 755 /etc/init.d/btsync
sudo /etc/init.d/btsync start       # test that the script starts
sudo /etc/init.d/btsync stop        # test that the script stops
sudo update-rc.d btsync defaults

Password Protect Web Interface

If you expose your web interface to the outside world (or if you don't trust your friends) you are going to want to password protect it. This can be done with a btsync config file passed to the executable at runtime with the --config flag. First, use btsync to generate a sample config file, modify it to fit your needs, and restart the process.

cd ~/.btsync
./btsync --dump-sample-config > btsync.conf
# browse the sample config file and change what you want
sudo killall btsync
sudo ./btsync --config btsync.conf

HINT: Use jsonlint to validate your config file if btsync complains. Also make sure to modify the /etc/init.d/btsync script to use the config file as well.

Questions? Hit me up on twitter @jackminardi

You can also follow the discussion on HN

Source:

Replace Dropbox with BitTorrent Sync and a Raspberry Pi
http://jack.minardi.org/raspberry_pi/replace-dropbox-with-bittorrent-sync-and-a-raspberry-pi/

Configuration

Depending on how you installed btsync and how you are running it, some things will be different. Typically, you do the fresh install from some package, which performs all the default directory creation and installs the default scripts and some configuration files.

[Later on, you may download the btsync update, which contains executable only version, (such as btsync_i386-1.2.91.tar.gz for version 1.2.91, the latest version as of Mar 7, 2014, which does not include any initial install/configure files and should not be normally used as initial install, unless you are desperate.]

If you installed btsync via some package, like xxx.deb, then it should have created the directory for the btsync databases and configuration files and installed the default /etc/init.d/btsync init script for you, and you can start/stop btsync by /etc/init.d/btsync [start|stop|restart] shell script, which will automatically provide the correct arguments to btsync process when it is started.

See also: BTSync versions: user vs. server

You can also provide your own configuration file on the command line to btsync if you are running it manually and not via init script.

BTSync uses the configuration files that contain user configurable parameters. If you installed btsync from some package, like .deb, your default configuration and/or btsync data files (not the location of your sync folder, but the location of btsync database and configuration files) will be located in the "storage_path" parameter of your default or custom made configuration file.

[from /etc/btsync/debconf-default.conf]
// "storage_path" is location of btsync configuration files, logs and databases
// for all currently running shares, default on Ubuntu: /var/lib/btsync
"storage_path" : "/var/lib/btsync",

To get the location of your currently running configuration file, use ps -ef | grep bts and look at --config parameter to the btsync process (default is /etc/btsync/debconf-default.conf).

The location of the config file depends on how you installed and are running the btsync and whether you have /etc/init.d/btsync init file (Ubuntu), which is used to start/stop btsync after reboot automatically.

See also:
Configuration file contents and meaning

How To Use BitTorrent Sync to Synchronize Directories in Ubuntu 12.04

Introduction

Syncing folders and files between computers and devices can be done in many different ways. One method for automatically syncing content is BitTorrent Sync. BitTorrent Sync is a method of synchronizing content based on the popular BitTorrent protocol for file sharing.

Unlike traditional BitTorrent, files shared using BitTorrent Sync are encrypted and access is restricted based on a shared secret that is auto-generated. While BitTorrent proper is often used to distribute files in a public way, BitTorrent Sync is often used as a private method to sync and share files between devices due to its added security measures.

In this guide, we will discuss how to install and configure BitTorrent Sync on two Ubuntu 12.04 VPS instances.

Install BitTorrent Sync

To begin, we will need to install BitTorrent Sync on both of our Ubuntu 12.04 instances. If you would like to install BitTorrent Sync on your local computer to allow you to sync with your server, you can find the binary packages here.

BitTorrent Sync is relatively easy to install on Ubuntu 12.04, but it is not included in the default repositories. We can use a PPA (personal package archive) so that we can have access to a maintained BitTorrent Sync repository and manage it with our normal apt tools.

Ubuntu 12.04 includes the PPA tools in a package called python-software-properties, which we can download through apt:

sudo apt-get update
sudo apt-get install python-software-properties

After this is installed, we can add the PPA that contains updated Ubuntu packages:

sudo add-apt-repository ppa:tuxpoldo/btsync

Press "enter" to add the new PPA.

Once the new repository is added, we should update apt to build a package index for the new source, and then install the BitTorrent Sync software:

sudo apt-get update
sudo apt-get install btsync nginx

You will be asked quite a few questions in prompts when you attempt to install. For now, press ENTER through all of the prompts. We will be reconfiguring our services momentarily in a more in-depth manner.

Configure BitTorrent Sync

Now that the software is installed, we're actually going to run the configuration script that prompts us for values a second time. This time, however, we will have access to additional options that we require for our purposes.

To run the script again, this time choosing our settings, type this on each server:

sudo dpkg-reconfigure btsync

The configuration directory for the service is:

/etc/btsync

Do not edit the config file generated by the menu system by hand. You can, however, copy the configuration to use as a template for another configuration if you'd like to adjust details not covered in the menu configuration.

This will run you through even more prompts than during the initial installation. For the most part, we will be going with the default values and you can just press ENTER.

Below, I've outlined the values that you need to configure:

• Web Interface Bind IP Address: 127.0.0.1
• The username for accessing the web interface: [Choose whatever you would like. We will keep the admin account in this example.]
• The password for accessing the web interface: [Choose whatever you would like. We will be using password for demonstration purposes.]
• Umask value to set for the daemon: 002

As you can see, for the vast majority of settings, we can accept the defaults. The above choices though are very important. If you mis-configure these, run the command again to correct your selections.

Configure SSL Front-end to the BitTorrent Sync Web Interface

Now, we have BitTorrent Sync set up for the most part. We will set up our sync directories in a bit. But for now, we need to set up our nginx web server with SSL.

You may have noticed that we configured our web interface to only be available on the local loopback interface (127.0.0.1). This would normally mean that we would not have access to this when running BitTorrent Sync on a remote server.

We restricted access like this because, although the BitTorrent Sync traffic itself is encrypted, the traffic to the web interface is transmitted in plain text. This could allow anyone watching traffic between our server and local computer to see any communication sent between our machines.

We are going to set up nginx with SSL to proxy connections through SSL to our BitTorrent web interface. This will allow us to securely administer our BitTorrent Sync instance remotely.

Again, we will need to do all of these steps on both of our hosts.

Generate the SSL Certificate and Key

The first step towards getting this set up is to create a directory to hold our SSL certificate and key. We'll do this under the nginx configuration directory hierarchy:

sudo mkdir /etc/nginx/ssl

Now, we can create our SSL certificate and key in a single motion by issuing this command:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/nginx.key -out /etc/nginx/ssl/nginx.crt

You will be asked to fill out some information for your certificate. Fill out the fields as best as you can. The only one that really matters is this one:

Common Name (e.g. server FQDN or YOUR name) []:

In this field, enter your server's domain name or public IP address.

Configure Nginx to Encrypt Traffic with SSL and Pass to BitTorrent Sync

Now, we can configure our nginx server blocks to use our SSL certificates when communicating with remote clients. It will then the information to our BitTorrent Sync web interface listening on the local interface.

We will leave the default nginx server block file intact in case you need to use this in the future. Since BitTorrent Sync operates on port "8888" by default, we will use this as the front-end SSL port as well.

Create a new server block file by opening a new file with sudo privileges in your editor:

sudo nano /etc/nginx/sites-available/btsync

Inside, we need the to add the following lines:

server {
    listen server_domain_or_IP:8888 ssl;
    server_name server_domain_or_IP;

    access_log /var/log/nginx/access.log;

    ssl_certificate /etc/nginx/ssl/nginx.crt;
    ssl_certificate_key /etc/nginx/ssl/nginx.key;

    location / {
        proxy_pass http://127.0.0.1:8888;
    }
}

Make sure you change the red text to your server's domain name or public IP address. This will tell nginx to bind to the same port that the BitTorrent Sync web interface is using on the local interface. The difference is that nginx will use the public address and require SSL.

It will use the SSL certificate that we created to encrypt the traffic to the client. It will then pass it to the BitTorrent Sync interface. In this way, the traffic between the server and the client will be encrypted, but the BitTorrent Sync interface will operate as if we were accessing it from the server itself.

When you are finished, save and close the file.

Now, we just need to link the file so that it will be enabled:

sudo ln -s /etc/nginx/sites-available/btsync /etc/nginx/sites-enabled/

We can now restart the service to implement our changes:

sudo service nginx restart

Make sure you go through these procedures on each of your two servers.

How To Configure Shared Folders

In order to sync folders with BitTorrent Sync, the btsync user or group needs write access to the folders. There are a few different ways to achieve this.

First, let's create the folder:

sudo mkdir /shared

We need to complete these steps on both of your VPS instances that will be syncing.

Giving the btsync Process Full Ownership

One way to give the btsync user access is to simply give ownership of the folder to the btsync user:

sudo chown btsync:btsync /shared

This will allow the BitTorrent Sync service to correctly serve the contents of this directory, but we are not able to write to this as a normal user. This may be what you want, but it usually is not.

Give Your Normal User Ownership and the btsync Process Group Ownership

If you have only one normal user on the system, you can give that user ownership of the folder and give the btsync group ownership of the folder:

sudo chown your_user:btsync /shared

You would then have to give the group write permissions:

sudo chmod 775 /shared

This will allow the btsync service to access the folder. However, any files created inside the directory will be owned by your user and group.

For instance, if we add a file to this folder called test, it would be completely owned by our user:

cd /shared
touch test
ls -l

-rw-rw-r-- 1 your_user your_user 6 Jan 16 14:36 test

This will cause problems for the syncing, because the btsync process cannot modify the file. We want to give it the same group permissions as the folder it is in so that the process has write access.

We can do this by setting the SGID bit on the directory. This will set the group on all new files created inside of the directory to the group of the directory itself. This will allow proper write access to modify things:

sudo chmod g+s /shared

Now, when we create a file, it will be given the permissions of the directory:

cd /shared
touch test2
ls -l

-rw-rw-r-- 1 your_user your_user 6 Jan 16 14:36 test
-rw-rw-r-- 1 your_user btsync 0 Jan 16 14:41 test2

This goes a long way towards getting the appropriate functionality, but it isn't quite right yet.

Delete the test files we created before continuing:

rm /shared/test*

Add Your User to the btsync Group and Give the Root User Ownership

The method above works somewhat, but files that are transferred with BitTorrent Sync are owned by the btsync user and group. This means that currently, any files synced by the service will not be editable by us.

We can change this by adding our user to the btsync group. This will allow us to modify files that are writeable by the btsync group, which is what we want.

Add any username that you wish to be able to use btsync to the btsync group:

sudo usermod -a -G btsync your_user

This will append the btsync group to your user's group definition. This will allow you to edit files created in the shared folder by the btsync process.

However, the directory is still owned by our user, which is not a good way of going about things if we have multiple users on the system. We should transfer ownership to the root user to avoid regular users changing folder settings. We should also allow group write permissions so that anyone in the btsync group can add content:

sudo chown root:btsync /shared
sudo chmod g+w /shared

You may have to log out and log back in for these changes to take affect.

In the end, the process for creating a shared folder that works well for BitTorrent Sync goes something like this:

sudo mkdir shared_folder
sudo chown root:btsync shared_folder
sudo chmod 2775 shared_folder
sudo usermod -a -G btsync your_user

The first "2" in the chmod command sets the SGID bit in the same way that the "g+s" did previously. This is just a more succinct way of combining these commands.

Accessing the BitTorrent Sync Web Interface

Now that we have a folder that is configured appropriately for BitTorrent Sync sharing, we can access the web interface to add our folder to begin syncing.

Again, we will have to do this on each of the servers that we wish to configure syncing on.

Access the web interface by going to your droplet's IP address, followed by the port you configured during install. By default, this is 8888:

your_ip_or_domain:8888

You will have to sign in using the credentials you configured during installation. The default username is admin if you didn't change it.

You will be presented with a rather simple interface to start:

Adding the Shared Folder to the First Droplet

Now that we are in our web interface, we can add our shared folder so that the btsync process can register it.

On your first machine, click on the "Add Folder" button in the upper-right corner. This will bring up a box that allows you to select a directory to share:

Find the folder that you configured for sharing. In our case, this is the /shared folder. Once you have a folder selected, you should click on the "Generate" button to generate a secret for the folder.

The secret that is generated allows you to sync this folder with another instance of BitTorrent Sync. This unique value is basically a password to allow the two services to connect to each other.

Click the "Add" button when you have completed these steps. This will add our folder to the interface and give you some buttons on the side to manage this folder.

Right now, we are only interested in the "Secret/QR" button. Click this to bring up a box that will allow you to choose how you want to share this folder.

We can grant access to the folder with read/write permissions through "Full access". If we only want to sync one way, like a backup, we can allow only read access. The secrets that are provided for each kind of access differ.

Copy the secret for the type of access you want. We will be using full access in this tutorial.

Adding the Shared Folder and Secret to the Second Droplet

Now that we have our secret from our first VPS, we can add the shared folder that we created on our second VPS and use the secret to sync our files.

First, you must log into the web interface just like you did with the first server:

second_ip_or_domain:8888

Once you are to the interface for the second server, click the "Add Folder" button again.

Add the locally created shared folder.

This time, instead of clicking the "Generate" button, we will paste the secret from the other instance into the "Secret" box:

Click the "Add" button to create the share.

After a moment, in both web interfaces, you should see some new information in the "Connected devices and status" section:

This means that our two instances of BitTorrent Sync have found each other! The icon in front means that we have given full access and files will be synced in both directions.

Test Syncing

Now that we have configured syncing, let's test to see if it works.

On one of your servers (it doesn't matter which one if you configured full access), we will add some files to our shared folder.

As a user that has been given access to the btsync group, create some files in the shared directory:

cd /shared
touch file {1..10}

This will create 10 files in the shared directory. We can check that these have been given the appropriate permissions by typing:

ls -l

-rw-rw-r-- 1 your_user btsync 0 Jan 16 16:16 file1
-rw-rw-r-- 1 your_user btsync 0 Jan 16 16:16 file10
-rw-rw-r-- 1 your_user btsync 0 Jan 16 16:16 file2
-rw-rw-r-- 1 your_user btsync 0 Jan 16 16:16 file3
-rw-rw-r-- 1 your_user btsync 0 Jan 16 16:16 file4
. . .

As you can see, the files are owned by your user, but the group owner is btsync. This is exactly what we want.

If we check our other server after a few seconds, we should see our files in our shared directory!

cd /shared
ls -l

-rw-r--r-- 1 btsync btsync 0 Jan 16 16:16 file1
-rw-r--r-- 1 btsync btsync 0 Jan 16 16:16 file10
-rw-r--r-- 1 btsync btsync 0 Jan 16 16:16 file2
-rw-r--r-- 1 btsync btsync 0 Jan 16 16:16 file3
-rw-r--r-- 1 btsync btsync 0 Jan 16 16:16 file4

As you can see, the files are given to the btuser and group. This is because the service can't be sure that the original username exists on the second system.

The last step is to get the btsync daemon to automatically set the file permissions of the files that it syncs to be writeable by the btsync group. This is necessary if you are providing full access in order for your user to edit the files that it has synced.

We can do this in by reconfiguring the btsync daemon. This will open up a lot more options than we were given when we originally went through the configuration. Begin to reconfigure on both of your syncing machines by typing:

sudo dpkg-reconfigure btsync

You will run through the configuration menu, this time with many more questions. For the most part, it should automatically select either your previous selection, or the default choice for any previously unset parameters. One question that you'll have to remember not to skip is the password prompt.

The option that you are looking is the default umask for files synced by the daemon.

We can set the appropriate umask to create files that are writeable by both the owner and the group (which our user is a part of), by typing this:

002

Finish up the configuration and the daemon should restart automatically with the new settings. Once you have completed this task on both servers, you should be able to create a new file on one server, and it will be given correct writeable permissions on the second host:

touch /shared/write_test

On the second host once the file syncs, you'll see somethings like this:

-rw-rw-r-- 1 btsync btsync  0 Jan 30 10:44 write_test

In the web interface, you will not see that your files have been synced, because the files don't contain any actual data. If we add some content to our files, the interface will update to show how many files we have synced:

for item in /shared/file{1..10}; do echo "some content" > $item; done

Conclusion

Now that you have your servers configured, you can easily transfer files between them. You can also configure multiple folders to automatically sync. This can provide some interesting options for dealing with configuration files and such.

The application is fairly flexible in how to sync between multiple computers. You can create one-time secrets to ensure that no one shares access to your directories, you can share only with specific hosts, and sync between your mobile device. BitTorrent Sync provides an archive version control system through the .SyncArchive file in directories and can rate limit to ensure that you have bandwidth available for other applications.

Install BitTorrent Sync on a Headless Ubuntu Server

[Note: this information is applicable if you downloaded an update to btsync, which only contains the btsync executable and does not perform any initial configuration and does not include the default scripts and does not create any directories for you.]

$ mkdir ~/.btsync

$ cd ~/.btsync && wget -O - "http://syncapp.bittorrent.com/1.0.134/btsync_x64-1.0.134.tar.gz" | tar xzf -

$ cd ~/.btsync && wget -O - "http://syncapp.bittorrent.com/1.0.134/btsync_i386-1.0.134.tar.gz" | tar xzf -

$ ~/.btsync/btsync --help

$ ~/.btsync/btsync --dump-sample-config > ~/.btsync/sync.conf

$ ~/.btsync/btsync --generate-secret

$ ~/.btsync/btsync --get-ro-secret A1B2C3D4E5F6GHIJKLMNOPQRSTUVWXYZ

$ mkdir ~/BTSync && mkdir ~/BTSync/example.com

$ cp ~/.btsync/sync.conf ~/.btsync/sync.conf.orig
$ vi ~/.btsync/sync.conf

{ 
  "device_name": "web server",
  "listening_port" : 8888,
  "storage_path" : "/root/.btsync",
  "pid_file" : "/root/.btsync/btsync.pid",
  "check_for_updates" : false, 
  "use_upnp" : false,
  "download_limit" : 0,                       
  "upload_limit" : 0, 
  "webui" : {
    //"listen" : "0.0.0.0:8888",
    "login" : "admin",
    "password" : "password"
  },
  "shared_folders" :
  [
    {
      "secret" : "A1B2C3D4E5F6GHIJKLMNOPQRSTUVWXYZ",
      "dir" : "/root/BTSync/example.com",
      "use_relay_server" : false,
      "use_tracker" : false,
      "use_dht" : false,
      "search_lan" : false,
      "use_sync_trash" : false,
      "known_hosts" : 
      [
        "101.102.103.104:8888"
      ] 
    }
  ]
}

$ ~/.btsync/btsync --config ~/.btsync/sync.conf

By using this application, you agree to our Privacy Policy and Terms.

http://www.bittorrent.com/legal/privacy

http://www.bittorrent.com/legal/terms-of-use

BitTorrent Sync forked to background. pid = 27732

$ kill -15 27732

$ cat ~/.btsync/btsync.pid

$ ~/.btsync/btsync --config ~/.btsync/sync.conf

Source:
Install BitTorrent Sync on a Headless Ubuntu Server
http://artofsimplicity.co.uk/install-bittorrent-sync-on-a-headless-ubuntu-server/

Note to editors

It looks like we had a couple of edits lost as a result of one node's edit stepping on the other. There were actually 3 edits on Mar 22, 2014 and we do not have time now to investigate what exactly was edited. From the preliminary look, it seems that there were some things that were different. We do have the history, but it would be easier for the author of edit to restore his edit. Our hands are full at the moment.

Unfortunately, sometimes it gets pretty hot and lots of things happen, and it was probably one of those times. So, if the author of those edits could do the diff and see what he wanted to edit and then correct some things, we'll try not to touch this document for now.

May be, if you intend to do the edit not from the master node, but from some other node, say something like "nodename: Now editing BTSync detailed manual" or "nodename: ready to save changes of edit" or whatever you feel like saying in your Device name message in Devices tab.

Try to use Notepad++ editor if you are on Windows, or some other editor that notifies you of changes on disk so that your edits won't get lost. And before you hit "save" in your editor, make sure it won't step on the version of others (in Notepad++, switch from and app to another or minimize and maximize the edit window and take the appropriate action if you see the file was update while you were editing it).

If you find a better way to assure edits are not lost, please say so in "Edit History" section (see below).

One of the things that can be done to inform others that you are doing or intend to do edits, is to add your message to the section Edit History. We need to find the most effective way of working in this environment and everyone's comments or suggestions, and especially the solutions, are obviously valuable.

Major Problems

There are some problems we can see some people experience. The good news is: the solutions are available. The bad news is: they are there and there is nothing we can do about it, at least at this time, primarily because BTSync simply refuses to even recognize them as something real, and even less they are interested in actually fixing them even though we have provided the solutions and suggestions on how to handle it.

But they are not even interested in listening about them for some strange reason. Actually, some of their "reasons" could be understood. Their marketing efforts are directed primarily at distribution of psychosis inducing materials in modern "music" or "entertainment" business, the more appropriate name for which is zombification business, the square purpose of which is to disconnect people from their roots, from their culture and heritage and instead to zombify their minds with all sorts of boom-boom-boom, so they would happily shake their rear ends at some zombie party and would become essentially the happy idiots, easy to control and easy to impose all sorts of things on them and inculcate their minds with all sorts of violence, psychosis and hysteria, precisely in line with the satanic NWO agenda of taking over the world and creating a two class society "elite" and "robots" or slaves. All others are not needed in this scheme.

And most of these people are simply interested in syncing their mobile devices with their home and work computers so they could be zombified wherever they go. And in this setting, they do not deal with the issues of general public grade, because they know all their devices and can change things any time they want. But in general public setting, you simply have no contact with people that might have some problems syncing. That is the basic difference and this is probably the reason for their "attitude". As some say: "you need an attitude adjustment, pal".

So, we will describe those most common problems and provide you with solutions that should fix your situation, at least in most cases.

Major problems new users experiences are the following:

Sync stopped: device time time difference more than 600 seconds

("Sync stopped: device time time difference more than 600 seconds" error message in Devices tab next to some device.)

First of all, when you see this error message in Devices tab, it, basically, means that the time on your computer is not set correctly from the standpoint of other nodes on this share, and, therefore, it is impossible to determine which version of some file is actually newer and is to be downloaded. When some file is changed, there is a need to find out which version is actually newer.

But, it may turn out that in reality time on your computer is set correctly including the time zone, but there exists a different time zone setting in different places in the world depending on which particular source you use to get time and time zone information. Some of them may not be set correctly for your location.

Therefore, do not get scared seeing this error. These are just minor unpleasant things and the algorithms in BTSync need to be radically redesigned in respect to this issue. But you will still be able to resolve this issue and get in sync, even though the solution might not make much sense to you and you might not feel comfortable about it.

The "good news" is that the solution is available. The "bad news" - you might want to puke. But this is just a psychological issue. Do not worry, "we'll get them" and they will indeed fix this issue, sooner or later. We have already created so much noise about it on their forums and provided so much evidence, that we are like a bone in their throat. And there is basically nothing they can do but to finally recognize this problem as THEIRS, instead of blaming the users for it.

Basically, the solution for now is for you to enable logging and then find that node which gives you that error and notice the time difference between you and it and adjust your clock accordingly. Most likely, the difference will be +/- 1 hr. But we have seen cases with +/- 2 hrs.

There is no need to set your time exactly to the second to compensate for the difference between you and the other node. Because the normal acceptable range for error is +/- 10 minutes. You'll just have to increase/decrease your time by +/- 1 or 2 hrs.

This is one of the strangest errors quite common among new users. Basically, it has to do with the case when BTSync needs to decide which version of file is more recent in case some file was modified on one of the nodes and, as a result, there exist several versions of the same file.

This is probably one of the most unpleasant problems with BTSync, especially in general public application where you have no idea about the other nodes on the share and have to way of contacting them and telling them that their computer time is not set correctly.

Furthermore, it is possible that the computer clock and time zones are set correctly, according to your local regulations, but you still see that error. Because in some other place in the world they might not have a correct time zone for your location. And their update service provider does not have the correct time zone tables for your location.

Most often, your clock time will be different from the other nodes by +/- 1 hour. In rare cases it will be different by +/- 2 hrs. It has to do with the DLS time adjustment for different time zones as seen by your local system.

See: If time is set incorrectly on your comp

Incomplete downloads/uploads

This is probably one of the most confusing issues new users may experience. Generally, it has to do with user modifying, deleting or renaming some files on the r/o nodes.

According to present logic (as of 1.3.77) BTSync will neither update (download) or distribute any files that were modified, deleted or renamed. So, if you decide to edit some file on the r/o node directly, you will no longer receive its updates from the other nodes.

See: BTSync behavior table

Therefore, if you want to modify some file and still receive its updates, better make a copy of it and modify that one instead.

Furthermore, those nodes will be shown to the other nodes as though they still have to upload some size to them. And, probably the most unpleasant and utterly unintuitive effect would be that even if you manage to restore those files back to their original state, or restored the deleted or renamed files, they will still not be updated from other nodes, no matter what you do.

If that happens, about the only thing you can do to restore those files to their original sync state is to use the Restore option:

See: Restoring the original sync status of some files that won't sync any more

So, the proper way of handling all the files that you indeed do not want to be updated in the future is to use the .SyncIgnore file located in your main sync directory.

See: Using .SyncIgnore file to exclude some files from sync

Enabling logging

To get more detailed information about the activity of btsync or to identify any problems or observe what is going on "under the hood" as far as file transfers or connection attempts are concerned, you may enable the detailed logging.

On Windows

Logging can be enabled/disabled by right mouse button clicking on the btsync taskbar icon and selecting "Enable Debug logging". The log file is named sync.log and it will be located in C:\Users\your_accont_name\AppData\Roaming\BitTorrent Sync folder, which is a regular text file.

On Linux

Create a regular text file debug.txt in the directory specified by "storage_path" in your configuration file, that should contain the string FF (Default on Ubuntu /var/lib/btsync/debug.txt).

See Configuration for more information.

Then simply restart the btsync. Logs can get pretty big during the periods of high activity, so you probably want to disable it once you find whatever you were interested in. Just rename debug.txt into debug_XXX.txt or something like that and restart btsync again.

BTSync versions: user vs. server

First of all let's clarify one thing: both packages install the same BitTorrent Sync executable. Because of that, there are absolutely no differences between what BitTorrent Sync is able to do. The difference stays in the way, your machine handles BitTorrent Sync:

• If you install btsync-user (or the upcoming successor btsync-gui), you get an environment in which BitTorrent Sync is launched when you log in into your desktop as a process owned by you. When you logout, the process may be stopped (this is currently not happening with btsync-user, but it is a bug and may change in future...).

  This implies that the BitTorrent Sync process is able to read/write only files/directories on which YOU have write access. By default you work with a basic configuration file automatically created at each start.

  You can set advanced settings via the Web UI. There is also a possibility to use a custom configuration file in which you can specify everything you want.

  You can change settings in the Web UI, but at every restart these settings are overwritten by the settings you have specified into the configuration file.

• If you install btsync (the server package) one or more BitTorrent Sync processes are started on system startup. They run independently from the users that may log in to the desktop.

  Obviously they work also on systems with no desktop installed. For each of those BitTorrent Sync processes there is a configuration file located in /etc/btsync and each of this processes can run under different user credentials (that are coded into comments in their respective configuration files).

  Additionally there is the possibility to specify the UMASK for the process, if you want BitTorrent Sync to create/write files with a different UMASK as the standard one.

Let's summarize: for you it makes sense to use btsync instead of btsync-user if:

• You want BitTorrent Sync to run totally independently from any user logged into the system

• You may want to have more than one instance running in future (this is the case, if you want to permanently synchronize the files of several users of the system).

Yes. But first take a decision if the server package is really the right thing for you. If you say yes, I will explain here how you may migrate your system.

(tuxpoldo, Advanced Member)

Setting and changing file ownership and permissions

There are settings for both of these: during the first installation, debconf asks for the most important settings needed to install the daemon. If you perform a service reconfiguration, debconf asks for all available settings. If you perform a reconfiguration with

you will be asked for the credentials and there you can specify both a user and a group of your choice for the btsync daemon.

Additionally you will be asked also for a UMASK for the daemon. This setting determines with which mask files are created and written. In order to make all files and directories group writable, you have to specify 0002

One question: are you sure, that the server packages are the right ones for your use case? If your Linux machines are used as regular desktop machines with a user that interactively logs in and wants to synchronise directories owned by himself, maybe the packages for desktop usage btsync-user may be more suitable for you.

(tuxpoldo, Advanced Member)

Main principles of operation and conceptual framework

First of all, the central idea of BTSync, as seen by the author of this manual, is a concept of dynamic torrents. BTSync, "under the hood" is simply using torrents to perform the bulk of most important work as far as node updating and file transfers with the exception that the BTSync torrents are dynamic in nature and are used only temporarily while performing the transfer of specific files, and this is one of the central concepts of BTSync design.

The principle is this: regular torrents, as used by all sorts of bittorrent programs are static and include every file in some collection. Once created, the torrent can not change and it includes every single file of the entire collection. That is why torrents get "stale" and outdated. Because real world information forever changes. Files get updated, edited, extended added or removed and so on. If you look at some torrents that were published several years ago, there is a good chance that many of their files are not longer current or interesting, even though some other files are still current.

With dynamic and temporary torrents that are created on the fly and exist only for the duration of file transfer or update, you no longer have the aging problem. Furthermore, since the entire collection no longer consists of a single huge torrent, but consists of multiple and fresh torrents that always contain the freshest version of the files, the collection can now be extended, modified, or some files may be deleted and so on. As soon as program detects that some file is no longer current and was updated in the file system, it creates a new torrent and sends the updates of this files to the other nodes.

The "beauty" of this approach is that BTSync does not have to reinvent the wheel and can simply utilize the bulk of programming code from the existing torrent applications. There are many excellent open source torrent programs that have all the necessary issues worked out and have all the necessary features and abilities to perform the torrent based transfers, node discovery via torrent tracker servers or DHT protocol and so on.

All this means that you can simply use the standard torrent engine "under the hood" and add some relatively simple higher level logic and algorithms to detect the file changes and schedule the updates.

Processing efficiency optimization

BTSync utilizes the file transfer efficiency improvement for the small files. For small files you still need to generate a separate torrent, which takes time. When transfer starts, you still need to transfer the torrent file and then the data itself, which takes more network cycles and slow things down.

So, what BTSync does in this case is to bulk several smaller files into a single torrent and transfer it all at one go, like a single file. This significantly improves that transfer efficiency. As a result, we have observed some nodes updating at the speeds of over 20 mbit, which is excellent performance. BTSync claims that they could go at the speeds of up to 90 mbit on a LAN.

The concept of a key (so called "secret")

Every collection of information, or a share, as they call it, is identified with unique key (or "secret" as it is called by BTSync, which is incorrect and misleading since it is not a secret of any kind) for this collection.

The only and square purpose of a key is to uniquely identify some collection of information (share). It has nothing to do with any kind of "secrets" and it carries no "secret" meaning underneath, and it is not even encoded because there is nothing to encode, really and nothing to be kept "secret". It is simply a unique hash ID for the collection. Sure, it can be obfuscated further by re-encoding it for some specific purpose, such as to confuse things more for some undesired third parties. But that is beyond the point here.

The key consists of a random collection of ASCII characters and digits. It is generated via standard O/S provided GUID generation algorithms generating the globally unique identifier. This identifier is then processed and hashed in such a way that at the end what you have left of it is a hash string of around 30 characters in length consisting of capital letters and digits that uniquely identify some collection.

It has to be understood that the key is not applicable to any particular node. ANY node that has access to that key has access to that specific collection without any further actions from the user. That is the difference between, for example, the Syncthing approach where a key uniquely identifies the node, not some collection. So, when you get a key, you can contact some node and see all of its published collections it their list of collections. From then on, you may chose to add some of its collections to your node as well and will become a part of that collection, subject to updates and distribution.

When new collection is added, first of all, it gets indexed. The program goes through each file and calculates its hash value similar to a checksum. It then adds this file to its internal database and sets the initial state of various flags for that file, such as propagatable, updatable, physically present in the file system, deleted and no longer active and so on.

Once database is built, the program then contacts the tracker servers. As it stands right now with BTSync, the domain names of several BT trackers and relays are hardwired into the program and can not be changed or configured by users. Actually, more precisely, the program, besides having some trackers and relays hardwired into the program, also contacts the well known BT server and obtains the list of current tracker and relays from it in JSON format. This way, their tracker lists are always current and do not get outdated with time.

Side note:
Monopolizing the "entertainment" industry distribution market

Interestingly enough, it has been observed, at least on one occasion that even if you uncheck all the check boxes in folder properties that have to do with trackers or node discovery, such as "Use tracker server", "Search LAN", "Search DHT network" and do not enable "Use predefined hosts", the nodes STILL can discover each other. Which means that no matter what you do with your settings, you will not be able to avoid going through the BT trackers or relays.

It is suspected that BT has a strong commercial interest in promotion for the entertainment industry and probably has some contracts with the industry that allow them to get the piece of the pie by distributing the promotion "bundles", in particular, via BTSync. In this light, it is interesting to observe that even though BT claims that BTSync will remain "free" forever, they, nevertheless, keep the program source closed. Strange, isn't it? Since they do not make any money on sales of the program, then what could be the reason to keep the source code closed, besides some commercial interest?

Well, it appears that BT is trying to "corner" the entertainment industry distribution market and obtain the monopoly on distribution of various "bundles", which they release on a regular basis. Every few days they release a new "bundle". Question: why would they do that if they did not have a commercial interest in this deal? Well, they simply might have a contract in which they get a set percentage of sales of those products that are promoted and distributed via BTSync. So, by merely presenting such things as the count of transfers and total amount being transferred, it is possible to calculate the size of their share of the pie.

And that is precisely the reason they do not release the source code to BTSync. Because they are probably interested in monopolizing the whole market of distribution via net. If you look at their blogs, things may become "clear as a bell" to you.

Performing file updates for the share

BTSync has two separate mechanisms to detect file modifications, deletions and additions.

First mechanism is to utilize the operating system file system events, such as updates, deletions and creation of files. These events are delivered by the O/S nearly instantly and could be acted upon immediately for updates of all nodes that are currently on line for a given share. But the nodes that are not on line at the time of file system changes are updated according to second mechanism, file scanning.

BTSync performs file system scans on a periodic basis at intervals of 10 minutes as a default. During those scans BTSync recalculates the file hashes of all the files in the share and if it finds that some file hash has been changed, compared to what is stored in the internal database, it immediately places that file in the update queue to be transferred to all the nodes currently on line.

But file updates are performed conditionally and it is done in a different way for r/w and r/o nodes. For r/o nodes, file is scheduled for either download or upload only if that file still has the original distribution flags set appropriately. For example, if some file was deleted, renamed or modified in the file system, that file is not subject either for download from other nodes or distribution to other nodes. It becomes a private copy of this node, and that is precisely the reason we are seeing well over a half of all the nodes on one of our main shares showing as being in the incomplete state, while it is considered complete or "done with" by the node itself.

The master nodes (r/w) have detected some file changes and informed all other nodes on line that it has the new version of some file. But the r/o nodes refuse to download them because according to their view, those files are no longer updatable or propagatable, like they do not exist. That is why the master nodes add the sizes of those files to the total size of the amount remaining to upload to some node and these nodes show as incomplete, while, from the standpoint of those nodes, they ARE "complete", or, more precisely, do not exists from the standpoint of updates.

This is a flawed logic and BTSync will never work in a consistent and predictable manner unless this issue is resolved. This is known as database consistency problem, which means there exist multiple copies of the same data and those copies differ, and, since master nodes do not know and can not possibly know what's in the mind of the nodes that have modified the file, the issue remains unsolvable on logical grounds.

The file transfer phase

Once these relatively simple logic rules are followed and some file was scheduled for update, what remains is actual transfer phase. As mentioned above, the transfer is performed via standard torrent transfer engine.

Each file consists of pieces or blocks. During the block download stage, the block is simply locked for a set period of time and can be updated only from a particular supplier node. If that block was not completely downloaded during the set time interval, it is timed out and rescheduled for repeat attempts.

These mechanisms are well known from regular torrent programs and we are not going to discuss them in this document. There is plenty of information on the net for technically savvy.

BTSync behavior table

Note: this table was initially created as a stub, a concept in an attempt to fully describe the behavior of btsync under all different conditions and modifications of the files by the users on both ends, the master (r/w) and slave (r/o) nodes.

The initial version was created without much understanding by just guessing things and referring to previous observations of some cases. The version you are reading was updated after verifying the behavior in most cases described here.

According to our understanding at the moment, there are a couple of principles that may help to understand the consequences of various user actions and changes to the files in the hierarchy of some share (parent folder/directory).

First principle: Basically, what needs to be kept in mind is that the central idea in the btsync design, as far as we can see, is to keep the files on all nodes in the same consistent state, that is the same as on the master (r/w) nodes. And, if we keep in mind the idea that the r/w nodes "dictate" the conditions and status of the files and r/o nodes have to exactly replicate the contents of the r/w nodes' files, if those files were not modified, deleted or renamed on the r/o nodes, then many things may become intuitively understandable.

The second principle is the rule: "user knows better", which means that if there are any conflicts between the the btsync update logic and the operations on the files performed by the user manually, such as file deletions, modifications, renaming and so on, then the user action takes precedence over the update logic in btsync.

Furthermore, there is a logical difference between the file modifications performed by the user on the r/w and r/o nodes. The behavior of the r/w nodes seems to be entirely consistent with logic and is equivalent to the ways it happens on the O/S level, that is, when you delete some file, it is simply gone with no memory of it left. But on the r/o nodes, the btsync behavior is not so consistent and in some cases may be considered as utterly inconsistent and utterly unintuitive. You will see it in the table below.

As of now, what needs to be kept in mind is that, generally, any modifications, deletions and renaming on the r/o nodes mark the modified (original) files as non-updatable and non-propagatable. Effectively, those files are permanently latched up in a state that essentially indicates that they do not exist. And, probably the worst part of it is that in some cases you can not recover from that state and restore the normal propagation, even if you enable the "Restore modified files to original version" via Folder preferences -> Advanced tab.

Furthermore, even if some file is deleted, it still remains the the r/o node's database and is simply marked as "dead" to btsync (non-updatable and non-propagatable) and any modification of it, even renaming it, won't change the status of the file with the same file name in the future. In other words, renaming the file does not carry the file access attributes to the new file. The attributes remain with the old file name, and they essentially mean "dead file", which seems to be logically incorrect design.

Whatever were the reasons there for this design, it causes quite severe consequences for the future of the file with the same file name and there is no way to change it back to its original propagation status no matter what you do, except of resetting the attributes to the original state for the entire collection (database) on the r/o node via "Restore modified files to original version" via Folder preferences -> Advanced tab on the Linux r/o nodes at least.

The btsync team could shed some light on these issues indeed. Otherwise, we are going to be guessing here as to what really happens "under the hood" and for what reasons. But at least we can try, and, hopefully, this table will become the beginning of detailed clarification of all possible cases, and, even more hopefully, with help of developers or support team, at least with clarifications and reasoning for the present behavior, which has to be done, sooner or later, no matter how you look at it. Otherwise...

BTSync Behavior Table

What is missing in current versions of documentation is the description of the procedure to properly reset the system if something happens that BTSync considers as something different than you might think. For example, for whatever reason, some files stop syncing and won't propagate to the slave machines. But this is not what you expected to happen and is not something you'd like to happen. In this case, how do you regain control of the system and syncing process?

There is also a need to know the name of the database file for some share. Currently, there seems to be no easy way to find out the name of database file for some sync folder. It would be desirable to be able to see the name of the database hash right from the GUI, regardless of whether debug log was on or off.

Current file update and propagation logic problems

It seems that btsync logic for r/o nodes needs improvement, unless we are missing something. Because it does not seem to detect various user operations on the file properly and simplifies all possible cases, such as file modification, renaming, deletion and restoration, by bulking them all together into a notion of "file is not updatable and not propagatable". But all these cases have different semantic meaning.

File modification means that this file does exist in the database and its "physically present in the file system" bit is set. In this case, two separate bits in the database - "updatable" and "propagatable" should be reset. But "physically present in the file system" bit should not be changed.

File deletion means that its "physically present in the file system" bit was set, but no file actually found in the file system. In this case, its "physically present in the file system", "updatable" and "propagatable" bits should be all reset.

If file is restored by the user, its "physically present in the file system" is tested, and if it was reset, then this file has been "resurrected", in which case its "updatable" bit is set, which means that this file will be immediately requested from other nodes and its contents, whatever it was when it was restored, will be overwritten with the version in other nodes with the assumption that it will be exactly the same as on the master nodes. Then, AFTER this file has been received and updated, its "propagatable" bit is set and this file may then be transfered to other nodes. Additionally, "physically present in the file system" bit is set.

When file is renamed, it is equivalent to 2 separate operations: 1) file is deleted, 2) new file was created by the user, which did not exist before. In case 1, the old file is marked as "physically present in the file system", "updatable" and "propagatable" - all false. For the new file (case 2), its "physically present in the file system" bit is set, "updatable" and "propagatable" - both - are reset.

These 3 bits will be able to handle all the cases that we can see at the moment without any problems and any user actions on this file. Furthermore, the net effect will be logical and intuitive to the user, who may restore some deleted file from a backup and would expect it to become a "normal file", subject to all rules, including its propagation and reconciliation of its contents with the version, currently existing in the master nodes, "master rules" rule applies. Because it becomes the same file as it exists on the master side, and the only one who dictates its contents is the master node and the copies of it on the r/o node.

ALL that user has to remember is that if some file is modified or deleted, it will be considered a "private" user file and will not propagate, and if file is renamed, the new file will not be updated nor propagated because it does not reconcile with the master version of it, which does not exist. It is the same behavior as if he added a new file on the r/o node.

Otherwise, everything should behave like a normal propagatable and updatable file. The logic becomes simple and intuitively obvious and everything should work just fine, unless we are missing something here.

Peer discovery principles

BTSync uses several methods of peer discovery.

• Known hosts: this is the simplest; you enter a Port number and an IPaddress or DNS host name and BTSync attempts to contact this host (This should also work with dynamic DNS if you wish)
• Search lan: this sends out multicast packets to the local lan (and rarely some connected ones) on port 3838. If a client receives one of these it attempts a normal connection to that peer.
• Tracker: BTSync sends the info hash of the share (basically the hash of the secret) to t.usyncapp.com. That host keeps a list of the IP:port pairs that have contacted it with that hash and gives them out to everyone interested.
• DHT: (Distributed Hash Table) this is very similar in concept to a tracker, except the hashs are not stored on one central server but distributed across all the peers in all the swarms.
  There does need to be a starter peer (one of which I expect is hosted by bittorrent.com), but once started the network is self supporting. (This one is real magic :)

use_lan_tcp is NOT use for peer discovery; it is a way of speeding up connections to peers that have already been contacted with UDP (like turning off lan_use_encryption).

The relay server is NOT used for peer discovery; it's for working around stupidly obstructive NAT and firewall devices.

What does relay server do?

Relay servers help get around firewalled states on one or more nodes.

DHT helps when the tracker servers are down.

Note: If you are connected to some node via relay server, then you will see the node's name exactly as it was named by the node itself, but its IP address and port will be the address of the relay server and not the node itself. In this sense, the relay server behaves like a proxy server and it hides the node itself from others.

What does tracker server do?

The BTSync tracker is the same as a Bittorrent tracker, it does almost no work.

For each combination of peer and share it stores:

• The hash for the share
• The IP address of the peer
• The port number of the peer
• The time to delete this record.

Once two peers have been introduced to each other the tracker has nothing more to do. If the tracker goes down after the peers have linked up there's no problem. That DNS name resolves to three independent servers which will (probably) be sharing information.

But, I expect, (deal killer if not!) there will be a configuration option to allow other trackers later on.

As I said the real magic is the DHT mode where if you know or remember the location of any node in the network you can use it to find the rest of the network.

I've tested the known hosts connection with Windows-7, it's working fine for me. I've tested it with only one side knowing the location of the other and that works too (depending on the firewall).

Remember you have to open UDP ports not TCP ports and usually tell the BOTH clients the public IP addresses of the other so they can work together to poke holes through NAT or connection tracking.

If the public IPs are dynamic you can use services like dyndns.org to get a constant dns name.

If it's still not working you might need to raise a bug report.

About minimum firewall requirements...

Okay, about minimum firewall requirements...

The relay and the tracker are found using DNS, these are the current settings for relay and tracker.
Note the TTL less than 5 minutes in both cases, this is the warning they need to give to change these.

;; ANSWER SECTION:
r.usyncapp.com. 203 IN A 67.215.231.242
r.usyncapp.com. 203 IN A 67.215.229.106

;; ANSWER SECTION:
t.usyncapp.com. 258 IN A 54.225.100.8
t.usyncapp.com. 258 IN A 54.225.196.38
t.usyncapp.com. 258 IN A 54.225.92.50

Normal operation is using the tracker to find peers and using direct connections between the peers to transfer data. All data is transferred using UDP packets.

Your BTSync has a port configured, say 20001.
The Peer has a port configured, say 20002.
The tracker has port 3000 configured.
The relay has port 3000 configured.

Requirements are:

• Unsolicited packets must be able to travel from your port 20001 to the tracker on port 3000.
• Solicited replies from the tracker on port 3000 to your port 20001 are required.
• Unsolicited packets must be able to traverse your firewall from your port 20001 to Peer's port 20002
• Solicited replies from Peer on port 20002 to your port 20001 are required.
• The public port that the firewall presents must be the same as the BTSync configured port. If your firewall renumbers ports unpredictably only the relay server can be used.

This assumes your firewall uses the normal 'timeout' method of noticing solicited responses. The problem is that the firewall will not see the request for the first response as it travels via the tracker. It must not do anything "unfortunate" when it sees this "response".

If your firewall is broken in this way then unsolicited packets must be accepted both ways between UDP ports 20001 and 20002.

If your firewall cannot be fixed connections to the relay must be opened for all peers that need to communicate with you.

If you wish to use DHT you must accept unsolicited packets on your port 20001 from any address.

If you configure known peers you can turn off access to the tracker; no packets then need to go to the tracker

Excluding some files and/or folders from syncing

If you want to exclude some files from syncing, so that they are not propagated to other nodes, one of the ways is to use the .SyncIgnore file and enter the names of all files and/or folder that you want to exclude.

The other way, that works on r/o nodes only, is to either modify or delete the files you do not want to either download or propagate. All the files that either modified, or deleted are marked in the database as "non-propagatable" and, effectively, become dead to btsync. This is basically not a recommended way and the sync folder will show the state of the share and the amount remaining to download/upload incorrectly, and it will look like there is still data to be transferred while no transfer will ever occur.

Using .SyncIgnore file to exclude some files from sync

The proper and recommended way to exclude some files or subfolders from syncing is to use the .SyncIgnore file, located in the top folder of the share (hash, key, sync folder). In this case, BTSync will exclude these file from downloading and propagation to other nodes and will properly report the statistics and the amount remaining to download/upload in the Devices tab.

.SyncIgnore is a regular text file (UTF-8 encoding) containing one file or folder per line.

You can use "*" and "?" wildcards in .SyncIgnore files.

• "?" is a substitute for a single character instance and matches any character,
• "*" is a substitute for multiple character strings.

Examples:

• *.dat would exclude from syncing all .dat files types regardless of name i.e. myfile.dat, anotherfile.dat, etc.
• abc???.dat would exclude from syncing files named "abc123.dat", "abcdef.dat", etc... but wouldn't exclude files named "abc1.dat" or "abcdefg.dat", etc.
• *.mp? would exclude from syncing file types such as .mp3, .mp4, .mpa, etc, but wouldn't exclude file types such as .mpeg
• Video - exclude all the files in the folder Video on any level (depth) of the path.
• Video/*.avi - exclude all the .avi files in the folder Video on any level (depth) of the path.
• /Video - exclude the entire Video subfolder, but only if it is the TOP level subfolder of the main folder.

The effects of .SyncIgnore are more subtle that it might look at the first glance and file specifications are not intuitively obvious and behave slightly differently than they do in the O/S.

In the following rules we will use the directory level notion, which is the same thing as directory depth in the file path. The 1st level is the root folder of the share. The 2nd level is a subfolder of the main folder, and so on.

Rules

• File names are specified relative to the root directory (main sync folder), and not the absolute path in the file system.
  So, the absolute path E:\folder\share_folder\*.mp3 is incorrect specification and will not match anything.
• To specify the subfolders, file names are specified according to the same exact rules as on your O/S in respect to directory separator characters, ie to specify the subfolders, use the backslash character ("\") on Windows, and forward slach character on Linux ("/").
  For example: /Audio/*.mp3 or \Audio\*.mp3 will exclude all the .mp3 files in subdirectory Audio of the main folder, but not in any other directory or subdirectory of any other level.
• If you specify some file name with or without wildcards, and your file name does not start with O/S dependent subfolder separator character ("/" or "\"), then btsync will check all the subfolders of all depths, from the root folder, down to the deepest level, and if it finds a file that matches the name(s) you specified, including the use of the wildcards, than those files will be considered a match and will be excluded.
  For example *.mp? will match any file of .mp3, .mp4, .mpa types in all the subfolders regardless of their depth in the path.
• If you want to exclude only files in the top level folder, then begin the file name with a directory separator character ("\" or "/").
  For example: /*.mp3 will exclude the files of .mp3 type ONLY in the top level directory.
• If you want to exclude only files on a certain subfolder level, then you begin the file name with directory separator character, specify the match specification for that level, possibly using the wildcard characters, then separator again, followed by the file name spec.
  For example, you have the following top level subfolders, Audio, Video, Books and in the Books, you may have another subfolder, PDF, and you want to delete only .txt files in any of top level subfolders. You would specify it as follows:

  (Note: this is a Linux version of directory separator characters: - /
  On Windows, use the backslash instead: \ )

  • /*/*.txt - exclude all the .txt files in the top level subfolders only (immediate subfolders of the root folder).
  • /*/*/*.pdf - exclude all the .pdf files on the 2nd folder level deep.
  • /*/PDF/*.pdf - exclude all the .pdf files in the 2nd level folder named PDF. The leading /*/ will match all the the top level subfolders (Audio, Video, Books). Then, on the 2nd level subfolder, it will match the PDF subfolder, and in that subfolder it will match any .pdf file.
  • The files or directory names may contain the blanks (" ") in their name. On Windows, just enter the file/directory name exactly as it shows on Windows explorer (simply copy/paste the file name).
    On Linux, the convention is, to precede the blanks with a backslash "\ ".

What is the meaning and function of .SyncIgnore file?

Question about .SyncIgnore, asked on btsync forum:

Does it mean "do not update this file from the master (r/w) node? Or does it mean "do not propagate this file to other r/o nodes? Or both?

Answer:

This implies that if other peers do not have the same files/folders in their .SyncIgnore file as this particular node, then they will still get them from the other nodes (r/w or r/o) that do not exclude these files in their .SyncIgnore.

It also means that .SyncIgnore has an effect on per node basis. If some node wants to exclude some files and not to download nor propagate them from/to the other nodes, then it adds those files to its .SyncIgnore.

If that node was already synced and already has those files, then after editing the .SyncIgnore, the unwanted files will not be automatically removed from the disk and may be simply deleted from that node manually.

But if you want to ignore the same files on ALL the nodes, then every single one of them should have exactly the same files in their .SyncIgnore file.

"Fighting" r/w nodes and magical appearance of duplicate files/folders

Finally, there is an interesting case of r/w nodes "fighting" with each other (in case you have multiple r/w nodes on some share). If you delete some files or folders on one r/w node, it won't help anything if you have multiple r/w nodes.

Even deleting it locally, will not prevent it from being re-downloaded again in few seconds from other master nodes if they are on line, and even worse, if you are the only r/w node on line and delete some file or subfolder and verify that it all worked fine, you may not realize that it was nothing more than illusion.

The same thing happens if you rename some file or subfolder, with even more amusing effects.

Because it still exists on other master (r/w) nodes. As soon as they come on line, you will get your deleted or old versions of renamed files or folders downloaded from them again.

Remember, all the fully synced r/w nodes contain the aggregate collection that is created as a result of UNION operation. Those files or folders that are absent on some nodes will be sent to them by the nodes that have them.

And even more interesting behavior is observed if you rename some subfolder. What happens if it was already uploaded to other r/w nodes is that now you have two different folders with the same contents, the one with old folder name on other r/w nodes, and the other, with new name on your node, and, since your new folder contains the same exact files as the old one, then it will immediately start uploading that folder to other r/w nodes as a new folder, and the other r/w nodes will send you a copy of the folder with OLD name. Once the sync process for these folders is complete, ALL the r/w nodes will have two copies of the same folder with the exact same contents, but with two different folder names.

And, even more interesting, once you notice that you have 2 copies of the same folder contents but with different name on your share, you may try to delete the folder with the name you do not like, not even suspecting that it came from other r/w nodes and wondering about where did it come from?. But it will be all in vain if other nodes have that folder. They will simply resend it to you again.

So, you MUST use .SyncIgnore if you decide to delete or rename some files/folders in case you have multiple r/w nodes, and some of them have already synced with those folders/files that you have deleted or renamed. Furthermore, all other nodes must also have them in their own .SyncIgnore file and they may not even suspect that they have some duplicate files/folders as a result of actions on other r/w nodes.

So, before deleting or renaming some files or folders, first, enter their names in the .SyncIgnore file and save it, and only then you can delete or rename it.

Using .SyncIgnore Master file

In order to prevent some unusual or strange effects arising as a result of deleting or renaming some files or folders (see: "Fighting" r/w nodes and magical appearance of duplicate files/folders ), what you can do is to create the .SyncIgnore.Master file on one of r/w nodes.

Also, in order for all the nodes to be in a consistent state and contain the same exact information as all other nodes, in case some files or subfolders are excluded from syncing for all sorts of reason, such as containing the private information, or scratch pad files, not meant to be propagated to other nodes, ALL the nodes need to have the same .SyncIgnore file contents. Otherwise, some of them will be updated with unwanted files and will propagate them to other nodes.

Note: .SyncIgnore file does not propagate between the nodes automatically. Because it is an O/S dependent file and Window's version of it won't work with Linux and vice versa.

But the .SyncIgnore.Master WILL be automatically updated and propagated, just like any other file, if ANY r/w node modified it or extended it.

The beauty of this approach is that nobody can dictate to your node which files you do or do not want to update and propagate, as it would be the case if .SyncIgnore was automatically propagated. Because that would be imposing someone else's opinion on what you do or do not want to carry on your node, which does not make sense and is not a "democratic" approach.

The only thing to be kept in mind is that it is an O/S dependent files and so on Linux, all the backslashes ("\") in the file names have to be replaced by the forward slashes ("/") and all the blanks in file or directory names have to be replaces with "\ " to follow the Linux convention.

What it means is that all the nodes can be automatically informed about the master nodes' "latest and greatest" .SyncIgnore contents via .SyncIgnore.Master file. So, they can simply copy/paste its contents into .SyncIgnore file and do some editing for the Linux version of it in order to comply with the O/S file naming convention.

This should work great and prevent all sorts of files or subfolders being distributed unnecessarily. Furthermore, all the discussions and arguments about whether to automatically propagate the .SyncIgnore simply resolve without a single effort of the part of BTSync development efforts and everything "works as is".

And even more important for general public distribution, where one node has no idea about the other nodes and where the information may be updated dynamically, at any point, there might be some changes desired to the .SyncIgnore file. But how do people know about the "latest and greatest" unless the .SyncIgnore is automatically propagated from the master nodes? And if it IS automatically propagated, then the latest master version of it may "step" on your local changes that correspond to your personal view on what to carry and what not on your node.

What will be shown in the Devices tab if some files are excluded via .SyncIgnore?

Q: What will be shown in the Devices tab if some files are excluded via .SyncIgnore? Will it still show that the node is fully synced? Or will it show some size still remaining to be uploaded?

This implies that if you see some peer node in your GUI that shows that there are still files remaining to upload to it, it could be that they are in his SyncIgnore.

What does it mean if some node used to show that it is fully synced and then it just keeps showing that there is still some size to be uploaded, and does not show that it is fully synced ever since?

Excluding some files from syncing by modifications

You can also exclude some files from syncing on the r/o nodes after they have been downloaded, by either simply deleting them, in which case you will no longer receive any updates on them, or by renaming them, or by modifying their contents, which is only applicable to non-binary files in most cases.

But this trick is not a clean solution. First of all, it won't prevent those files from the initial download. Then, if, at some point, some other files may stop syncing for you for a variety of reasons, and if you have excluded some files by modification tricks, other than renaming the file, and you decide to restore their original update/propagation mode via Folder Preferences -> Advanced tab -> "Restore modified files to original version" checkbox, then all those files that you have excluded via modification trick will be restored to their original state, which is exact copy of the file on the master node.

But this should work fine and clean if you rename the existing files in order to fix some version of the file so that it won't be updated from other nodes and not propagated to other nodes as well.

But file deletions and modifications could not be considered a clean solution as a general case. So, use the .SyncIgnore file to make it clean and compatible with BTSync documentation and recommendations by the team members.

How to run BTSync as a service

Encrypted folders

You can create the encrypted folders with a little bit of funky manipulations. BTSync does have this ability even though it is not presented in a clean way via UI. Furthermore, creating the encrypted folders on nodes that do not have the hardware support for encryption will significantly slow down the process and will load the CPU significantly. Nevertheless, here is the instructions to create the encrypted folders as presented on BTSync forums:

1) AR7GC6JIVCTKG2XNPM7GGOSV3FI5BDDNH
2) DR7GC6JIVCTKG2XNPM7GGOSV3FI5BDDNH
3) EYF7Y3OOYZEZALVLFDQDAARXQTV2HO7IZCO2V2ZRZEAN32VY7RFH7HGOKRI
4) FYF7Y3OOYZEZALVLFDQDAARXQTV2HO7IZ

DABCDEFGHIJKLMNOPQRSTUVWXYZ234567
EYRCW2XHXF3NRDT4Z44CL45Y5ZH2HO6ADW33KXUGY4EZN4B5RQDELP7IQQE
FYRCW2XHXF3NRDT4Z44CL45Y5ZH2HO6AD

2) DR7GC6JIVCTKG2XNPM7GGOSV3FI5BDDNH
3) EYF7Y3OOYZEZALVLFDQDAARXQTV2HO7IZCO2V2ZRZEAN32VY7RFH7HGOKRI
4) FYF7Y3OOYZEZALVLFDQDAARXQTV2HO7IZ

Are claims of security and encryption real?

Well, this is one of those issues BTSync is not likely to make a public statement about, and even if they do, how much do you think you can trust all those claims about traffic encryption and security made by BTSync? Consider this:

Interestingly enough, one individual created a thread on BTSync forum where he brought up an issue of cooperation of BT with NSA. He says the thread was pretty popular, but not a single comment was made by BT about the issues he brought up, and those issues were related to security of BTSync and claims made as far as traffic encryption goes. Finally, that thread was simply deleted without comment. Among politicians, such things are equivalent to "no comment".

Here is a little article on BTSync forums about FSF attempts to create an Open Source BTSync clone and ramifications of the fact that BTSync is not an open source in the context of the NSA and other agencies' global and invasive spying.

Which hosts are silently contacted by BTSync?

BTSync silently contacts a number of hosts for various reasons without informing the user. Some of those hosts, such as trackers and relays, are automatically contacted and there is a specific URL which accesses the lists of automatically contacted hosts (see below).

BT claims that they do not collect any information beyond traffic statistics. But, it knows all your keys to all your shares, and so, from then on, to get any kind of information about any nodes and shares represents no difficulties whatsoever. And if you look at it from the standpoint of allegation of cooperation with NSA, then spying is simply implied with all its consequences.

Here is some information on it from BTSync forums:

{
"trackers" :
[
   {"addr" : "54.225.100.8:3000"},
   {"addr" : "54.225.196.38:3000"},
   {"addr" : "54.225.92.50:3000"}
],
"relays" :
[
   {"addr" : "67.215.231.242:3000"},
   {"addr" : "67.215.229.106:3000"}
],
"mobile_push_proxies" :
[
   {"addr" : "54.235.182.157:3000"}
]
}

    $ strings /home/btsync/bin/btsync \
      | grep -E '[^ ]{3,}\.com' \
      | sed -e 's#.*://##g' \
            -e 's#\.com.*#.com#g' \
      | sort -u

    apps.bittorrent.com
    bench.utorrent.com
    config.usyncapp.com
    download.utorrent.com
    gui.com
    link.getsync.com
    raptor.utorrent.com
    remote.utorrent.com
    router.bittorrent.com
    router.utorrent.com
    search.conduit.com
    syncdev.bittorrent.com
    sync-help.bittorrent.com
    tracker.openbittorrent.com
    tracker.publicbt.com
    update.utorrent.com
    www.bittorrent.com
    www.usyncapp.com

What do the device icons mean?

The icons that appear to the left of the device name in Devices tab mean:

Direct Connection - BitTorrent Sync is able to establish a direct connection to this device

In-direct (Relayed) Connection - BitTorrent Sync isn't able to establish a direct connection to this device, and therefore has connected via a intermediate relay server. It is preferable to have direct connections wherever possible, as this will generally provide faster transfer speeds over that of a relayed connection.

If you change, delete or modify some files

If you change, delete or modify some files in any subfolders, the effects will depend on whether you did it on the master (r/w) node or the slave (r/o) node. For more detailed description see:

BTSync behavior table

Generally, if you did it on the read-only (r/o) node, then you will no longer be updated on any modifications or updates of those files. They will be considered as non-syncable or "ignored" from then on.

Also, your changes will not be propagated to other nodes and will be considered non-propagatable to other nodes. These files will become your private version of whatever information they contain. Furthermore, at least in some cases, their original propagation status can not be restored from then on, no matter what you do. The only way to do it in some cases would be to remove that share/folder from BTSync (but not physically from the file system) and recreate it again in the same exact folder with the same exact key.

But if you did it on on the read-write (r/w) node, then the behavior of btsync will be quite predictable, just like if you did it in the O/S file system.

File update history: Finding out which files were updated or modified on the master side

BTSync has the ability to keep history of updates. Whenever some file is changed and a new version of it was downloaded, the latest copy of it, just before the change, will be saved in the .SyncArchive folder at the same place in the directory hierarchy as the original file was. Its file name will have the numeric file version number appended to the file name.

There is also an option "Store deleted files in the SynchArchive". It can be accessed by clicking on the Folders tab, right clicking on the folder, selecting "Show folder preferences", then Properties tab and then clicking on the "Store deleted files in the SynchArchive" checkbox. If this checkbox is not checked, no previous revisions will be kept.

All the files in the .SyncArchive directory tree that are older than the default value of 30 days will be automatically deleted. Alternatively, you can delete them by hand without any harm to operation. You can even delete the entire contents of the .SyncArchive folder. But do not deleted the .SyncArchive folder itself.

There is a parameter that can be set - sync_trash_ttl from Preferences tab. Click on Advanced button at the bottom and then you can change the number of days to keep your previous versions. After that many days, the oldest files are deleted automatically.

How BTSync knows about new files?

Normally, on Windows and on some recent Linux versions btsync knows about new files and updated or deleted files within a few of seconds. Then, after a few more seconds, typically within a minute or so, it performs the transfer to all nodes that have that file updatable.

What are .SyncID, .SyncIgnore, .SyncPart, .SyncTemp .SyncOld and .!Sync files and the .SyncArchive folder?

When you add a new folder to BitTorrent Sync a number of hidden files/folder are automatically created in the folder. The purpose of each are as follows:

.SyncID = A file containing the unique, internal "ID" of the folder. This file should not be manually modified, or deleted. If you do, the folder will no longer be recognized in BitTorrent Sync.

.SyncIgnore = A user editable file allowing you to "exclude" certain files/sub folders for being sync'd

.SyncArchive = BitTorrent Sync, by default, won't actually delete any of your files/folders. If a corresponding file/folder is deleted on another device, it will simply be "moved" into the .SyncArchive folder on all other devices, rather than being permanently deleted.

(In Sync pre v1.1.40, this folder was named .SyncTrash and was later renamed to .SyncArchive as this folder now stores both local files deleted on remote

When a file changes, does BitTorrent Sync transfer the entire file again, or just the part that's changed?

Files smaller than 4MB are transferred again in their entirety when changes are detected.

Larger files, however, are split into 4 MB "chunks" of data, if one chunk changes, but the others don't, only the chunk that's changed will be transferred (instead of the whole file again).

How long are files kept in .SyncTrash/.SyncArchive for?

From v1.1.30 onwards, deleted files are kept in .SyncArchive (or .SyncTrash pre v1.1.40) for a default period of 30 days, after which they will be automatically removed.

You can change this default period by adjusting the advanced "sync_trash_ttl" setting.

What data is collected/seen by BitTorrent Inc when I use Sync?

According to BitTorrent Inc, "There are 3 points of contact with BitTorrent infrastructure: tracker, relay, and check for update service. If you don't use any of these - we will not get any information about you."

Edit History

You can keep adding your entries to the list below. Identify your node, so others can communicate to you via their Device name message. Currently, there is other way to communicate in BTSync, even though we have proposed it to them on the forums nearly a month ago. But they don't seem to care about this issue.

You might want to add the GMT/UTC time stamp to your entry.

• 2014.03.30 20:42, Master node:
  Edit History section is added.
• 2014.03.30 21:19, Master node:
  Entry 2:

In this section we will try to clarify the issues of "attitude" of BT, the state of the product and issues of "priority items" on the list of priorities, at least the way it all manifests itself as you can see yourself on the BTSync forums.

Is this a real product, or just a prototype

This is probably one of the fist things anyone would like to know when dealing with some new product.

"The bottom line", as far as BTSync as a product and its state of completeness go, it is FULL of problems. Literally "full of it", as they say. Luckily, most of those problems are not so bad as to prevent one from using the program, but MANY of them create the situations for the users that it simply does not work at all, and because of the most irrational reasons.

Furthermore, displaying the important information to the user, so he could understand what is going on and track some issue down is simply horrible. And it is as primitive as it gets. Even to classify BTSync as a professional product from this standpoint is simply laughable, if not pathetic.

And the program is full of bugs. But, at least if things are done with understanding, it does work eventually, if you know how to avoid those mines while walking on a mine field, which it is.

So, we have tried to provide the most necessary advice and our opinion on BTSync forums, but all of it was simply ignored, and it eventually even went as far as "last warning" threat, which is simply nothing short of insane.

We have done such a tremendous amount of work on their pathetic documentation that even a representative of BT corporation contacted us with thanks and asked for permission to use it in their own docs, which we initially gave them. But when all this insanity and utter denial of the issues and problems that are as real as your own five fingers, which ended up int the most disgusting insults and threats, revealed the true nature of what we are dealing with. At that point our support was totally withdrawn and they were told specifically that they have no permission to use our documentation in any way, shape of form.

Why should we spend months of time of the developers of the most senior level there is to provide any "free" help to those, who keep their sources closed, exhibit "piss on you" attitude and go as far as threats?

So, all the information and assistance to BTSync in our most detailed posts with analysis and proposals and requests for the most basic changes, that would take in most cases minutes to implement, were edited out and the most valuable content was removed.

Let BT know this: you guys have to do your work yourselves. You have already received the assistance worth a mid 4 figure dollar amount, if this were done under the contract. So, enough for you.

Your attitude "sucks" and your product "sucks" just as well. But it does work, even though just barely. Unfortunately, at this particular junction there isn't yet a product that is ready and cross-platform compatible, even though Hive2Hive http://Hive2Hive.org/ library, written in Java, is already available in its initial final release version, which means that we are going to see the first open source projects appearing in no more than 3 months. Actually, with all that library has in terms of features and interfaces, it should take about 1 month to create a product compatible to BTSync in terms of features and power. And that product is an open source library and it is much better designed than BTSync can even dream of, at least from the preliminary estimate and initial look at it.

So, as far as BTSync goes, it is our opinion that this product has no future, regardless of how many zombies are going to use it to download some psychotic "music" to their mobile devices. Because it is a closed source product, and on the top of it, it barely works, and, instead of fixing bugs, they keep adding new bells and whistles to the product they have announced to be in beta phase, which means nearly stable product, ready to be released.

The rules for the beta products are simple enough: You do not introduce ANY new "features" or bells and whistles. Beta means exclusively bug fixing and problem resolution phase. But the way BTSync looks to us, it isn't even in alpha stage. Basically, it is in experimental or prototype phase, and we are seeing somewhere between 30 to 50% of people having various problems and can not even initially sync, which is simply bizarre.

Under no conditions, except of some network related issues, should the user be prevented from the initial sync, that is from downloading the whole share as it exists on the master nodes.

As far as computer time difference errors, we have proposed a solution that would fix this problem in minutes. But they simply exhibited a "piss on you" attitude, like this problem is purely a user error issue and said something like: "we'll see what we can do to HELP users to deal with this problem".

WHAT? HOW can you possibly "help" general public that just installed your program and it does not work? First of all, it is not users that need "help". It is YOU that need all the help you can get. Because there is sufficient evidence that it is not a user problem. We have seen cases where we observed this error with several nodes simultaneously and when we adjusted our clock time to match two of those nodes that showed this error and that error was the same for them, still, one of them would show the error while the other one no longer showed it. Yes, this issue is not as simple as it might look without detailed analysis, but it does not look like a simple user error where user is simply so dumb that he can not see his computer time clock. It simply makes no sense.

Secondly, how can you "help" some user if you don't even know who that person is? You can not send him a message with advice and provide him a link to a detailed document which explains the nature of his problem. HOW can you possibly "help" him? By doing what kind of miracle?

So, we will present a few selected posts here and then will do some editing and leave only the most interesting things in it.

A member of BTSync support team said: "we do care about our customers". - No, you don't. You don't even care about the highest level developers giving you a free advice on how to fix your most ridiculous problems literally within minutes.

Compared to documentation we have created for YOUR product, your documentation, if it can be called such, simply sucks worse than a black hole and it does not answer a single question on the most critical issues that prevent users from even doing the initial sync, not to mention the most ridiculous problem with incomplete syncing and so on.

Shame on you, BTSync and double shame on you, BT. You even have guts to claim the increased security as a result of encryption of traffic. But, as one security professional said: "you can not possibly make any claims regarding security and encryption for a closed source product". Yes, PRECISELY correct. And a hole in security may turn out to be in the places no one even suspects, especially in light of some claims that BT is cooperating with evil agencies of this satanic NWO scheme of taking over the world, such as NSA, for starters. One person even said that he has observed certain things happen that clearly indicated to him that they are in fact cooperating with the NSA.

Yes, to verify it from the outside is not an easy job to do. They are not going to announce some things in public statements, but the very fact that some thread dealing specifically with the NSA cooperation was removed and not a word of comment has been provided by BT staff regarding the extent of their cooperation with NSA, if any, simply indicates the same thing as "no comment".

So, this is going to be the context for what you will see in the following chapters. Except we do not have time at the moment to organize it all and make it nice and clean and simple and clear to understand. So, if you are interested in reading it, just keep patience. We'll get there eventually. We are not going to let these issues "slide".

But some things might be too technical for some users and you might find too much detail in descriptions of the problems and the solutions proposed. So, it might not be such an easy read for some, unless they are curious to know more of what they are dealing with.

We are going to discuss the following issues:

• User interface design issues
• Time difference error issue
• Incomplete sync issue

At the moment, we will only drop a few posts from the BTSync forums here...

Time difference error issue

Here's just a few posts from BTSync forums. This needs an initial edit. Right now, it is just a "dump" of sorts.

Device time difference more than 600 seconds

Nice, eh? Well, the original post he had edited contained the following (in response about "we are going to see what we can do to help the users" in dealing with timediff problem):

Mind you, we are talking about one of the nastiest problems of BTSync as a result of which new users that just joined some share would not even start to download it because they would be presented with the error message "Sync stopped: Device time difference more than 600 seconds". The reason for that error is that BTSync "thinks" that the user's clock is set incorrectly and, therefore, it was impossible for BTSync to determine if his version of some file is newer or older than the version of the same file on other nodes.

It is quite a nasty problem as far as technicalities of it go, and it would require quite an explanation of what is taking place and why, which we will omit here to avoid bringing up purely technical issues. But "the bottom line" is this: this particular error is as real as your own five fingers and at different times we have seen up to 30% of users on some shares that would simply sit there and their nodes would not even perform the initial download of the share, which is a disaster of an issue. Because to them, this whole thing would simply look like: "well, this thing does not really work", and, as a result, after spending some time sitting there and waiting for the sync process to start, they'd simply leave the share eventually.

And about the most ridiculous aspect of this error is simple: Under no circumstances whatsoever the node should be prevented from the initial full download of the data. Period. Simply because there exist no technical reasons to prevent even the initial download, regardless of which version of which file is seen by BTSync as "newer" or "older" than on some other nodes.

The initial sync simply MUST be able to download even those files that could theoretically turn out to be "older" versions as seen by some nodes. Because initially, it is utterly irrelevant which version is newer or older, at least compared to utter inability to download the file itself. After initial sync, BTSync would be able to update some file to the newer version and the process is fully automatic. There isn't even a thing to be done for it to happen, or even talk about. It already works "as is". Because this is the very nature of the sync process.

So, what could possibly be the technical reason to prevent even the initial download? Therefore, this entire issue is simply ridiculous, if not outright bizarre, from the standpoint of program logic and function. It is a purely artificial issue which can only exist in a totally broken and utterly inconsistent design.

But when we did some investigation on this issue and have discovered some pattern of behavior indicating that it is not merely a user problem but rather a flawed design, we were simply laughed at. So, eventually, seeing all the hopelessness of the situation and utter denials, we have posted the above message which pointed out that they simply deny the problem as such and were trying "to talk it away".

So, what was so deserving to be edited, which is considered to be an intellectual crime among the writers? Because by editing someone else's post, you may distort and pervert its very meaning and the essence of the issues, which is EXACTLY what happened in this case, and in the most perverted way, done by a man of profound dishonesty, and making it look like not an insult of intelligence of an author by the most ridiculous responses "we'll see what we can do to HELP the user", knowing full well that no such help was possible.

And that is precisely the reason it was said "you do have a sense of humor, I have to admit".

And a quote by U.G. Krishnamurti was "right on the money" in the context of utter denial of the issue as such, without ANY investigation.

What "spamming the forum with irrelevant/off-topic posts" this bozo, full of self-pride and having access to the big red button, labeled "kill", is talking about? You see, he started imagining that he has "power" with his "kill" button, and he can edit the posts of other people, and he can delete the posts, which is what he has done in an attempt that exposed his dishonesty when he lied and tried to make this timediff error issue look like something no one cares about, which isn't the case. Because, in reality, quite a few people read that thread about timediff error.

And so he simply deleted the post that contained a single line of text essentially, saying that only during the last 24 hours there was 90+ views of that thread. What a coward!

So how it could be so insignificant if nearly 100 people read it every day? What does it tell you, unless you are a complete zombie having a black hole in place of a head? How many thousands of people do you need to see reading that thread before you consider it as something significant?

Because the original post was "right on the money", and it was addressing the way this most ugly issue of timediff error was affecting about 30% of all the people on one of the most important shares, not counting those, who came, wasted a few minutes, and not seeing it "working", simply left.

Interestingly enough, this issue is known to BTSync for at least 8 months. But it has NEVER been really addressed or investigated, at least until we pointed out that this must be a time zone issue related to DLS (daylight saving) time adjustment, and, furthermore, there were indications that it would be present even in the situations where users' clocks were in fact set correctly.

And every time this issue was brought up, their canned reply was simple and essentially said: users are stupid and have no idea of how to set their clock so it would show the correct time. Can you imagine this level of "brilliance"?

Device time difference more than 600 seconds

Device time difference more than 600 seconds

Advanced Member
Members
PipPipPip
88 posts
0 warning points

Posted Today, 10:50 PM

[the rest of this post has been removed]

<again removed because it's an abusive personal attack instead of an on-topic point for the thread. You were told to drop it and you haven't. Final warning.>

Edited by Harold Feit, Today, 11:22 PM. Removed abusive personal attack.

Device time difference more than 600 seconds

stanha

Advanced Member
Members
PipPipPip
88 posts
0 warning points

Posted Today, 10:17 PM

I would agree to that.

There is one interesting release related issue here. In my experience and understanding, beta is basically a nearly stable official release of the product.

At beta phase, no new bells and whistles are added and no new functionality and, especially, the changes of behavior are to happen.

The very definition of term beta is bug fixes ONLY. It is a final problem resolution phase of the product.

Secondly, not making releases in small and gradual improvements, and accumulating massive amounts of changes and/or bug fixes, you risk to be exposed to the bomb effect, when things massively do not work and you see tons of issues appearing out of nowhere, as it happened with 1.3.67 release.

For me, personally, probably the worst "improvement" was replacing folders in both, Devices and Folders view with only the last element of the path. That is a disaster for me, since I have several shares that now have exactly the same name and once I switch to one of those tabs, I have to start thinking, moving my mouse around and trying to recall what relates to work.

That is just an additional load on perception. There is such a notion as "burden of perception". Perception is not "free". It takes energy and concentration, and, in this case, utterly unnecessary. I work with several totally different projects in totally unrelated area as a normal mode. That means, when I switch from the text editor, or coming back from site status statistics and some other things, it is like coming from a totally different world. Sometimes, it is not even easy to type as you just worked with totally unrelated language.

And, most importantly, what does this new folder presentation IMPROVES?
What does it add?
What does it clarify?

Instead of providing MORE useful information, it makes it LESS useful.
When I switch to Devices or Folders tab, which, in my opinion, more appropriate name for what is called My Sync. Because EVERYTHING in BTSync is about "My Sync", every single parameter and every tab in every dialog page.

When I switch from one tab to another, I'd prefer to have as complete of a picture of everything as possible without making ANY additional efforts on my part. I do not have spare or idle energy to throw around.

So, to me, to call this product beta stage is simply misleading.

About the only thing that is of any use or changes anything, as far as our own situation is concerned, is that additional tooltip that shows pending transfers. I like that one indeed. Because it really ADDS more information and gives a fuller picture of the state of the nodes.

I helped me to see exactly the reason I consistently see the incomplete sync with couple of nodes. Because it shows exactly which files remain to be uploaded, even though they were uploaded at one point, but were modified on the r/o node and they do not even realize what they did, most likely.

So, if BTSync still continues to develop and/or add all sorts of new bells and whistles, for marketing purposes or otherwise, it would be more appropriate to reclassify BTSync as Alpha state.

In beta state, you are dealing with basically a releasable product where only the final touches remain in terms of fixing bugs ONLY. It is a state of the product when massive testing occurs with all sorts of test suites or otherwise.

I am still willing to run it for a day or two, the way it feels right now, just to see what other changes of BASIC behavior I see in other nodes. And it looks like I am risking some very unpleasant potential consequences, looking at all the posts on this tread and other 1.3 related threads.

User interface design issues

File size and transfer progress in Transfer tab

stanha
Posted 29 March 2014 - 11:40 AM

It would be very useful to see the File size and transfer progress in percentage points in the Transfer tab. Otherwise, it is problematic to estimate various things and see the progress.

Plus, it could prove useful to developers from the standpoint of optimizing the performance related issues, especially in cases of smaller files (between 30 and 200k approx.)

Update #1

It seems the contents of Transfer tab should be logged into a separate log file so that one could see what was actually transferred. Just a few minutes ago I noticed one node coming on line after several days of absence and there was quite a bit of changes in the files since then. He had over 100 megs to download, which is strange. When I returned from the kitchen, he was gone and was fully synced. I'd like to know what did he actually transfer because it might indicate some issue.

It looks like he began to transfer some pretty big files that he should have had when he was fully synced. But there was an issue with duplicate folders at one point as a result of renaming, and I'd like to know what happened with this particular transfer today.

User Interface Design

Well, I'd like to discuss some issues with BTSync's GUI.

At the moment I'd like to cover the following:

1. General design issues
2. Device tab
3. History tab
4. Preferences tab

General design issues

These issues include the common accepted standards in GUI design, such as ANY displayable field, no matter what kind of field it is and in what column of what tab it is displayed is subject to copy to the clipboard operation.

That is the rule, and not some "luxury".
And it is the BASIC rule.

Many, if not most, well designed programs have the menu bar and a status bar.

In order to enable/disable logging or pause/resume the sync process, it is highly UNDESIRABLE to be performed via right mouse clicking on BTSync icon in the Windows task bar and then select a property choice.

Because it is HIGHLY unintuitive. It took me several days to realize that there are features like these available in BTSync, but ONLY if you use the right mouse click on a task bar icon. And I stumbled upon it only after reading lots of articles on BTSync forums. Why? Where is it described in the manuals?

But what do you have a concept of a program menu for? Why not utilize the STANDARD tools, methods and principles of well designed GUI? Why should I leave the main BTSync window to exit the program? What does it "buy you"?

Why can't you copy ANY field from ANY column from ANY tab in a dialog box to your clipboard, considering the fact that some of the things displayed you will have a hard time to replicate or copy to your editor or to the message on this forum?

Device tab issues

There are 3 columns currently on this tab. The column status is used to display either the amount to upload or the error message(s), in such cases, for example, as "Device time difference more than 600 seconds".

The nasty thing about that error message is that it is not displayed constantly, but toggles every 30 to 60 seconds, to the "Amount to upload", which, in case of timediff error shows totally incorrect amount that is equal to the total size of files in the share, even if that share is FULLY synced.

This design is not only "wrong". It is simply horrible. First of all, because of this totally wrong and horrifying message that tells you that your whole share have just disappeared, even though it is fully synced and all the files are there.

So, what could be a solution?

Well, first of all, if you decide to toggle the messages in the same column (Status), then you have to toggle them much faster, say, every 5 to 10 seconds, so the user has enough time to read it.

Furthermore, when you toggle it from the error message about timediff, to amount left to upload, you do NOT, under ANY circumstances show the user utterly incorrect data indicating that has to download the entire share, even if he is fully synced. If you can not estimate what would be left to upload in case of timediff error, you either show the CORRECT amount, if you can calculate it accounting for all the files that could be affected by the time diff range, or you simply say something lie: "?????", or "Can not be estimated in case of timediff errors".

But you do NOT, under ANY circumstances, present the user totally incorrect information, as it is done today. First of all, on the basis of what kind of logic you have made such a conclusion? Is there ANY logic involved in the process of displaying this utterly erroneous error message?

Or, you simply have 2 separate columns: 1 for the amount left to upload/download, if you can calculate it correctly and reliably, and then you have a totally different column for the error messages, that could also be used for the status messages, such as "Last contacted at ...", or "Fully synced at ...", or some meaningful things like these.

History tab issues

The message "Finished syncing with xxxx" is insufficient in the applications where there are multiple shares. Because it does not tell you which Share it was syncing with. So, it places additional drain on the energy, and, in case some Device has left the swarm when you are reading it, you have no idea which folder they were syncing.

This tab should show not only the messages that were the result of full sync, but also the messages of last contact and the amount left to be up/downloaded in case it could not fully complete the sync. Otherwise, you have no idea of whether the contact was actually made with some node, and whether that contact was successful, at least to some degree and resulted in some transfer. It would probably be great to see like "transferred NNN Kb, NNN Kb remaining".

Because the way it stands currently, is that you do not get ANY messages in History tab if you were not FULLY synced upon the last contact.

So, it seems to me, the message in Event field would look like:

Finished syncing "Folder name" with "Device name" transferred NNN Kb, NNN Kb remaining.

So, this message would provide the complete picture as to the activity and would also give you additional information on the state of the link with that node, such as whether it is functioning or not, whether files are indeed being transferred, even if link is OK, WHEN the last contact was made and so on.

For example, in the case of "Mac mini", described in "Device time difference..." thread, the other day, I have seen this node joining the swarm. So, I decided to help him out and changed my clock time by 1 hr. so he could sync, ans so he did. I was busy with all sorts of other things so I could not sit there for hours watching him to sync 1 Gig. After a while, he was gone, and when I looked in the History tab, there was no record that he did indeed completed the sync. I wanted to know how much is left for him to sync, but there is no information available, which means the operation was partially successful, but not entirely. But to what extent - only God knows.

Or, alternatively, you could add a new column(s) for "Folder name", "Transferred" and "Remaining", so, in case the device is too long to be displayed, it could be expanded in the maximized dialog view and Device column size could be enlarged by the user instead of being simply chopped off and replaced with "..." in case device name is too long.

Also, as far as user alterations of the files (delete, modify, rename) should also be recorded somewhere, even though may be not necessarily in history, simply because of the bulk of it. May be, may be not.

Because those kinds of alterations have profound effects and could have consequences that might not be easy to comprehend at the moment.

And all these modifications are simple, really, and could be implemented in a matter of hours, if not minutes, at least for the most part of it. But the benefit and informativeness of them is orders of magnitude more beneficial in terms of providing the user with as complete picture, as possible.

Does it worth the effort? - In my personal opinion, yes, indeed. Because the payback is something like hundreds to one.

Preferences tab issues

Device name edit field should be expanded, to say the least. Or, alternatively, the Device Message is to be added to this panel to be displayed either in Devices tab separate column, or, like in any half-decent program for Windows, in a STATUS bar at the bottom.

Status bar is one of the required elements in a well designed Windows programs according to Microsoft's design guidelines, known for tens of years.

The user simply MUST be informed about any messages to him, considering the number of various error conditions or undesired or unpredictable effects of various user actions, such as file modifications, deletions, renaming and so on, the results of which may be simply unpredictable to the "newbie" and he might not even suspect that he has done something that will prevent his node from being updated in the future.

So, there needs to be a way to communicate to him a message from other nodes, especially from the master (r/w) nodes, that could give him an advice on what to do in current situation in order to correct the situation and bring the sync to the normal and expected status and condition.

User Interface Design

#3 [User Interface Design: post #3] stanha

Advanced Member
Members
PipPipPip
107 posts
0 warning points

Posted 24 March 2014 - 12:44 AM

This is obvious to a five year old. Because it is a default behavior of the O/S GUI.

Yes, the folders tabs themselves are not copy/pastable. But there is a catch here: they are not DATA. They are just like labels, even though I, personally, would prefer to be able to copy the labels as well. For example, if you look at the Windows file - Properties tab, you will notice that generally, data itself is highlightable and copyable. But the labels to the left are not. Why?

Say, for example, I want to copy the entire contents of the Properties dialog box. I may have a huge tree of folders and I'd like to know the exact state of those folders and contained files written down somewhere in one of my documents, such as revision history, major updates to the system, and so on.

Now, do you think it is REASONABLE for me to be able to CTRL-A even on the entire dialog box and it should highlight all the text fields and labels, or to region-highlight any section of the dialog box and copy all those fields at once? - Well, why not? It would be great indeed.

Well, you can certainly maintain this position. But I suggest to look at even Windows explored property dialog, or nearly any dialog or window in the Windows GUI. Just try to mark some text, no matter what it is, and do a CTRL-C to try to copy it to the clipboard.

Once you do that, it would be great if you provide the statistics of all the cases where you could do it in relationship to the cases where you could not do it.

What you might find is that, even though it is generally true, that it is not a required operation, but, nevertheless, in any situation where you have things like labels: and any kind of message or data, you are in fact able to highlight that text, which means it is copyable to the clipboard, which you can verify.

The point here is: since some of the thing that are displayed in various columns of various tabs in GUI, contain text, that could be highly desirable to copy to the clipboard and paste elsewhere, such, for example, as hash codes or long device names or the state of the sync completion and so on, which you may preserve somewhere, but you can not copy/paste those things, it becomes a royal pain on the neck and royal waste of energy trying to save that information elsewhere.

The question here is simple: is it really NECESSARY? How much of a development effort would it be to make ANY data fields, editable or not, copy/pastable?

Yes, I am fully aware of that "new wave" trend, and Windows-8 is a perfect example of making a computer screen look like a smart phone, which, in my personal opinion is simply insanity. I have not installed the Win-8, but I hope the user is at least given a choice to go back to the "classic GUI" version, instead of being forced into this NWO insanity of imposing the idiocy on everyone and forcing them to submit to it, not even giving them a choice.

And I hope you don't mind me deciding for myself things like these.

Note: some further quoted text will be in italics because of the board limitations...

In Sync the traditional "menu bar" has been replaced by simple to navigate tabs, allowing direct access to the key functions of Sync.

But then you have a problem of sorts. How do I turn logging or transfers on and off and exit the program? Well, you need to KNOW that right clicking on the BTSync's icon in the Win task bar there will be a property choice popup.

Question: how would new user know that there is such a functionality available, but ONLY from that icon? Initially, and for days, I had to stop BTSync by hard killing it from the Task Manager, because I did not even notice the Exit option on the BTSync icon. And terminating the process from the Task Manager may have some very nasty side effects, such as abandoned named mutex or other synchronization objects and so on. Or, some drivers may be left running and all sorts of other nasty things.

I hope that when I exit BTSync from the BTSync icon Exit choice, it exits gracefully and fully restores the system to a stable state and unloads all the objects from memory.

Well, to some extent you can claim that debug logging is something for the "expert" users and so on.

But if you look at functionality of debug logging, currently it bulks everything into a single file. But, in my opinion, there is a need for some log of the transfers and their completion status, and, especially, of all the transfer errors discovered.

So, fine, debug logging, just as its name implies, might be considered as something for those who know more than the plain, ordinary users.

But transfer/error logging is something that is a must, in my opinion, and, probably, should be enablable/disablable individually for each folder, and, alternatively, in bulk - for "all" or "none".

But I am not going to argue this case at the moment, because it is nothing more than a royal pain on the neck to look at those logs, containing ALL sorts of information of not very useful value for the bulk of it.

But what I'd indeed like to see in some log, "debug" or some other, is:

1) Connection attemps with node names and hash.
2) Transfer completion entries for each file and their status (success or error, in which case, what was the cause of that error).
3) Definitely errors
4) Descriptive messages of transfer attempts, such as:
Sending the file xxx from folder yyy to peer "pear name". Something of this sort.

So, by merely looking at this log, any user, even not "expert" would be able to see exactly what is going on.

We need to keep in mind that BTSync is much less predictable in terms of the results than, say, torrent programs, and side effects of various user actions on the files locally may have some unpleasant side effects that he might not even realize.

So, it seems to me that informing the user in the most detailed way is one of the key points.

But some, if not most of those things that I see in the log, such as ping this, pong that, received response this, contacted the relay that, are not necessarily the things I like to see in the operation/transfer/completion log.

These are two separate animals to me.

To tell you the truth, "I could care less" about debug log as it stands right now, except for clear error messages, such as the case with timediff problem.

But I do care for the operation log. That is just the way I am, I guess. Actually, the last program I have developed in Jave automatically generates the running logs and you can click on the GUI checkbox in the main dialog and enable the "detailed" version.

And you know what? Since implementation of that feature, I was able to even find some bugs in a manner of minutes, that would otherwise take me hours, if not days.

Because all the exceptions of relatively important operational significance are ALWAYS automatically logged. And so when I finish the run for some job, which takes hours, and if I have any doubts of anything whatsoever, all I have to do is to open that daily log and look and find exactly the issue of my interest, down to the amount of memory taken by the process and remaining available to the JVM. Because those jobs are humongous in size and process hundreds of thousands of articles.

If I would not have that logging, I would not be able to develop that program to the level which it is in right now, and it is a tank grade program. The last time I have seen a bug, not even talking about crash of any kind, was years ago. I don't even remember how many years.

That is the point I am trying to address.

Proper logging is a tremendous help to both, the plain users and the "experts".
Simple as that.

And sorry, I am not willing to accept the arguments justifying the present state and quality of logging. To me, there is plenty left to be desired, saying it in the mild terms.

Because Sync is designed to always be running in the background, if you've got lots of windows open/programs running, it's often easier to exit Sync by right-clicking on the tray icon (which would take two clicks, 1 - right click on the tray icon, 2 - click "Exit"), than it would be to first locate/open the Sync UI (1 click), click a "File" option in a menu bar (another click), then choose "Exit" from a file menu (another click) = 3 clicks vs 2 clicks!

Well, I do not mind having "Exit" feature on the icon in a tray. But the problem is "newbies" are likely to have "no clue" about it. How do you assure that the most basic things are obvious to ANY "clueless" guy?

Well, the "classic" approach is the Program Menu bar. And I fully agree with it. At some early point, Win-95 period, Windows released a document or an article for the developers with the guidelines for "good program design".

To me, that document was probably one of the best documents and/or advice provided by MS in their entire history. And that document even specified what is the "must have" tabs on the program menu bar. They were, File, Edit, View and Help tabs, if I recall correctly. I was even baffled at that time being surprised with requirements to always have an Edit tab. In some applications, it seemed, there simply no such functionality available, like for example in the image viewers and so on.

And now I am saddened seeing this "new wave" approach in GUI design. To me, quite often it looks quite insane.

You see, with program menus that are consistently used across the multitude of apps, user eventually "gets" the basic idea when using any program, intuitively knowing what and where functionality is likely to be found where in the maze of the program dialogs, functions and features.

But with "new wave" approach, it seems "anything goes" for as long as you have some graphical representation of everything and all the program functionality is accessed like it is done on smart phones.

But quite often, those images are meaningless and are not consistent across the apps.

When I see a text, like "File" in the main menu, I already know, without even accessing that menu that there are going to be the standard choices available there, like open, save, save as, exit and so on.

But hey, everyone can do what they want and go insane with all those flashy images. And the only question I'd have is: why do YOU have to follow the pack? Is there any REASON that provides some ADDITIONAL benefits or clarifies something, say, for example, in the current version of BTSync GUI, where main program windows looks like it was designed on the base of the dialog class, which is not a good idea in my view? Because the most important program functionality of the GUI design is not available to the dialog derived classes, assuming they did development in MFC.

Dialogs are not designed for that, generally speaking. That is why they called dialog boxes, and not windows or views. Sure, you can design anything in any way you like. No one can force you. But... Well, this would be enough for now.

Screenshots are a good way to provide this information in the forum if you need to do so. Additionally, further information will be available in the Sync log.

The problem here is that you have a "newbie" that just installed your program and has not a slightest clue of anything. He may even have a hard time entering the key ("secret"), not realizing that is is done via Generate button, just for the sake of argument.

So, he fires up the program for the first time and manages to create his share. But then, magically enough, nothing happens, or, if he is lucky, he will see the other nodes joining the swarm. But it is not even intuitively obvious that his main operating tab should be a Devices tab. So, he might be sitting there watching the Folders tab and expecting some miracle to appear there. But he does not even see the main action. Sure, eventually, he'll get curious and will look in the Transfers tab and so on.

But, what if he has a problem? What is he to do?
Does he have to come to the BTSync forums as a must?
Does he have to read those sketchy "manuals" that are, sorry to tell you, are simply laughable as they basically describe nothing of the "real world problems" and issues.

As to the sync logs, you yourself just said above that logs are not meant for the "mere mortals". They are meant for the "elite" users and "experts".

So, what are you advising here to the plain, ordinary "newbie"?

There are improvements coming to the devices tab in Sync 1.3. Most notably, you'll be able to see exactly which files are awaiting being synced, or which are out of sync.

I'd LOVE to see that kind of level of detail. Again, with BTSync approach and dynamic nature of the process and unpredictability of some user operations and actions, it is simply a must. The more, the better.

As long as it is not all bulked in one huge log, but is properly and functionally separated into different kinds of logs, and I'd recommend an running rotating daily log, where new log file is automatically created when the clock time passes 00:00.

And again, in case of any kind of difficulties or problems the user may experience, the "Detailed log" checkbox would help indeed in my opinion.

This would be something more for the "Devices" tab - which shows current/active information, whereas the "History" tab shows action which has already taken place/completed.

I agree that a "time remaining" indicator would be a useful addition to have, but as with any suggestion/feature request - please post it in the dedicated wishlist thread.

I did post a few things on that thread with some ideas and/or suggestions. But, at this point, my feeling is that it is nothing more than a garbage dump that no one reads from the developers team. Some of those suggestions were written over a month ago I think and, since then, there were two or three program updates. But those ideas or suggestions seem to have gone straight into a black hole.

So, I no longer have any interest to post to that dump, filled mostly by all sorts of wild ideas and dreams of the people. Yes, all of this would be "nice to have", at least eventually. But right now there are CRITICAL issues in BTSync, that need URGENT attention in my opinion, and if even the most basic things and suggestions simply go into a black hole, then what do you expect me to do?

I can not even convince people that timediff error is nothing short of the disaster, at least in my case, and I mean it in quite a literal sense.

About all I see in terms of response: "tell those clueless to set their clock right". But HOW do I tell them?

I proposed some sort of primitive messaging facility how long ago?
And what is the result?

So, HOW do I talk to all those "clueless", and what if it turns out that their clock and even time zone are set correctly? And now I am nearly convinced that they are in fact set correctly. But which node sees what time zone is not a universally and globally available from some central "authority" for all the places in the world. For example, in Windows, the time zones of several cities in different countries are bulked together. But this may be utterly incorrect. Because in some countries the politicians constantly pass the laws about the daylight saving time, and so the Microsoft time zone, chosen for your city may be incorrect, compared to other sources or the local laws.

The problem here that yes, it is already pretty long. But if there is some problem with some node, you might have to somehow explain to them and possibly give them the URL where they can look it up, and that message might be longer than 94 chars, which, btw, has been even shortened in later versions.

The problem we are dealing here with is the case where BTSync refuses to move a finger in order to provide even the most primitive facility to display some more or less descriptive message in the device name, which is visible to all other users. This way, they have a chance to see that message. So, that was a "solution" that would take about 5 minutes to implement, but would allow to display a much longer message. Because I want to tell them what is wrong and also provide the url to exact chapter and topic in the detailed user manual where they can read a full description of their problem. And for that, I'd like to see preferably about twice as long the device name edit field.

Technically, the physical size of the edit field does not imply the maximum amount of text you can put there. It is already auto-scrolling. So, you can put even a 1 k long string into it and if you want to read all of it, you can simply do a standard select-all and copy operation to copy it it to the clipboard and then paste it into your favorite text editor to see the full text.

What is the problem with this?

And again, it was meant for the case that BTSync is not willing to move a finger to provide any other way to display some device message. But even having a separate column on device tab for device message should take no more than a few minutes to implement.

Yes, this is clear even to the five year old, without even mentioning. But you need to first of all create a design, as quickly as you can, that fulfills the most critical needs of informing the users in case they have ANY kind of problem or difficulty.

And you need to provide them with the FULLEST picture of everything.

Question: how do you do that?

What EXACTLY do you propose to be done, if anything, but waiting for new instructions of the party line kind, telling you that you are simply in a state of delusion, and everything is nice an rosy?

Well, I am glad to hear that indeed, and I hope when I finally see it, I will not be disappointed.

But I've been hearing these "real soon now" messages for over a month and it reminds me the politicians forever talking about the "bright future".

But where is the substance I ask you?

Where's the beef?

What are the top items on your "priority list"?

I'd LOVE to see that list. Who knows, may be my time would be better spend doing my main things that I do, instead of trying to prove that I am not a donkey with long ears.

The love affairs come and go, and you can spin your wheels, getting nowhere really fast, for only that long. And then come the "boom" part of the affair, and you simply decide to switch to Java Open Source project, such as Hive2Hive, where you can throw together an application compatible in functionality and with even better and more reliable security arrangements in less than a month, in my current estimate.

My hands are beginning to itch. But not enough at this moment and I am already overloaded with things that MUST be done with other areas of my activity.

So...

User Interface Design

stanha

Advanced Member
Members
PipPipPip
107 posts
0 warning points

Posted 24 March 2014 - 02:13 AM

What?

"However, personal attacks, "flaming", threatening or abusive language"?

Hey, I tell you what, now I really feel that energy going full blast.

Well, what can I tell you, but: it dissolves, no matter what.

Yes, you did indeed help a little bit for my fingers to move.

But, you see, it is simply dishonest to edit the posts of other people.

Yes, you can delete them as a whole, and it seems to me that you have given that "authority".

But I operate in different lands and their name is:

Truth.

Let me ask you: was it you, personally, who has deleted one of my posts yesterday on the other thread?

It seems you have some problem here of personal kind. But you see, in my opinion, you did not provide any solution to any of the issues that are there on the table, regardless of the "priority list".

As I said before: I am only interested in solutions and I like to see things working on the proper, and that is fully professional level. Because just today I have seen 3 people out of 5 on one of my active shares having a problem.

And I do not know how many other people have abandoned the efforts to download the information and receive live updates.

My rough estimate is that it could be 50% of all the nodes I have seen on that share. Sure, statistics is just that - statistics.

And I have provided so many versions of solutions to some of the most critical issues for the general users, that, sorry, but you have to have some other reason to concoct your last message.

So, Be it.

User Interface Design

Harold Feit
Community Manager
Administrators
4,582 posts

#6 [User Interface Design: post #6] Harold Feit

Posted 24 March 2014 - 02:19 AM

What can a poor guy like me say, but "Seig Hail, my fuehrer"!

User Interface Design

stanha
Advanced Member
Members
PipPipPip
107 posts
0 warning points

Posted 24 March 2014 - 02:41 AM

Unfortunately, you are talking to the wrong party in this.

The message I saw as a followup to my post was a total insult.

And this is "official" now:

The support of BTSync product, in any way, shape or form is totally withdrawn by this party.

Whether it means something to some of you or not, this is not of our concern.

BTSync was given what I would classify as tremendous degree of support by the powers that I would not even like to mention here.

That support is no more.

Incomplete sync issue

Node Update Logic Is Profoundly Broken And Needs Redesign

Basically, from what I see, the node update logic the way it stands right now it fundamentally broken as a result of bad design.

It inherently creates the inconsistency across the nodes. The very concept of database to accommodate and save the update state of the share is not a reliable concept in case other nodes do file deletion, renaming of modification.

If, for whatever reason, the database gets damaged for example on the r/w node and one has to delete and re-add the share, all the memory of the last state of the share is lost and all the nodes become totally inconsistent in respect to what they should have in terms of the latest state.

Therefore, unless you implement a database journaling system, you can not possibly guarantee the consistency or database recovery. If database is damaged, which happens no matter what, how do you know which exact files are subject to update and propagation?

Secondly, as it stands right now, there is no such a concept as a "reference node". Therefore, it is impossible to determine what should be the correct state of all the files that might have come and gone as a result of user modifications, deletions and file renaming.

Furthermore, the r/w node can not be considered as a "reference node" or a "master node" with present logic, as it can not fully control what happens in the r/o nodes if they have made some file modifications, deletions or renaming.

Once the r/o node has done some of these operations, it is no longer controlled by the r/w node in respect to updates of the modified files, and, therefore, it is no longer consistent with the r/w node.

From what I see, with current update logic, about the only way to make it work reliably is to prohibit stopping of updates in case r/o nodes performed any file modifications. Those modifications should be undone and files should be restored to exactly the same state as they exist either on r/w nodes or their exact or the "best fit" copies.

If user renames some files, those should remain in renamed state, but the files with the original file name should be re-downloaded.

If user deletes some files because he does not want to download them any longer, he should add them to the .SyncIgnore. Otherwise, they will be re-downloaded again.

If user wants to modify some file under original file name, his modifications will be lost as the original versions will be re-downloaded. Therefore, if he wants to modify them, he should rename them and then edit them the way he wants. But the originals will be re-downloaded in exactly the same version as on the r/w nodes.

Therefore, any files that are not in .SyncIgnore, will be restored to their original state as on r/w nodes.

Else, I see no way, even theoretically, to eliminate what is called a "database inconsistency problem" as known in relational database theory.

It is simply logically impossible with current update logic and design. The very idea of BTSync database to maintain the latest state of the files is inherently unreliable, unless journaling system is implemented. Because, first of all, databases eventually get damaged for all sorts of reasons and only journaling systems may prevent the effects of damage and restore the database to its correct state.

Furthermore, there are two copies of the state - one in the physical file system, as eventual reality of what is going on, and the other one, that may differ as a result of user file modifications, in the database.

On the top of it, several nodes may have totally different state of various files as a result of local file deletions, renaming and modifications.

As a result, you have a total inconsistency across all nodes and, in case of general public situations where you can not even contact the other nodes and tell them what to do to fix the situation so that their nodes fully update, what you have is unresolvable problem logically and you can not possibly assure the consistency.

From what I see, in current design, about the only way to resolve the consistency issue is to make it a MUST to use the .SyncIgnore file in case users do not wish to receive the updates on some files for whatever reason.

All other user modifications that are inconsistent with the r/w nodes should be undone by redownloading the user modified files if those modifications are not in the .SyncIgnore. That is the only way to assure consistency across all nodes as far as I can see, at least with non-journaling database system.

The "bottom line" is this:

The current design and update logic will never work, in principle, at least in general public situations where you don't have access and control of all the nodes and their data.

Defensive Open Source: Proposal to go Open Source

Posted on:
May 15, 2014 08:53 AM

OK guys. This is beginning to get on my nerves.
BTSync is FULL of bugs and it is probably the most unreliable and inconsistent program I have ever seen.

And the problem with it is this:

I have provided quite a lot of proposals to BTSync with specific suggestions, analysis of the MOST critical and literally vital issues and all of it was ignored. In some cases, they even promised to take care of one of the nastiest design flaws related to timediff error, which is clearly a flaw in design in my opinion.

But ALL of it was simply ignored, and its been months now since these issues were raised.

And the sorriest thing is that we can not even fix their bugs because the source is not open and those bugs are most critical, at least to our applications and use. So, they simply MUST be fixed, or else...

Now, here's the kicker:

BTSync may be reverse engineered in no more than a month from preliminary estimate.

The bulk of the program is based on Torrent protocol, which is an open source by now. The rest of the program is not that well designed logic and some minor encoding, hashing and node update logic, which is simple enough as it looks right now.

So, what we have on the table is the proposal:

Proposal:

Either BTSync goes open source IMMEDIATELY or it will be reverse engineered by the most experienced and senior level developers in the Silicon Valley, which means in the world who does not know what it means.

That source code will be released to the public and if something is found in it that even remotely hints on their alleged relationship with the agencies, such as NSA, the Bittorent as a company will be disgraced and the whole world will know about it.

So...

The clock starts ticking NOW.

My humble suggestion: think well before you try to challenge this offer.

Edited version

Posted on:
May 15, 2014 08:53 AM

OK guys. This is beginning to get on my nerves.
BTSync is FULL of bugs and it is probably the most unreliable and inconsistent program I have ever seen.

And the problem with it is this:

I have provided quite a lot of proposals to BTSync with specific suggestions, analysis of the MOST critical and literally vital issues and all of it was ignored. In some cases, they even promised to take care of one of the nastiest design flaws related to timediff error, which is clearly a flaw in design in my opinion.

But ALL of it was simply ignored, and its been months now since these issues were raised.

And the sorriest thing is that we can not even fix their bugs because the source is not open and those bugs are most critical, at least to our applications and use. So, they simply MUST be fixed, or else...

Edited by GreatMarko, on May 15, 2014 , 12:47 PM.
Post truncated due to inappropriate, agressive, and threatening tone in the remained of the post

Final message to BT

Posted on: May 16 2014, 08:22 PM

OK, I am willing to put my energy into it, for the last time and try to explain what that message meant in reality and considering facts, and not fiction and fabrications of all kinds.

First of all, it isn't a treat. There is a difference between a threat and reality. This is how it is and once top level developers commit themselves to making a compatible product, that's it. Nothing can stop them and it becomes just a matter of time. Here's just a few facts.

Fact: OSF (Open Source Foundation) has made a BTSync compatible product one of their TOP priorities and "hot" projects. That, by itself, means that some of the best developers in the Silicon Valley will certainly get involved, and I know some of them personally, including the inventor of the UDP tunnel, some creators of computer languages and so on.

Fact: To experienced developers decompiling the BTSync represents no problems whatsoever. In fact, just within the first day of analysis, all the "bowels" and key algorithms and some "esoteric" and even some quite shocking things were discovered.

Considering the program size of about 2 megs, decompiling and reconstructing the program should take no more than one month approx, and that is based on knowledge of specifics of the task.

Fact: It will be impossible to create a BTSync compatible product without decompiling because of some of the specific code, related, for example, to hashing, name mangling, some semi-encryption (the encryption itself is handled by the standard encryption related libraries), etc.

Fact: Sync approach on P2P basis is vitally needed, for example, in global information warfare, carried out by the "dark forces" in attempts to suppress the truth of what they have done on this planet.

So, as far as copyright type of issues and well-being of humanity and its future go, there is an "individual good" and there is a "common good" on behalf of all mankind. When "individual good" of any individual, party or even a country goes against the good in terms of mankind as such, the "individual good" can not possibly prevail, for this is the Law, Law of Life itself. It isn't an "esoteric theory" of "talking heads".

From the legal standpoint, even though am I not competent in that area, but did have to deal with one competent attorney working exactly on the expert systems project related to copyright issue, there are provisions in the law to accommodate for principles of the "common good". Furthermore, competent attorneys are of the opinion that there exists no copyright violation in cases where the amount of creative work "borrowed" is less than 30% of the total volume of creative work.

That directly translates into near certainty in legal terms that to "borrow" small sections of code, particularly of those convoluted hashing, mangling and semi-encoding, raise no legal issues.

Furthermore, there exists a possibility that BT itself is in violation of the copyright law in the Linux version of their product. Because they, nearly inevitably, used various tools, compilers, development environment, libraries that are subject to Open Source provisions of their licenses, some of which may require the source to the BTSync to be opened. Else, they are in violation of the licenses. Certainly, this issue has to be examined further by the competent attorneys.

Fact: If BT corporation does not release the source code, or at least the key issues and algorithms related to security and those convoluted things, then those developers that have committed themselves to create a compatible program will have no choice but to look at decompiled binaries, which is not illegal.

Fact: the result of their work WILL be released to public by the sheer fact of the Open Source principles, and this is not a threat, but reality.

Now, if BTSync does decide to release the source code or at least the vital parts needed for creation of a compatible product, it may be to THEIR benefit. Because, again, just from the first look at the code it has been discovered that there are some "unusual things" being done from the standpoint of security, privacy or tracking, to put in the mildest terms. But before the conclusion can be made, a detailed code analysis has to be performed.

This, in turn, means that if BT releases what is needed to create a compatible product, they may avoid that close scrutiny, which, in case there ARE some "hidden tricks" present indeed, this information will become public. And again, this is not a threat, but merely an inevitability of the outcome.

It is not likely that BT has any intentions to release neither the entire source code, nor even the relevant parts to creation of compatible products. Simply because they are likely to hope that they can "corner the market" in the "entertainment industry" which they use for the purpose of commercial gain by distributing various "packages" or "bundles" of indie artists, and so on. So, it is likely that they do not want anyone to be able to also provide the tools with which these "bundles" may be distributed by others. Probably because of the fact that all the nodes by default are known to BT trackers and it is likely that they get the percentage of of the "pie" based on counts of transfers or at least their estimates.

So, here you have in addition a monopolization of the market issue with all its consequences.

Now, when you put all these things together, and there are more issues, then what do you have as a result?

"THREATS"?

or

REALITY, that is about to bite the BT pretty hard if they maintain their present stance.

Because what we have right now in BTSync is pretty immature product with ALL sorts of problems and you depend on the "priority lists" of BT and what ANY of you think is a VITAL issue, may be not even on those lists, and even if they are, how can you possibly know when these issues will be resolved, if ever?

Furthermore, even to classify BTSync as a product in Beta stage is misleading and may have legal consequences, especially if combined with the issues of "cornering the market". Because in beta stage what you are dealing with is a product in near-ready to be released state. In beta stages, there is no longer any development done, nor there is any adding of "bells and whistles" to existing code base.

Not many people would dare to download and install a product in Alpha stage of development. But nearly everyone would be willing to download the betas, in the understanding that this is already a nearly-ready to be released product and, therefore, worth a try, which is NOT the case with BTSync, by ANY measures.

Therefore, it raises the legal issues of "bad intent" and luring the potential customers into a trap and thus gaining "unfair advantage", at least over those manufacturers and developers who are honest and do not advertise their products as nearly ready to be released, while in fact they can hardly be classified to be even on the level of Alpha stage.

As it stands right now, even if you COULD fix some of those critical issues yourself, or in a joint Open Source community effort, you simply can not. There is NOTHING you can do, but to cry and complain and bitch and moan, even while you see nearly half of the nodes on some shares either in the state of incompletion, which is a disaster, or stuck for days and not downloading the share, or showing the timediff errors and so on.

How do you like to see people LEAVING the share simply because they think the whole thing simply does not work? Well, in OUR case, it is not only a disaster, but a disaster on a global level.

"The bottom line" is this:

No "individual good", such as desires, greed, ambitions or attempts at "extraction of individual benefits" may go against the COMMON Good of Mankind as such.

For this is the Law of Intelligence itself, the law of Life itself, the law of Creativity itself, which can no longer be violated by "the dark side".

That game is over.

Now is the time for recovery, reconstruction, healing of nature abused by these forces and reestablishment of the Laws of Intelligence. The present corrupt system of control, domination, greed and "ones own interest" is being disassembled as we speak.

I hope this post clarifies the matters and gives the BT staff yet another chance to have a hard look at the issues. Otherwise, the "interests" of BT as individual entity will not and can not prevail, even in principle. This can not possibly be allowed under the circumstance. And this is not a "threat" but inevitability.

The original message of the party that writes this to you, which was interpreted by some as a "threat", was in fact an energy cluster and a message, that was meant "to ring some bells" in your consciousness.

Because the individual that writes this to you made certain commitments which he so far honored in full account and which he has no intention to dishonor as of this moment.

Even if the "dark forces" decide to KILL him, it won't change even iota. Because all the necessary measure have been already implemented and the processes that have started as a result are unstoppable, and for several reasons.

And this message is likely to be one of the last messages these forums will see. The work has been completed and the rest will proceed in "automatic" fashion.

"Munched by admin" a abusive

[Note: this is what was left of the previous message (above), within few minutes after its publication.]

"Empire strikes back": BT response to the "munched" post

Question arises: are these people REALLY insane, or just pretending?

Useful information and links

Protocols

• Bittorrent Protocol Specification v1.0
  https://wiki.theory.org/BitTorrentSpecification
• The BitTorrent Protocol Specification
  http://www.bittorrent.org/beps/bep_0003.html
• Azureus messaging protocol
  http://wiki.vuze.com/w/Azureus_messaging_protocol

DHT

• DHT Protocol
  http://bittorrent.org/beps/bep_0005.html
• Kademlia (DHT)
  http://en.wikipedia.org/wiki/Kademlia
• Mainline DHT
  http://en.wikipedia.org/wiki/Mainline_DHT
• Kademlia: A Peer-to-peer Information System Based on the XOR Metric
  http://www.cs.rice.edu/Conferences/IPTPS02/109.pdf
• Kademlia (search results)
  http://en.wikipedia.org/w/index.php?search=Kademlia&title=Special%3ASearch&fulltext=Search

Torrent files

• Torrent Loader
  http://sourceforge.net/projects/torrentloader/
• TorrentEditor
  http://torrenteditor.com/
• BEncode Editor
  https://sites.google.com/site/ultimasites/bencode-editor
• What are peers, seeds, torrent, tracker, DHT, Peer Exchange (PEX), and Magnet Links?
  http://wiki.bitcomet.com/peers_seeds_torrent_tracker_dht_peer_exchange_pex_magnet_links

Magnet links

• Extension for Peers to Send Metadata Files (magnet links)
  http://www.bittorrent.org/beps/bep_0009.html
• Magnet URI scheme
  http://en.wikipedia.org/wiki/Magnet_URI_scheme
• How do BitTorrent magnet links work?
  http://stackoverflow.com/questions/3844502/how-do-bittorrent-magnet-links-work
• What's the difference between a torrent file and a Magnet link?
  http://superuser.com/questions/199970/whats-the-difference-between-a-torrent-file-and-a-magnet-link
• Generate a torrent/magnet link from a single file in a torrent collection
  http://stackoverflow.com/questions/16333165/generate-a-torrent-magnet-link-from-a-single-file-in-a-torrent-collection
• Magnet file to .torrent as hyperlink is to .html?
  http://stackoverflow.com/questions/9350804/magnet-file-to-torrent-as-hyperlink-is-to-html
• Create Torrent Magnet Links With uTorrent & Magnetizer
  http://www.addictivetips.com/windows-tips/create-torrent-magnet-links-with-utorrent-magnetizer/