Synchronization

By Mike Blanchette
Vice President and CTO of Puma Technologies

They are sometimes called "Personal Digital Assistants" (PDAs), Electronic Organizers or Handheld PCs (H/PCs). They are small, lightweight, and powerful handheld devices such as the Windows CE-based H/PCs, the recently announced Windows CE Palm PCs, a wave of new "smart phones" (from Nokia and AT&T) and "smart pagers" (from NEC). With them, mobile professionals have the ability to keep key contact, schedule, and task information with them wherever they are. These devices are also called "PC companions," but to warrant that name, they must become seamless extensions of the user's PC environment.

This means that handheld users must be able to easily and seamlessly synchronize information in their mobile device with the information in their desktop PC, notebook PC, or network server. Given the importance of the data involved, this synchronization must not only be fast, it must be accurate and reliable -- all the time, every time! A good synchronization solution looks simple to the user, but there's a lot going on behind the scenes in a synchronization program. Let's take a look at some of the issues surrounding effective synchronization solutions, and the problems faced in creating them.

"One-size-fits-all" synchronization a tall order

There are a large number of mobile devices and applications on the market today, and many more are on the way. The Windows CE Contacts, Calendar and Tasks applications are standard in the CE-based H/PCs, but the desktop PC PIM software they will have to synchronize with varies. On top of that, not everyone uses H/PCs. (And some H/PC users employ contact management software instead of the built-in PIMs). In addition to all this, both the devices and applications that they use today are certainly going to change in the future, requiring an architecture that is both adaptable and extensible.

Because of the differences in these devices and applications, a "one size-fits-all" synchronization solution is a tall order. It's also a necessary order. A synchronization solution that works with only one or two devices or applications simply cannot satisfy the variety of needs of a large number of users. Developing such adaptable and extensible synchronization architecture, however, is a daunting task, requiring enormous effort and insight. Not only must it accommodate the device/application disparity that exists today, it must take into account devices and applications that have not even been developed. Is it easy? No! Is it required? Yes!

Getting at the data

Synchronization solutions must be "content-aware." That is, they must be able to understand the data to be synchronized at the record and field level. The challenges in accomplishing this fall into two distinct areas:

Data Formats -- It is highly likely that the structure of the data on the device and applications to be synchronized will not be the same. The synchronization solution must understand:

1. The details of the data formats in both applications (i.e., what data fields each has, how long the fields are, what type of data field each is, etc.).
2. The interrelationships between and among the fields and records (i.e., which fields in one application are associated with which specific fields in the other application, whether the synchronization has to split or combine data from the field(s) of one application to the field(s) of another, etc.)
3. How the PIM application stores its data (the hidden data structure of an application that the user never sees).

Synchronizing data from the same application on two different devices is relatively easy. Synchronizing entirely different data formats and structures between disparate devices and applications is far more difficult. And, given the lack of standardization in data formats, and often the lack of published documentation on them, this problem grows exponentially as the number of devices and applications increase.

Over time, this problem may get a bit easier as standard data formats, such as those from the Internet Engineering Task Force (IETF), are defined. However, by themselves, standards are hardly a panacea since they only solve a part of the problem, leaving additional and/or vendor-specific features up to the synchronization developer to support.

Application Programming Interfaces -- In order to even access the data, an application programming interface (API) is normally used. Here, issues such as unique and permanent record IDs for each record, record add/change/delete flags, and field-vs- record-level access capabilities come into play. With a robust API, one that has been specifically designed for synchronization (e.g. it has unique record IDs and add/change/delete flags, and other necessary features), the problem is more straightforward. However, the reality is that in many cases, particularly with many existing applications, the API has not been designed for synchronization, and does not contain all of the necessary elements. In these cases, the synchronization solution must be that much more intelligent, often working around the shortcomings of the API. And in many cases, the device or application is "closed", and does not have a published API, making the problem that much more difficult.

While work is under way to define a few useful APUI standards, software vendors often provide access to their best features exclusively through a proprietary API. The reality is that synchronization solutions will continue to imply a mixture of standard and proprietary interfaces for the forseeable future.

The synchronization process -- now the fun begins

Once the complex issues of accessing and understanding the data between disparate devices and applications have been solved (a major accomplishment to be sure), the myriad of issues revolving around the synchronization process itself must be faced. It is in these areas that many solutions fall short, delivering less than reliable, accurate synchronization. The following are just some of the issues that the synchronization developer faces.

The features don't match -- what do we do now?

Often times a feature in one application doesn't work the same way, or do the same things as the analogous feature in the other application, and/or vice versa. When this occurs, the synchronization designer must figure out a way to make similar features, implemented differently, or perhaps even missing on one side or the other, work in concert even though they were never meant to. One good example, but not the only one, is "recurring calendar events" (e.g. a staff meeting every Friday at 9:00 AM for the next 2 months). When one device or application supports recurring events, but the other side does not, the synchronization solution must know how to "fan out" the recurring event from one side to individual events on the other side. It must also be able to perform the opposite action when going the other way. To compound the problem, the synchronization solution must also take into account "exceptions" to the recurring event (e.g. there will not be a meeting three Fridays from now), and maintain that exception list across future synchronization operations. If this is not done properly, the user will end up with missing appointments on one side, and multiple duplicate appointments on the other. Remember, accuracy is all important in synchronization!

We both made changes -- who's right?

While on the road, you change next Thursday's meeting from 11:00 AM to 10:00 AM. Back at the office your administrative assistant (for whatever reason) changes the same appointment to 1:00 PM. When you get back to the office, you synchronize the two appointment books. It's easy for the software to recognize that the values are different, but how does it "know" that both of you made a change and, therefore, there's a conflict. The answer is that many times, the synchronization software must have historical knowledge of previous synchronization operations to know exactly what changed.

In the above example, the synchronization program would consult its own historical records of your appointments on both sides, note that they both reference the same previous appointment (i.e., the former 11:00 AM appointment) and infer that you both made changes. The program would then prompt you to resolve the conflict. On the other hand, if only you had made the change (11 AM to 10 AM), the synchronization program would note the discrepancy, check the historical records, note that there was no change on the office side, and infer that the change was a simple update from your side, not needing conflict resolution.

Some synchronization applications provide "change" flags with the data to simplify this issue, but many do not. Consider even the simple case of deleting the contact record of someone you no longer work with from the mobile device. Without either change flags or historical knowledge, how would the software know, at the next synchronization, whether to delete the record from the application, or add it back to the device. In these cases, the synchronization software must be able to figure changes out on its own, or the accuracy of synchronized data will be compromised.

A rose by any other name

Another key issue to consider is the actual "mapping" of the data fields from one device or application to another. What one application calls "Title", the other may call "Position." You might not give this much thought, but the developer of the synchronization program had better! If he or she doesn't, you'll discover it soon enough.

The developer of a good synchronization solution will have already looked at the different applications it will be synchronizing and will have already figured out what data fields get mapped where. Most of the time, you will be able to go with the synchronization program's defaults, but a good synchronization program will let you change the field mappings when necessary, and even protect you from attempting an invalid mapping (e.g. mapping a date field to a single character type field). A flexible field mapping capability is critical with applications such as Outlook, ECCO, GoldMine and others, which allow the user to define new fields, or rename existing ones. Without such a field mapping capability, implemented properly, there would be no hope of accurate synchronization in many cases.

I'm Special, Can You Help Me?

Often, applications have unique capabilities that differentiate them from the competition. It might be the ability to store information in sub-folders, emulating the directory structure of a PC, or the ability to organize to-do items in an outline fashion for added convenience. Synchronization solutions must accommodate these application-specific features, and "map" them to the capabilities of the individual device, or users will most certainly view the solution as less than optimal, and the device itself less powerful for his or her usage model. As these features can take many shapes and forms, this is one of the greatest challenges facing the synchronization designer, requiring a high level of inventiveness and creativity.

Maybe I Don't Want It All

There are many cases where the user may not want to synchronize all of the information in the application database with the device. Perhaps the actual database contains too many records for the device to hold. Or, it may be a group database, and the user only wants to synchronize the records for which he or she is the owner. It might be that the user stores business records in one database, and personal records in another database. In order to satisfy these, and other similar user needs, synchronization solutions must provide a powerful, but easy-to-use data filtering capability, allowing the selection and synchronization of only the set of records desired. Without such a custom filtering capability, the user would be stuck with an "all or nothing" solution, and too often this just doesn't fit the bill.

Doing It A New And Different Way

While not strictly a functional issue that users need be concerned with, intellectual property rights, or patents, are, none the less, a major issue for developers. As with other technologies, significant and substantial patents exist in the area of synchronization that challenge the creativity of even the most savvy developer. Not only must they properly address all of the technical synchronization issues we've discussed here (and all the ones we didn't have time to discuss), they must do so by not violating the existing patents that other developers already hold. This, in fact, may be the most difficult challenge of all to overcome.

These are only some of the issues that face synchronization developers. There are many others, equally as challenging. An easy-to-use, well crafted, and feature rich synchronization solution may not look like rocket science to the user, but it does to the developer. However, armed with the knowledge of what to look for, and what questions to ask, mobile professionals can more easily find the right solutions, from the right vendors, to satisfy their needs. It's too important a problem to be left to anyone but experts.

ABOUT THE AUTHOR

Mike Blanchette is the Chief Technology Officer at Puma Technology, Inc. Mr. Blanchette is responsible for leading development of the core synchronization components used in all Puma products. He joined Puma Technology in 1996 when IntelliLink Corporation merged with the company. At IntelliLink, Mr. Blanchette served as President and Director of Engineering. He holds a Masters Degree in Computer Science from Boston University.