[tool] Merge annotations (comments) between documents

[Mathew]

This tool attempts to merge all comments in multiple pdf documents without overwriting more recent changes, and detecting conflicts (ie comments simultaneously edited).

Background
If more than one person is adding comments to a pdf document, they either have to share a single document (but then it's locked when one person is editing it), or there needs to be a way to merge the comments in multiple documents. Other pdf software has set up subscription-based online editing to handle this, but then you need to trust your documents to their servers, and it's a proprietory solution. A simple brute force method is to export an FDF from all the documents, and import all of them into the document you want. But that will overwrite anything that was changed in the second document, and does not handle deleted comments.

Installation
Extract the zip and save in the Javascripts folder either in the application folder, or in %APPDATA%\Tracker Software\PDFXEditor\3.0\Javascripts More detail here: viewtopic.php?p=196006#p196006

mergeAnnots v1.3.zip

Usage
After restarting PDFX-Change, it will add a button in the "Share" ribbon, (or in the Comments menu in the classic UI):

image.png

Open the documents you want to merge, and run the tool. It will open a dialog:

image(1).png

The left side lists documents that will be merged. If you ran the tool before, it will list the last documents you merged on the left, prefixed with [File] to indicate files that are not currently open. Below the list of files, it shows some brief information about the selected file.
On the right side is a list of the currently open documents. Use the < or > buttons to add or remove documents from the merge list on the left side.
The 'Browse...' button opens a dialog to open a file directly (if you didn't open it before running the tool). PXCE will pop up a security alert before allowing the script to open files. You can stop this for a particular file, directory or drive (I have a server mapped to a drive letter, and under Preferences > Security I added the drive letter as a trusted folder.)


At the top left is a button to set the merge settings. For now, I recommend keeping "Show merge conflicts dialog for all merges" so you can see what it's going to do before the merge happens:

image(2).png

Merge Data file
The script can save a data file either on the computer, or as an attachment in the document to improve the speed and accuracy of merges.

• If saved as a document attachment, it's titled 'commentMergeData.txt'.
• If saved on the computer, you need to allow the script to read/write to the file. You'll get a security alert about the script trying to access /C/Users/[User Name]/AppData/Local/Temp/PDFXEdit/CommentMergeData.pdf

The merge data file allows the following improvements:

1. Keep track of which annotations have been deleted: If there is no data file, the script will assume that if an annotation is missing (i.e. deleted) in one document, that it is new in another document, and will add it back (so to delete an annotation, you'd need to manually delete it in all documents before merge)
2. Determine if an annotation has been updated since the previous merge: If there's no merge file, it assumes everything was updated and compares modification dates to all other files. This slows the merge, but if only the modification dates differ, it won't update the annotations in PDFX-Change before build 388.
3. (in PDFX-Change before build 388) Keep track of modification dates: When the script adds a missing annotation, it can't set the modification date, so that annotation will appear newer in the next merge; there are cases where this could lead to overwriting changes in another pdf during a subsequent merge. (If only the modification dates differ, it won't update the annotations in PDFX-Change before build 388.)
4. Optionally save a log of changes.

Conflicts
Depending on what the merge settings are, a Merge Conflicts dialog will show before the merge starts:

image(3).png

For all merges, the script finds the most recently modified annotation, and sets that document as the source for that annotation. Under Merge it will show ?? if there's no default action for that conflict in the settings. You'll need to either set it to Skip or >>>>> (Use Source) to proceed with the merge.
The buttons across the top allow you to set what happens to that annotation in the merge:

• Skip the merge for that annotation,
• --> Go --> apply the source annotation change to the destination, or
• Change Source move the currently listed destination document to the 'source' (ie revert to an older version)

At the bottom, there is more detail about the currently selected annotation, and buttons to show the annotations.


Limitations/todo
This is a tough nut to crack. Handling the various conflicts is difficult to figure out (for my brain), and there are quite a few aspects of pdfs that are opaque to javascript. I wanted this tool to be able to handle merging more than just two documents at the same time: This makes handling conflicts particularly difficult.

• I've only tested this in PDFX-Change build 386 & 404.
• If the documents have different number of pages, the script will fail! I don't know how to know if pages have been moved, inserted, or deleted - the script currently assumes all page numbers between the documents match. I may try something like adding a hidden marker on each page to identify it, but I don't know what the best approach is. There may be some internal identifier to pages that I don't know, or a way to identify them; if you know of a way, please let me know.
• There's no built-in way for javascript to set which comments are in front of the others (ie their z-index). One can work around this by temporarily moving annotations off the page, and back on in the desired order, but haven't added it to this tool yet.
• JavaScript can't add stamps the way it can for other annotations. If you try, the application will look through the currently installed stamps, and try to add the same stamp. This is OK unless the stamp isn't installed on the merging computer or is a pasted image (in which case it will fail) or is a dynamic stamp (in which case it will pop up the dialog associated with that stamp instead of just copying it). I've made a workaround (that requires the source document be saved) copies the whole page, moves the stamp to the desired page and deletes the copied page; but watch out: It requires a registered copy of PXCE! If you want to disable it, edit line 666 and set to false:

  Code: Select all

    let APPREGISTERED = true; // will only try to use copyStamp function if registered

• There's currently no option whether to restrict merging comments (ie by author, locked, status, etc).
• If two annotations have the same name on the same page number, it will assume they are the same annotation. This will only be a problem if someone manually sets the annotation name.
• The script doesn't currently handle form fields. It could, but I think it would need a 'diff' algorithm to deal with changes to text fields. That could be helpful in dealing with merge conflicts in text annotations also.
• The log is just pasted onto the end of the merge data file and only holds onto one merge. It could be set up to use as a way to revert changes after a merge.
• I'm still fiddling with this tool, so haven't set up a translation data file yet, although the script can handle one.

[KD952]

Wow!
Now we know where you have been for the last few months. Working hard.
Kind regards,
Daniel

[Daniel - PDF-XChange]

Hello, KD952

Indeed, this looks quite powerful, thank you once again for your hard work Mathew!

Kind regards,

[David.P]

Mathew must be kiDdiNg!

This is unbelievable

wow.png

[Mathew]

... i think it was a bit too soon to upload this: I've run into quite a few bugs I'll upload an update when I've got them squashed.

[Mathew]

OK, I think it's working in version 0.7. Probably some other bugs lurking in there so still needs some testing. I updated the file in the first post also.

mergeAnnots v0.7.zip

Changes in v0.7

• new property for state prior to merge,
• fix some state --> action mappings,
• fix saving updated modification dates to data file,
• move to Comments menu in classic UI,
• add file info to first dialog

[Dimitar - PDF-XChange]

Thanks, Mathew.

We appreciate your effort.

[Mathew]

A bug was preventing the tool from running. Fixed in this minor update.

mergeAnnots v0.7.1.zip

[Daniel - PDF-XChange]

[Mathew]

Over 700 downloads of v0.7.1!

I've been fiddling with this quite a bit lately: I think I've fixed some of the major merge issues but it's definitely got problems still. I've also added some new functionality "under to hood" so it can work better for another tool I'm working on, and added a new feature to store the mergeData files in the Temp folder instead of in files themselves. I think this is a better approach generally so that one is not adding miscellaneous data to pdfs, but it only works if merging always happens on the same computer.

image.png

This version won't work in PXCE before ~v10 (I've removed a lot of code and used a multi-column list view to the dialog which I don't think was supported before v10). I won't update the main post yet, so if anyone runs into bugs, please let me know so I can fix them. Thanks!

mergeAnnots v1.2.zip

Changes in v1.2

• fix showing newer/older when one date undefined;
• handle cases with deleting annotations (added "none" actionMap);
• handle updates when no dataFile and one annotation has newer modification date;

Changes in v1.1 (not posted)

• fix saving of mergeData to file;
• fix getting modDate of source document annotation for dialog;
• changed the saved mergeDate to be current date if modDate can be set, otherwise set it to match saved modDate.

Changes in v1.0 (not posted)

• use priveliged function for app.browseForDoc in dialog;
• remove bookmarks brought in with copied stamp;
• clarify license;
• misc code fixes;
• don't add info to mergeData if copying an annotation fails;
• fix closeDoc option so it doesn't just assume hidden documents need closing;
• add addMergeData option to save mergeFiles locally;
• automatically clear local mergeFiles older than set time;
• update preferences dialog and fix bug when it is cancelled;
• use MCLV for merge conflicts dialog;
• put source and destination side by side (similar to table) in conflicts dialog;
• list modification date and time (and add "newer"/"older");
• new forceSource option to force one-way sync of the files (overrides all conflicts and makes document forceSource the source document);
• revised switching documents and clarified conflict dialog;
• changed action for 'deleted' in both docs to 'nochange';
• rearranged conflicts table layout

[Mathew]

[edit: I've realized my choice of location for the locally saved mergeData was poor: Currently it gets deleted each time PXCE closes. Maybe it should be the Documents folder or 'resource'(?) Needs further thought.]

More bugs fixed in v1.3 attached. I also found that moving a popup annotation does not change the modification date of a markup, so this tool misses that change and doesn't merge it. I'll add a check in a future version.

mergeAnnots v1.3.zip

Changes in v1.3

• add .finish() method to AnnotMergeAction;
• clean up unused code and misc code changes;
• fix conflict at none->update;
• skip adding row to dialogActionList if merge options > 1;
• fix bug assigning modDate to sourceDoc mergeData;
• if there are no conflicts, assign 2 (-->go) to merge;
• add to add-ons toolbar for builds before PXCE 388

[Mathew]

The next iteration of this tool is actually two tools: One to do a single merge between documents, and a new tool that allows multiple people to mark up the same pdf document at the same time, which I've been tinkering with for quite a while now. This diagram is an overview of how I'm thinking of two ways to share and simultaneously edit pdfs:

image(7).png

A lot of collaboration functionality (sharing markups, centralized search index, maybe links or bookmarks depending on feature requests mentioned in another post) can be accomplished through javascript. I think it may be better as an extension, but I don't have that ability, so here's my first step. I'd really appreciate if I could get some help debugging/suggestions/usability/dialog layout, etc. I'm posting it here rather than on a separate topic because it should definitely be considered early Beta testing: Only use it on files you have backed up (there's a definite possibility of data loss!)

This will add two tools:

image.png

The new tool is "Live eXchange" and currently it only works on annotations (markup) and can do the following:

1. Synchronize a document's annotations among multiple concurrent users
2. Set up an "eXchange" index that links multiple pdfs

The second option allows another user to open the eXchange index and get links to all the shared documents in that eXchange.

markupExchange v0.3.zip

File Permissions
This tool will do a lot of file opening, closing and saving, so unless one allows scripts to access the folder that you will be sharing files from, and the local files, PDF XChange will show a lot of warning dialogs.

The setting for this is in Preferences > Security. It's best to add the folder(s) that will be used for the eXchange index(es) and central files as a File/Host Name Pattern in the form /path/to/eXchange/folder/*

image(8).png

That will eliminate most of the alerts. The rest of them are for a local data file in the Temp folder, and for local copies of the shared files, all of which it's easier to just check the box on the alert and accept them so PDF XChange adds them automatically.


Brief Instructions
When the tool is installed, it adds two buttons: If you click on "Live eXchange" it opens the following dialog:

image(1).png

At the top is a dropdown to select joined eXchanges, or manage them.
[singe file syncs] is for sharing individual files.
... any joined eXchanges are listed here ...
Manage allows you to add/remove/reorder your shared exchanges. At the bottom of the Manage dialog, it lists the path to the eXchange index file.
Select "New eXchange…"

image(2).png

This brings up a dialog to set up or open existing eXchanges:

image(3).png

Enter a name for this new eXchange, and select "Make New eXchange". Save it to a shared location (ie a folder on a server that other users can access). I've only tested it on an SMB file server with mapped drives. It theoretically would work on a share such as a dropbox desktop app (or google's desktop app, etc) but I've not tried it yet. If someone could test it that would be great.

After pressing "OK" it should return you to the main dialog, and list the newly made exchange.

Below the dropdown on the right there's a button that will open a settings dialog, but I've not got it set up yet.

Below the text "Central files" there are three buttons:
[↓] Load File loads the currently selected central file
[+] Add File adds another central file to the exchange
[-] Remove File removes it from the exchange (it doesn't actually delete the files from the eXchange folder - I don't think javascript can do that - so if you want them actually removed, you'll need to delete them manually from the folder.


Select "Add File" to open a dialog to start adding "Central" files (these are the actual pdf files that are being shared). Click "Browse" to select a file that will be shared.

image(4).png

After selecting the document, if it's not already in the same folder as the eXchange index, it will copy the file there with a randomized file name (to prevent filename conflicts on the eXchange folder). Then it will open in PDF XChange. There's an important option "Make an annotation only copy": If checked, it makes a copy of the central file that only has annotations, and this is what is used whenever a sync is run. The idea is to try to reduce the size of the file that is constantly being opened and merged with. It probably won't make much difference on small files. Both options need to be tested.

Next it will ask for where the local version of this file is to be stored. If you initially selected a file outside the eXchange index folder, it will suggest the file you selected. Otherwise press "New" to find a location to save your local copy (this is the copy that you edit):

image(5).png

It should return to the main dialog and list this file with a check mark next to it. The check mark indicates that the file is currently synchronized. Click "Close" to start editing the synchronized document. Close and save the pdf when you want to stop the sync.



Adding Users
Another user would share the same eXchange by going to "New Exchange" > Browse... to select the same exchange file (ie Exchange 1.pdf in the above example). You can also get the path to that file at the bottom of the Manage eXchange dialog. It will then load the list of shared files for that eXchange for that user.

The script currently logs a lot of information to the javascript console, to help with debugging. I'll reduce this in future.

To re-open an eXchange, run the tool again to bring up the dialog. At the top dropdown, select the exchange you want to view, and it loads the current pdf's for that exchange below. To load a document, select it, and press "Load". If you don't have a local file set up for that document yet, it will ask for a location to save it


Limitations/Known Bugs

• If pages are added/deleted/moved it breaks the sync! Currently, if pages need to be added or removed, the only way I can think to do it is to remove it from the eXchange, add/remove the pages, then re-share it. If someone happens to have it open at that time, they will need to manually copy them to the new share. This is a limitation of the original merge script I wrote. I've not come up with a good way to handle this yet because the script needs a way to know which pages map to each other. I'm thinking the best way would be to add hidden markups on each page of the document, but I really don't want to add miscellaneous stuff to pages.
• It only works on markups, and uses a rather kludgy workaround that requires a licensed PDF XChange to copy stamps.
• I've had some problems with deleted markups not syncing, and some markups not getting updated; I think I've got it figured out, but please comment if you see problems.
• If two markups are edited at the same time, one of the people will get a merge conflict dialog.
• Updates are currently set to
  • check every 2 seconds for updated markups. If there are changed markups, it syncs with the central file;
  • if the central document is locked by someone else, it waits 3 seconds before retrying;
  • every 20 seconds, even if there are no locally changed markups, it loads the central file to check for changes from other users.

[Daniel - PDF-XChange]