IStream opening issues when using in 410 (vs 396) SOLVED

[zarkogajic]

Hi Support,

I've updated my SDK to the latest version 410. I was using 396 before that.

There are now some strange issues when opening documents via IStream approach (IPXV_MainFrame.OpenDocFrom(my_file_stream,...).

Simply opening some document ends with (note: document has content):

Screenshot_410-issues.png

Screenshot_410-issues-info.png

Then, opening the Bookmarks pane - and the file gets silently closed.

Note: opening the same document (not using IStream approach, but op.openDoc) - opens the document without any problem (no problems in file found) and the Bookmark pane can get displayed.

For some other file that info line does not get displayed but document just gets closed (as if was not open).

For some other my app crashes (not responding actually).

For some other - opens without issues.

Have any idea what could have been changed?

p.s.
I have reverted back to 396 version just to double check this is not something in my code. And no, 396 works as expected (and as it did many versions before)

-žarko

[zarkogajic]

Hi Support,

Some new info.

For that one file which displays "broken info" and then auto-closes when Bookmark pane is made visible...

In my code I'm prefixing file path with "\\?\" when opening a file using TFileStream (so using Delphi), this TFileStream is then used by TStreamAdapter (wrapper of IStream)

Once I removed "\\?\", the file gets open and diaslayed in viewer, the "broken info" does NOT get displayed on top - but opening the Bookmark pane - still auto-closes the document...



-žarko

[Daniel - PDF-XChange]

Hello, zarkogajic

I cannot say what has happened at this point, but it would be helpful to see a sample project where the issue occurs, and a copy of the any offending document.
Could we ask for those, in order to investigate?

Kind regards,

[zarkogajic]

Hi Daniel,

In my app where this happens - I do a lot of heavy stuff with pdfx - so I'm afraid a small sample project will not have issues ...

I will though try to create a new small app - and if the issue happens - will send it (but cannot do that before end next week) - but it will be in Delphi and not of lot help for you.

In the mean time: would it be possible to download past SDK versions - as I would like to pinpoint the exact version this happened in between 396 and 410. Maybe your developers will then more easily figure out what was changed.
On my "your products.." page I can only download the latest 410 sdk version.

p.s.
I do not believe this has to do with a single document - multiple documents have the same or even stranger issues (pages view does half display and is empty and my app hangs without any idea where it goes in circles).

I've tested more: if using IStream to open from array of bytes (in memory) or similar - all seems ok.
It seems only when using IStream to open the file from disk - strange problems happen.



-žarko

[Daniel - PDF-XChange]

Hello, zarkogajic

Thank you for the details, the samples will still be helpful, when they are possible to send over. For now though I have nudged the Dev team for some assistance here incase there is anything they can offer.

As for older versions of the SDK, if you still have valid maintenance on your current SDK license, you can email [email protected] to get in contact with our director on the topic.

Kind regards,

[zarkogajic]

Hi Daniel,

Thanks. Will do so...

-ž

[Daniel - PDF-XChange]

[zarkogajic]

Hi Support,

I've been going through past versions .. and have found that the issue first appears in version 405.

That is: in version 10.7.6.404 - my IStream loading is working as expected, while in version 405 (and in all versions till 410) - something is off.

My I please, please ask developers to check what was changed related to loading / working with file streams in between version 404 and 405...

p.s.
To repeat what I noticed: trying to open (not visible by default) the bookmark pane for a particular document - and the document gets auto closed.

Not just trying to open the bookmark pane, but a simple click on the "View" menu does the same (document gets closed):

image.png

I get no exception in my code - it just vanishes.

Also, sometimes the document does not fully load and my app gets stuck (no idea what's "spinning").

-žarko

[zarkogajic]

Hi

Here's another observation, difference:

In 404, for a file I *can* open via IStream, I can execute the Save As command and the file would save ok.

In 405, the "End of Data reached" error upon Save As is reported (for the same file). If I do "Retry" - the "Invalid object structure" gets reported (and save ofc fails) - and if I select to Close the Save As dialog - the file gets auto-closed from viewer...

p.s.
This, what's even more crazy, means that some files in 405 open without issues and some not.

-žarko

[zarkogajic]

Hi,

Ok, so I created a super simple Delphi app to demo this problem.

Coding as simple as it can get:

Code: Select all

var
  s : string;
  fs : TFileStream;
  stream : TStreamAdapter; //implements IStream
begin
  s := 'E:\TEMP\pdfx\sometimes-ok.pdf';

  fs := TFileStream.Create(s, fmOpenRead OR fmShareDenyNone);
  stream := TStreamAdapter.Create(fs, soOwned);

  PXV_Control1.OpenDocFrom(stream, nil);
end;

One button to just open in PXV_Control:

image.png

I'll send the documents and the project to support email...

-žarko

[Daniel - PDF-XChange]

Hello, zarkogajic

Thank you again for the extended details, I have added this information along with your email to the ticket we have on the topic, and informed the Dev team directly there is new information for their investigation.

Kind regards,

[zarkogajic]

Hi Support,

I’ve been investigating this a bit more.

[EDITed .... since more info gained ...]

Since this all looks like some kind of threading issues (file opens then closes after a moment, file partially loads, file never loads / gets stuck, etc) I went to see from what thread the IStream.Seek methods gets called.

Using Delphi’s TStreamAdapter, the IStream.Seek:

* in version 404: first gets called multiple times in background thread(s), then a few times in the main thread, then a few more times in background thread(s) and the file fully opens with no issues.
* in version 405: all calls happen in the main thread - at least up until things get berserk.

I’ve turned to C# to see how things work there: I’ve used your IStream implementation and did a simple C# project to test (matching Delphi’s TFileStream and IStream project I’ve sent). All works as expected both in version 404 and 405. But, in 405 all calls to Read work as in 404 (so background thread + main thread + background thread) – not the same as I see with 405 in Delphi.

Then I’ve went to see what’s different between Delphi vs .Net and: there’s a difference: .NET's COM interop wraps managed objects with free-threaded marshalling implicitly while Delphi’s TStreamAdapter by default does not implement IMarshal.

This suggests you have two paths of execution in 405: if IMarshal start or do IStream.Read from the background thread (.Net way), if no IMarshal do from the main thread (Delphi way).

Further, this also suggest that in 405 you, at some moment, force IStream processing in a worker thread even if the IStream object does not implement IMarshal. Or, maybe not force in the background thread, but some kind of deadlock, wrong assumption, missing pump or alike happens. So, the no-IMarshal path of execution somehow got broke in 405.

Now, to try to prove the above, I’ve created an inherited TStreamAdapter which implements IMarshal (using CoCreateFreeThreadedMarshaler).

And yes, all started to work as it should and the IStream.Read again being first executed in the background thread. And the file loads ok.

However, the funny thing is: if I return E_NOTIMPL from GetUnmarshalClass – then no other IMarshal methods are being called, but still all works as it should. Which is fine I guess as E_NOTIMPL suggest “I do not need special marshaling, call me direct/free-threaded, no marshalling required”.

So, to conclude: can you please confirm the above makes sense and is actually what is happening?

If so, will you be fixing that issue when IMarshal is not implemented by the IStream object?

Or, should I be using the TStreamAdapter which implements IMarshal from now and. And if so, returning E_NOTIMPL from GetUnmarshalClass would be enough?

And also this: do you expect the IStream Read/Seek, etc to be thread safe? (as by default: not in Delphi nor in .Net)

-žarko

[Daniel - PDF-XChange]

Hello, zarkogajic

Excellent work on this, and thank you again. I have forwarded these details and your new questions to the Dev team for review and response as soon as they are able. We will get back to you as soon as possible.

Kind regards,

[Vasyl - PDF-XChange]

Hi Zarko.

In the upcoming build, we will fix that issue.
The issue was caused by almost unconditional marshalling of the input IStream-object to another thread where the document is opening.
In an ideal situation, such marshalling shouldn't cause any problem, but it seems that some implementations of IStream don't support it properly.

'almost unconditional' - means that marshalling is used when openDocParams["Sync"] == false, which is false by default. And the problem is that Sync==true - it is unusable in most cases because it may lock the caller thread (the main thread typically) for a while...

In the new build, we added the openDocParams["UseMarshalingForIStream"], which is false by default.

Cheers.

[zarkogajic]

Hi Vasyl,

Thanks for the info.

Can you provide a case when one would use the new openDocParam UseMarshalingForIStream set to True ?

That is: what are the expected differences / benefits?

-žarko

[Vasyl - PDF-XChange]

For example, the UseMarshalingForIStream=true is used internally when the user DnD attachments from Outlook to Editor directly. There are 'external' streams that must be marshalled to the non-main thread for use there...

[zarkogajic]

Thanks, all clear.

-žarko

[Daniel - PDF-XChange]

[Vasyl - PDF-XChange]

By the way: the new v11 of Editor SDK requires a special Common folder, next to the PXCEditorCore.<platform>.dll file. Like this:

Code: Select all

Common
	.Current
		OCV
			<platform>
				ImageProcess.dll // required for EnhanceScans, IdentifyForms, etc
	ICU
		icudt74l.dat // critical data-file, https://icu.unicode.org/
	Languages
		*.xcl // all UI-localizations
	RedactPatterns //optional, required for Find&Redact feature
	Tesseract // optional, 'Default OCR' engine, also used by IdentifyForms            
	
Plugins.<platform> // optional
....
Resources.dat // critical data-file
PXCEditorCore.<platform>.dll

[Sunilkumar]

Thanks Vasyl, that explains the behavior perfectly.

For anyone else debugging similar IStream-related issues after 405+, the key takeaway seems to be:

prior versions effectively behaved as if Delphi-style apartment-threaded IStream implementations were tolerated implicitly,
while newer builds introduced background-thread usage that exposed missing/free-threaded marshalling assumptions,
especially with wrappers like Delphi’s TStreamAdapter that do not implement IMarshal by default.

After testing, implementing IMarshal (or disabling marshaling through the new parameter) resolves the issue consistently on my side.

Also appreciate the clarification regarding Outlook attachment streams and other externally-provided COM stream sources — that use case makes the new UseMarshalingForIStream parameter much clearer.

[Daniel - PDF-XChange]