[javascript] bugs? Problems dealing with UTF-16 strings

Forum for the PDF-XChange Editor - Free and Licensed Versions

Moderators: PDF-XChange Support, Daniel - PDF-XChange, Chris - PDF-XChange, Sean - PDF-XChange, Paul - PDF-XChange, Vasyl - PDF-XChange, Ivan - Tracker Software, Stefan - PDF-XChange

User avatar
Mathew
User
Posts: 947
Joined: Thu Jun 19, 2014 7:30 pm

[javascript] bugs? Problems dealing with UTF-16 strings

Post by Mathew »

I'm having trouble encoding UTF-16 strings using both doc.createDataObject() and util.streamFromString(). In the example below, I'm encoding an arbitrary UTF-16 string with JSON.stringify() first, then comparing this string with what comes from the above two functions. I think that util.stringFromStream() is including the 'FEFF' byte order mark as the first character (bug).

I get the following:
doc.getDataObjectContents() I get an additional first character char code 'FEFF', but only if the text included requires two unicode units.
util.streamFromString() seems to fail entirely on characters with two unicode code units.

Am I doing something wrong?

Code: Select all

// test string from stream

function testStringStreamUTF16(str) {
  const cName = 'tempTest.txt';
  const tempDoc = app.newDoc();
  tempDoc.createDataObject({
    cName,
    cValue: JSON.stringify(str,null,1),
    cMIMEType: 'utf-16'
  });
  function outDiffs(q,p) {
    try {
      console.println([q.length, p.length]);
      for (let i in q) {
        if (q.charCodeAt(i) !== p.charCodeAt(i)) console.println(i + ": " + [q.charAt(i),p.charAt(i)] + " : " + [q.charCodeAt(i).toString(16), p.charCodeAt(i).toString(16)]);
      }
    } catch (e) {
      console.println(e);
    }
  }
  
  const q = JSON.stringify(str,null,1);
  let p = util.stringFromStream(tempDoc.getDataObjectContents(cName), 'utf-16');
  console.println("data object");
  outDiffs(q,p);
  
  p = util.stringFromStream(util.streamFromString(q, 'utf-16'), 'utf-16');
  console.println("stream from string");
  outDiffs(q,p);
}
  
testStringStreamUTF16([`ヸ摭㝑ů᪌ڬม疯ⅲॼᲥㆬ₌兼ৡ⏵c拀㒧刾⁊炨ᚃ屚Ñዘ㚧㨯晢຋Ⳃᠽઔڹ㒥޲戔က瀣‸∂勁⿨稫䂥ٌ̦ቝ偸懠㑧瀪悇ᅀۙ⍣愇䗆㘲扡ࢩ瀥⥦疼况ငⷲ瑆懑剒ᮥ⌣䛆े≦䠭備怠࣎㢌Ạᐣ倨儞懲ᅧ簱䤁ੀኯ䲠჌䗁翆䑆ؚհ癴庨凌ᜄ䕶摻慤䠱╗⁐捆Ɉ⯧≅拐ϐ䊢ᐨ凚ສ憤ʼ姴䌂喓㑚ĩ䎠䜨ṟᅞᲀ䢠崴䌮ᐮ澵⯙甦።ଂ坒ŵ愎ᬫ‬䐈䉈疢琭戆 㑨ˊ䋄桀\"宁߃挈䣏ਯᄰ䬀`]);
Last edited by Mathew on Mon Nov 16, 2026 6:18 pm, edited 1 time in total.
User avatar
Daniel - PDF-XChange
Site Admin
Posts: 13314
Joined: Wed Jan 03, 2018 6:52 pm

Re: [javascript] bugs? Problems dealing with UTF-16 strings

Post by Daniel - PDF-XChange »

Hello, Mathew

Thank you for the report, I will pass this up to the Dev team for review and see what they have to say on it.

Kind regards,
Dan McIntyre - Support Technician
PDF-XChange Co. LTD

+++++++++++++++++++++++++++++++++++
Our Web site domain and email address has changed as of 26/10/2023.
https://www.pdf-xchange.com
[email protected]