What is the smallest black and white image format?

[kio]

Good morning:

As I mentioned in a previous post,
I converted an 11,000-page PDF file from the internet into 11,000 images, then merged them using PDF-XChange.
So far, I've found that the format that takes up the least amount of space and best preserves black-and-white quality
is JBIG2, with the "dither" option set to "no" (to save even more space).

But now I'm wondering:
Is this the best black-and-white format
(that takes up the least amount of space and best preserves image quality),
or is there something else I could do to make the file even smaller?

(I don't want PDF optimization because I've seen every way it reduces image quality.)



PS:
I tried using this online "jbig2 to PDF" converter, "reaconverter,"

https://online.reaconverter.com/convert/jbig2-to-pdf

because I noticed it saves a lot of space (250kb),

13 pages with Reaconverter.pdf

with the same quality as XChange (990kb), with the same file:

13 pages with XChange.pdf

I'm wondering:
Is the jbig2 to PDF conversion that "reaconverter" will do as SAFE as the conversion with XChange?
I don't want to create a jbig2 file and then convert it to PDF using "reaconverter"
if "reaconverter" has errors in its conversion program,
and sooner or later I'll get the error "XREF chunks not found...etc"
(which I've seen before with PDFs downloaded from the internet)
due to the file being corrupted
by a conversion performed by an outdated or incorrectly programmed PDF program. ("reaconveter")


Thanks.

[David.P]

Hi kio,

1. Is JBIG2 the Smallest Black and White Format?
Yes, definitively.
JBIG2 is the undisputed champion for compressing monochrome (1-bit black and white) documents. It uses advanced pattern matching to recognize and reuse recurring shapes (like letters and symbols) throughout the document. This results in incredibly high compression ratios, creating unbelievably tiny file sizes. You should absolutely continue to use JBIG2 as it is the strongest compression option for B&W PDFs.

2. Clarification on Downloads and Compression
It appears the files you provided do not match your description. One file contains only 1 page (not 13), and both seem to utilize ZIP compression instead of the expected JBIG2 encoding.

[kio]

1.
There was an error:
I had uploaded the 13-page jbig2 file to the "REA" website,
and I clicked "convert to DPF," thinking the REA website would give me a 13-page PDF file (250kb).
Only now, however, I've noticed that "rea" converted ONLY the first page of that 13-page jbig2 file:
for this reason the file was 250kb.

2.
I can't upload the jbig2 file directly to this forum, because it's not supported by the forum,
so I had to upload the two jbig2 files, converted directly to PDF with XCHANGE and REA.

3.
So, the best solution to convert my 11,000 PDF pages to images remains this one: jbig2.
The PDF file resulting from this large conversion is handled fairly quickly by XCHANGE,
however, I occasionally see small stutters, and it takes 10 seconds to save the file each time:
=
I've tried changing the "performance" settings in "preferences" several times,
but I find that no matter what I do in performance,
the result is always the same: it doesn't improve speed.

4.
Final question:
I'm trying to convert this 11,000-page PDF file to jbig2
using a single 11,000-page jbig2 file, not 11,000 separate jbig2 files.
I wanted to see if the single file would be smaller and take up fewer MB than the 11,000 separate jbig2 files.

But even if it were smaller, I wonder:
Is it safer (since I'm CERTAIN that the 11,000-page PDF file is corrupt)
to separate the PDF file into 11,000 individual jbig2 "chunks"
(compared to a single jbig2 file) to PREVENT the "XREF chunks were not found..." error?

I repeat, I know that this "XREF..." error is present inside this PDF
(I know because it has already happened to me several times,
because it is a file that Google has assembled/merged, via its "Google PDF")

5.
PS:
I've been trying several times with "PDF Tools" to convert PDF to a single jbig2 file,
but every time, after a few minutes, "PDF Tools" gives me the error: "Insufficient disk space."
And yet, I don't understand why "PDF Tools" converted my PDF into 11,000 individual and separate jbig2 files,
and it didn't cause any problems.

I will do one last test, but no longer via "PDF Tools",
but directly by opening the PDF file with XCHANGE,
and then from within the program itself in the "convert to images" section

Anyway, thanks as always for your reply.

[David.P]

Hi kio, not sure what happened during your conversion, but maybe you want to do the conversion in two steps:

First, convert it to (black and white and) zip compression to stabilize the file, and then in a second conversion step convert it to JBIG.

If you convert every page separately and then put the pages together, you will probably end up with a much larger file because then every page will have its own JBIG symbol library instead of one library for the entire file.

(PXE team, please correct if this should not be the case, i.e. if JBIG compression doesn't create file-based but only page-based libraries).

[kio]

thanks, I didn't know this:
but, briefly, in which "setting" do I have to go,
to convert it FIRST to black and white zip, and THEN to jbig2?

and then above all, my main problem is that by doing so
(i.e. with this new conversion via ZIP/JBIG2),
I would like to be sure to solve the "XREF..." problem,
which the PDF causes due to the corrupt and ruined assembly of Google PDF.

[David.P]

I'd use the "recompress images" dialog which is easily found through the search box in the title bar or toolbar of PDF-XChange Editor, if you don't see it in the GUI right away.

[kio]

1.
I tried all the options with "recompress image":
but nothing, it doesn't change much.

2.
I tried "create an optimized copy", but nothing:
the file is more or less the same size as the JBIG2.

3.
I tried converting to JBIG2 directly into a SINGLE jbig2 file,
but this time by opening the PDF file directly with the "XChange" program itself, and not with "PDF Tools",
and this is the error I get (the same error I got with "PDF Tools"):

error.png

[David.P]

You can right click on a page and choose "recompress image". Then, you can see the current compression type and size and play with different color and compression formats.

[kio]

OK, thanks. I wasn't aware of this great option.
However, after trying everything using the "recompress image" option,
I couldn't find any improvement.
Perhaps because I really believe I can't get any better than this.

I also tried "rasterization," but the result is the same.
In fact, the various options—"rasterization," "recompress image," and "export images"—
seem the same to me, at least in some respects.

My only remaining doubt is whether, by making a SINGLE JBIG2 file,
I can achieve a better result in terms of size.

[David.P]

Yes, I realized that for some reason, with your file it doesn't seek to make much difference whether you compress using ZIP, G4 or JBIG.

JBIG files COULD become much smaller, per page, with increasing page count - but that depends on whether JBIG builds libraries per page or per file.

Possibly, JBIG is even implemented in PXE in a way that doesn't use any symbol libraries. I believe that JBIG symbol libraries have fallen somewhat into disrepute since the findings of David Kriesel.

[Daniel - PDF-XChange]

Hello, David.P

Here is the reply from our Dev team:

While the problem David Kriesel found with JBIG2 is related to JBIG2's symbol compression, it only affects lossy compression.
JBIG2 compression implementation in the Editor does not use lossy compression. We support symbol compression in our JBIG2 codec, but I'm afraid it is not activated in the Editor.

In practice, inside of a PDF, the "best" image format to use will vary significantly based on your image content and compositions. I cannot recommend any single option for anyone, as it will change not only "user-to-user" but even "file-to-file"...

Trial and error, as well as some self guided exploration/education about the various image formats and their respective strengths and weaknesses will help to make the correct decisions for each of the countless contexts you could need image compression functions.

Kind regards,

[David.P]

We support symbol compression in our JBIG2 codec, but I'm afraid it is not activated in the Editor.

Thank you Daniel, for this information from the Devs. This is interesting, and explains why PDF-XChange Editor does not produce those extremely small files that can be obtained when JBIG symbol compression is used.

Anyway, JBIG still creates the smallest files in PDF-XChange Editor, and it's lossless, so I would still recommend using JBIG to kio.

[Daniel - PDF-XChange]

[kio]

error.png

[kio]

1. ok thanks.


2. and what about this error, posted a few pages above?

error.png

[Daniel - PDF-XChange]

Hello, kio

Sorry, I missed that. "Insufficient Space on Disk" would indicate that the file cannot complete the conversion because there is not enough storage space available to hold it.

EG, if the generated file is 0.5GB in size, your hard drive has less than 0.5 GB free, so it cannot be stored there. (during conversions, the file is generated in a temporary location - so this can happen on opening from other formats, not just in saving the file.)

Kind regards,

[kio]

ok thanks,
but I don't know if I understood correctly,
because in fact my "C" drive of the hard disk (where I installed "Windows" and all the programs, including "XChange")
has 150GB available,
while my "D" drive of the hard disk (the one where I told "XChange" to insert my unique JBIG2 file, after having created it)
has 600GB available

[Daniel - PDF-XChange]

Hello, kio

There is a chance that it is an item which was lost in translation, and the error may be simply saying "memory" in English (we have volunteer translators, so this does sometimes happen). It may be worth changing your application UI language to English to see the original error message.

If it does specifically say "not enough memory" - then the issue actually relates to the RAM use by the process (not the disk), which... converting 11,000 image based pages to PDF would most certainly be able to use a very large amount of memory, but it should not be quite enough to cause problems on most modern systems..
In that scenario, one thing you can try is increasing the memory limits of the Editor, via the Performance preferences:

image.png

By default, we use around 25%, and politely ask the system if we can use more when possible, but if you manually change this setting to 75%, the editor can be a bit more aggressive with its memory demands, which might help in this scenario.

This is only a workaround of course, but I do hope it helps while we are waiting to hear back from the Dev team.

Of note - we have created a ticket to investigate this issue further:
RT#7725: Investigation: Insufficient Disk space on jbig2 import
Unfortunately it is a low priority ticket at this time, due to the unusual use case, and low frequency with which it would be a problem... (I do apologize)

In the meantime, you may be able to get around the issue by using the "save each Page Range to a multi-page image file".

image(1).png

This would allow you to insert the pages into the output file in batches. It would be a tad inconvenient of a workaround, but better that simply not working at all.

Kind regards,

[kio]

1.
I've discovered: the problem isn't RAM,
but the "C:/User/[name]/Appdata/Local/Temp/PDFXtools" folder, which was full.
That is, when I convert the 11,000-page Google file into a SINGLE "JBIG2" file,
I get tons of temporary files every minute (most of them 50MB each),
which quickly fill up my entire "C:" drive.

2.
Another question:
I converted this 11,000-page Google PDF by setting 11,000 individual JBIG2 images.
I then merged the 11,000 JBIG2 images using "right-click/merge in PDF XChange."
But the resulting PDF file is NOT "JBIG2", but "CIT 4/ZIP":
In fact, if I go to any PDF page and press "right click/recompress images,"
I find it says NOT "JBIG2" but "CIT 4/ZIP":

3.
So, I do a second pass using the "recompress image" option,
where I tried all the settings, namely:
- "monochrome ZIP"
- "monochrome CIT 3"
- "monochrome CIT 4"
- "monochrome CIT 3/ZIP"
- "monochrome CIT 4/ZIP"
- "monochrome JBIG2":

Now, (aside from the MB difference),
all the resulting PDFs had more or less the same speed,
- both in page scrolling (using the side scroll bar)
- and in saving time.

4.
Finally: if, after obtaining the "CIT 4 / ZIP" (see "number 2" above),
I go to "recompress image" and set it to "monochrome JBIG2" (see "number 3" above)
the resulting file actually shrinks by several MB (i.e., from 850 MB to 600 MB)
but, STRANGELY, the file becomes slower than the "CIT 4 / ZIP,"
for example, when scrolling with the side scroll bar!

PS:

5.
For this 11,000-page Google PDF file, I tried both:
- "optimize PDF" after creating it (and again, STRANGELY, it seems to run slower after optimizing it!)
- and "print" via "PDF-XChange Standard", but again, the MB and PDF speed are still the same as in "number 3" above.

6.
I tried converting the 11,000-page Google file, at 2,000-page intervals each,
for a total of 6 JBIG2 files ["1-2000, 2001-4000, 4001-6000, 6001-8000, 8001-10000, 10001-11,000"]
and it works: that is, it doesn't give me the "out of disk space" error;
and I also noticed that the resulting PDF saves faster,
than the same PDF I previously obtained after merging 11,000 individual JBIG2 files.

7.
Regarding "RT#7725" which has a low priority, there is no problem whatsoever, rather, thank you very much in advance:
just by the way you respond so quickly to all your users' problems, you deserve to be the first in the world, as PDF producers!

[David.P]

Finally: if, after obtaining the "CIT 4 / ZIP" (see "number 2" above),
I go to "recompress image" and set it to "monochrome JBIG2" (see "number 3" above)
the resulting file actually shrinks by several MB (i.e., from 850 MB to 600 MB)
but, STRANGELY, the file becomes slower than the "CIT 4 / ZIP,"
for example, when scrolling with the side scroll bar!

Regarding the speed of scrolling/browsing through a PDF file, I can confirm that JBIG usually is considerably slower than other compression formats.

[Daniel - PDF-XChange]

Hello,

I have added these notes to the ticket to help with the investigation!
Regarding the conversion during creation, I am not entirely certain but I expect there is a reason we do so. I will try to find out more and let you know if I get a clear answer.

Beyond that, David is correct regarding the speed of browsing.

Kind regards,

[kio]

I'm updating to provide you with information that might help you better understand the problem:

1.
I converted the 11.375-page PDF into a SINGLE JBIG2 file
using another PC with 1.000 GB (1 TB) of free space.

The conversion process into a SINGLE JBIG2 file (600 DPI, NO Dither)
created 11.375 temporary files (in the "Temporary Files" folder), each 57.339KB (57 MB) in size,
for a total of 650 GB.

2.
Immediately afterward, I tried to "search" (with the "magnifying glass" = CTRL+F) on any word,
comparing the search speed between the two documents:
A. between the 11.375-page PDF converted into 11.375 JBIG2 files,
B. and the 11,375-page PDF converted into a SINGLE JBIG2 file:
=
The search time for the same word in the SINGLE JBIG2 file was almost half (e.g., 10 seconds)
compared to the file with 11,375 individual JBIG2 files (20 seconds).
(Although I must say that the speed of scrolling between the 2 files [e.g. with the sidebar],
was the same).



3.
I tried converting another 435-page PDF (see photo) into a SINGLE JBIG2 file.

687MB.gif

To my surprise, this time, on page 236 of 435,
my old PC (the one with 160 GB of free space) gave me the "not enough disk space" error.
This error occurred because, in the temporary files folder,
each temporary file (corresponding to each individual PDF page) occupied 687.574KB (687MB !)
= 687.574KB x 236 pages = 162.267.464KB (162 GB).

4.
I attribute this last result to the fact that:
A. the PDF pages were very large (893mm x 1413mm: see blue arrows),
compared to the 11,375 page PDF, of which each image was 123mm x 213mm;
B. that 11,375-page PDF, was not JPEG but JBIG2-made by Google Books, while this PDF is all JPEG.
C. or who knows what other reasons.

[David.P]

Interesting. Can you elaborate a little what you mean with "a single JBIG2 file"? I suppose you are talking about a PDF file with JBIG2 compression? However, if so, then I don't understand why you would have the temporary files.

1.
I converted the 11.375-page PDF into a SINGLE JBIG2 file
using another PC with 1.000 GB (1 TB) of free space.

The conversion process into a SINGLE JBIG2 file (600 DPI, NO Dither)
created 11.375 temporary files (in the "Temporary Files" folder), each 57.339KB (57 MB) in size,
for a total of 650 GB.

[kio]

000014.gif

1.
I converted this PDF, created by Google Books, to a PDF created by XChange,
because it's more secure.
In my experience, in fact, I've noticed that Google Books sometimes doesn't convert its PDFs properly,
and leaves "errors" inside its PDFs.
These errors sometimes result in the error message: "one or more XREF chunks not found,"
which literally destroys my PDF.

2.
Yes, with the option I highlighted in the photo, these thousands of temporary files are created on my PC. However, I don't know, if, for example, these same files would be created
if I simply decided to convert the Google PDF into 11,375 JBIGS files, and not into a single JBIG file.

[David.P]

Okay, so you exported everything to a MASSIVE multi-page image with JBIG2 compression. However, I would not advise to do that because this process creates new images, instead of simply using the existing images and only changing their compression to something else.

With "Export to Images", even if you accidentally get the DPI in both directions correct, there still will be some resampling involved.

I'd rather export the existing page images without resampling and conversion, for example using the PXE PDF Tools, and then work with these images.

Alternativly, simply use the existing PDF, run Recompress Images on it, for example to ZIP Black and White, and then work with that file and convert it to JBIG2, again using Recompress Images.

[kio]

I updated my post (see below)

[Sean - PDF-XChange]

Hi Kio - okay, thanks - we can continue the discussion once you have.

Kind regards,

[kio]

1.
Yes: in fact, the JBIG2 file that was output, was a huge 550MB image, which I then had to:
- reopen as a PDF (e.g., it outputs in CIT4/ZIP and NOT in JBIG2)
- resize
- and use OCR.

2.
But I did this, both because I didn't know about the "recompress image" option,
and, above all, because I don't want to use "recompress image"
because I'm not sure it will PERMANENTLY remove the "XREF" errors
that Google PDF contains within its pages:
and therefore, sooner or later, the PDF might be destroyed
(as has happened to me on other occasions: see this link for more information:
viewtopic.php?t=48502).

So, instead, I believe that using JBIG2 on a SINGLE image, will remove any "XREF" errors.
In fact, I've been using the new PDF created with "export images (to JBIG2)" for over a month,
and the "one or more XREF chunks not found" error hasn't reappeared:
could this be the same thing as using "recompress image"? I don't know,
I don't want to risk having a PDF that's a "ticking time bomb",
which could "explode" with an "...XREF" error at any moment.

3.
Anyway, I did the test a little while ago, also with "recompress image", and I noticed that,
if, for example, I switch from Google's JBIG2, to CIT 4/zip, from this window that I put here in the photo,

000016.gif

it says: "No changes made."

While if I switch from Google's JIBG2 to the CIT4/zip, of this other photo,

000017.gif

this time the PDF is modified: is this a bug?

4.
What is PXE? I tried to find it in the website's guide, but it doesn't say anything.

5.
And then, for example, regarding the 11,375-page PDF, if I were to use "PXE" (or "Recompress"),
is there any difference in terms of MB saved or final speed?