"none of these words" in advanced search doesn't always work

[m.rison]

I've just joined this forum, so I apologise if this information is somewhere and I've failed to find it.

I frequently want to search for something but exclude it if it's followed or preceded by something else. So for example I might want to search for "red car" but exclude all hits for "big red car" or "red car paint". Or maybe I want to search for "red" or "car" but not in the context of "big red" or "car paint".

This sometimes works if I put:

red car

in "this exact phrase" or "all of these words", for first query
or in "any of these words", for the second query

and:

big paint
or
"big red car" "red car paint" [for the first query only]
or
"big red" "car paint" [for the second query only]

in "none of these words" but sometimes doesn't. I've been unable to work out when it will work and when it won't work.

Is this a known issue? Or am I doing something wrong? Where do I find the most detailed documentation on the search functionality (that will tell me e.g. how the multiple boxes interact, and how to use multi-word terms, and how "proximity" interacts with this for the "all of these words" but not "this exact phrase" option)?

[Sean - PDF-XChange]

Hi m.rison,

The documentation for this feature is available here:

https://help.pdf-xchange.com/pdfxe10/search_ed_2.html

If that is not sufficient to answer your query, please get back to us.

Kind regards,

[m.rison]

Hello Sean,

Indeed it is not sufficient.

How do I search for "red car" but exclude all hits for "big red car" or "red car paint"?

If you have a standard test PDF file please tell me the URL and I'll try to construct a specific example based on that.

Thanks,

Mark

[Sean - PDF-XChange]

Hi Mark,

I don't believe it's possible to do what you're trying to do - if you are able to get that to work, please take a screeshot of the search terms/options used and add it here so we can take a look.

Kind regards,

[rakunavi]

Hello Mark,

The sample file contains 38 sentences that include the phrase “red car”.

• sample.pdf

Among these, there are the following three sentences that include the phrase “big red car”.

• 3. That big red car belongs to my neighbor, who loves vintage vehicles.
  8. I saw a big red car speeding down the highway.
  25. The kids were excited to ride in the big red car at the amusement park.

Additionally, there are the following three sentences that include the phrase “red car paint”.

• 2. I accidentally scratched the red car paint while opening the garage door.
  19. The shop specializes in custom red car paint finishes.
  21. He chose a glossy red car paint for his new sports car.

When searching as shown in the video, 32 results were found, excluding the six mentioned above.

• figure.png
  
  
  Video.webm

Best regards,
rakunavi

[Sean - PDF-XChange]

Excellent job - thanks rakunavi.

Kind regards,

[m.rison]

Thanks rakunavi, that's a very useful test file!

OK, then here's a failure (incorrectly matching "red car paint" in sentence 2):

image.png

[rakunavi]

Hello Mark, thank you for your feedback.

If you reduce the number of search keywords, changing the Proximity setting from the default “Adjacent Words Only” to “Words from the Same Paragraph” should work well. Also, with a structure like the sample file, enabling the “Detect tables by text layout” option in the “Page Text” category of the Preferences dialog might yield more expected results. The following video demonstrates verification using the state you described as the initial state.

• Video.gif

Best regards,
rakunavi

[m.rison]

Thank you for identifying this pair of configuration settings that work in this case, rakunavi.

However, it does not seem adequate to me. As a user:

1) I generally want to be doing adjacent-word searches, not just same-para searches

2) I don't want to have to be fiddling around with the somewhat awkward Options... drop-down. I even less want to be having to toggle some configuration option about table detection

3) I wouldn't have confidence that this configuration will always work in other cases

4) It presents as a bug: I asked to exclude "red car paint" but I still got a hit for "red" in "the red car paint while"

In fact, there seem to be more bugs under the surface. For example, if I just change the proximity to same-para, without fiddling with the table config option, it stops finding the third "red", which isn't in a "red car paint" context:

image.png

[Daniel - PDF-XChange]

Hello, m.rison

Just quickly chiming in here, I will admit that at this point, I am unsure whether the case is a lack of my own understanding for why there is a difference, or if there truly is a bug here, but I have passed this chain, the sample file, and search examples/results along to the Dev team for review. I also asked for their assistance in explaining the case here, if this is working as intended (which I feel increasingly sure is not the case).

I will let you all know what they say, once we have heard back from them on the topic.

Kind regards,

[m.rison]

Part of the issue is that the semantics of the advanced search are (as far as I know) not specified in full detail.

Let's assume for now proximity is "adjacent only". The rule is I think something like (based on experience) to match if:

{
all the "all of these words" are adjacent somewhere
AND/OR [not sure which of these it is, if both first two boxes are non-empty]
"this exact phrase" is found somewhere
}
AND
"any of these words" are present and adjacent to the words matched by the first two boxes [or possibly overlapping with those?]
AND NOT
any of the "none of these words" are present and adjacent to the words matched by the first three boxes [or possibly overlapping with these?]

where in all cases "words" can actually be "sequences of words delineated by double quotes" in which case they are treated as an exact phrase to appear contiguously in that order
and the amount of whitespace is not significant

But then it gets a bit more messy if proximity is "same paragraph", say. This could explain the problem I saw: if the tool decided the two lines were part of the same paragraph, then even though "red" was matched in one line, the match for "red car paint" in the other line would have caused the "red" to be rejected.

Perhaps as a starting point I could suggest that the spec should be something like to match if:

{
all the "all of these words" are proximate (including overlapping)
OR
"this exact phrase" is found, where any double quotes are invalid or ignored
}
AND
"any of these words" are proximate (including overlapping) to the words matched by the first two boxes;
if the first two boxes are empty then it's just a plain "any of these words"
AND NOT
any of the "none of these words" are proximate (including overlapping) to the words matched by the first three boxes
WHERE
"proximate" means whatever the proximity setting says is in range,
"overlapping" means a word is used to meet more than one row's conditions,
in all cases except the second box "words" can actually be "sequences of words delineated by double quotes" in which case they are treated as an exact phrase (i.e. to appear contiguously in that order),
the (non-zero) amount of whitespace is not significant

What do you think?

[David.P]

I frequently want to search for something but exclude it if it's followed or preceded by something else. So for example I might want to search for "red car" but exclude all hits for "big red car" or "red car paint".

image.png

[m.rison]

As discussed above, that does not always work.

For example, if I wanted to search for "car red" then what you have come up with would incorrectly match "1. The red car was the only clue left at the crime scene."

And if you try using the second field (this exact phrase) then it incorrectly matches "3. That big red car belongs to my neighbor, who loves vintage vehicles."

[David.P]

Right, the hits marked in red definitely seem to be a bug.

image.png

This is again with "Proximity" set to "Adjacent Words Only".

[m.rison]

Just quickly chiming in here, I will admit that at this point, I am unsure whether the case is a lack of my own understanding for why there is a difference, or if there truly is a bug here, but I have passed this chain, the sample file, and search examples/results along to the Dev team for review. I also asked for their assistance in explaining the case here, if this is working as intended (which I feel increasingly sure is not the case).

I will let you all know what they say, once we have heard back from them on the topic.

Is there any update?

[Daniel - PDF-XChange]

Hello, m.rison

I still have not heard back from the dev team, but I will be sure to post here as soon as I have.

Kind regards,

[m.rison]

OK, thanks. Please ask them to consider the spec suggested in viewtopic.php?p=197669#p197669 and in any case to be clear on the intended operation of the feature when multiple fields are used (in conjunction with the proximity setting etc.).

[Daniel - PDF-XChange]

Hello, m.rison

It has indeed been passed along, but all I can do now is wait for a reply.

Kind regards,