Weird URL escape behaviour when calling TrendSpec in a WebX client

[rlao]

I had this trip me up on CS2017R2:

1. Write a script to build and call an ad-hoc trend of an analog point using TrendSpec.
2. Read the units property of the analog and include it in the YAxis Label string.
3. Run this script on an analog point with units '%'.
4. Ad-hoc trends builds fine on ViewX.
5. Run this same script on the same analog point in a WebX client.
6. WebX throws up an XML error.
7. Change the units of the analog point to '%25'.
8. Run script on WebX again, it magically works.

My understanding is the TrendSpec string is encoding the '%' character to '%25' for the URL version of the SCX address. However, because the '%' character is in another nested string within the SCX string, it needs to be encoded twice ('%2525' instead of just '%25') which ClearSCADA is not doing. I seem to be able to hack this by replacing the '%' with '%25' somewhere in the script (or in the units field).

Is anyone aware of this issue being fixed in later versions of ClearSCADA?

[BevanWeiss]

I can't say I've encountered this myself, which is odd, since I have used scripted TrendSpec, but I try to avoid WebX, so that might the way that I've dodged it.

Interestingly, rfc3986 which specifies the percent-encoding of URIs specifically says not to double encode

2.4. When to Encode or Decode
...
Implementations must not percent-encode or decode the same string more than once, as decoding
an already decoded string might lead to misinterpreting a percent data octet as the beginning
of a percent-encoding, or vice versa in the case of percent-encoding an already percent-encoded
string.

I guess the way I would have interpreted rfc3986 is that everything within the reserved characters, we should be responsible for percent-encoding prior to supplying to ClearSCADA / GeoSCADA Expert.  But unreserved characters should be encoded by ClearSCADA (if it wants)...

In saying that, I think this entire rfc is quite borked...

Obviously the most common item which needs to be escaped in URLs is the SP (Space) character.  It appears that rfc3986 doesn't care about the space character at all, despite using it in its examples.  It's not in the definition for reserved characters.  and unfortunately the '%' character is similarly treated.  I'm surprised that this is such a foundational 'standard' for the web, and yet it appears so lacking.  Is there another reference that people actually use instead?

In the mean time, I would recommend that you percent-encode the rfc3986 restricted items (plus space and '%') prior to supplying them to the ClearSCADA navigate functions.  I think I'll add something similar to a script library so that I can just call it as required.

[rlao]

Interesting that you're not meant to encode/decode the same string more than once. Analyzing the final WebX URL I get from the script, I noticed the following.

When my script runs and builds the URL from the SCX path with the TrendSpec string, it performs only one single instance of percent-encoding: it encodes all restricted characters (including Space -> %20) within all the child string expressions embedded within the parent TrendSpec expression, e.g. Axis labels, Formats, Expr strings.

Interestingly enough, the encode ignores all restricted characters outside of those embedded strings. So all the apostrophe and brackets in the TrendSpec expression do not get encoded and remain as-is in the URL.

So to me that means the SCX-URL conversion part would obey that single-encode rfc rule. I assume that this whole process outlined above is done by the WebX server?

However, double-up occurs when WebX/IE navigates to that destination URL. It looks like WebX client (?) is seeing the TrendSpec string and it's percent-decoding all embedded strings within the TrendSpec expression to match the corresponding encode that was done in conversion of the SCX address to URL. However when that decoded URL hits IE, it tries to decode it again. This is where it sees the already decoded '%' character and tries to decode that, at which point it fails and throws up an error.

I suppose the first decode maybe should not be occurring and WebX should just pass the encoded string as-is to IE for it to decode?

[tfranklin]

I'm fairly confident that we've run into this with our TrendSpec stuff over WebX, but I can't seem to find what release it was fixed in.  One of the monthly patches in 2017R2/3 seemed to fix it, but it was quite a while ago.  The patch notes didn't address the % specifically but they handled special characters in the URL better.  Should also be noted, when we ran into the issue it was only with the newer IIS version of WebX.  The legacy WebX was not affected by the issue.  I think one of the 2 notes below fixed it, but of the two I'm feeling like it was October 18. 

2017R2/3 April 19:
[SUP-10791] Fixed problem with Trend Spec with Expression with single quote when saved as favorite.

2017R2/3 October 18:

[SUP-10408] Fixed problem with handling of SOURCE token in Trend Spec in (new)WebX.

Consequently -- there's this from 2017R3 October 2019, but it doesn't really address much of what the issue actually was. 

[SUP-10958] Fixed problem when opening a trendspec from a mimic script.

What version are you on? 

[JChamberlain · Accepted solution]

I had a quick look and it doesn't look like we specifically fixed this problem at the time, but it might have been generally fixed...

@rlao could you please raise this with support if you haven't already?

[rlao]

Hi Jesse,

No worries, I'll shoot a formal email to support.

We are already running CS2017R2 Oct 2018 patch (79.6884) so it is probably not covered by SUP-10408.