Invite a Co-worker
Send a co-worker an invite to the Exchange portal.Just enter their email address and we’ll connect them to register. After joining, they will belong to the same company.
Send Invite Cancel
79543members
344375posts

Arithmetic modul calculates wrong

Sven_Kermer
Lieutenant JG Lieutenant JG
Lieutenant JG
0 Likes
1
168

Arithmetic modul calculates wrong

Hello,

I'm using an ION7550RTU to capture kWh-pulses. I use an arithmetic module for multiplying the pulse counter with the pulse weight. In my example, the pulse counter has the value 758093 and the pulse weight is 1.2

The arithmetic module multiplies both values. The result not  909711.6 as expected, rather 909711.625. The customer and me are confused, why the arithmetic module calculates something wrong. Of course the difference is very low, but not explainable.

This issue is existing at an ION7550RTU with firmware V371.

What can I do, to get a right result?

By the way:

At the VIP of a PME 7.2 this issue don't exist, the arithmetic module calculates the right result

Best regards

Sven

1 Reply 1
Hal_Etheridge
Janeway Janeway
Janeway

Re: Arithmetic modul calculates wrong

Internally the ION devices use a float32 for doing (most) calculations.  A float32 has a precision of around 7-8 digits.

The VIP uses a float64 (or double) for its calculations.  A float64 has a precision of 15-16 digits.