New in the Community? Get started here

Schneider Electric Exchange Community

Discuss and solve problems in energy management and automation. Join conversations and share insights on products and solutions. Co-innovate and collaborate with a global network of peers.

Register Now
Resource Advisor - Performance Analytics Forum
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Lieutenant JG

How to Create a Data Presence (Trend) Analysis

This document shows how you can use calculated expressions to identify (and alert on) missing data. This recipe is part of the Calculated Measurements Cookbook .


  1. Find the number of samples that *should* be there for a day, based on the sampling interval of the meter:

    ExpectedNumberOfPoints = SUM PER DAY( (FIRST PER HOUR(SUM PER HOUR(60)/ IntervalSizeMinutes([Electricity:Energy]))) )

    How it works:
    The expression looks at the expected number of points in each hour and adds them up. Not all days are 24 hours long if DST is observed. That is why each hour is inspected individually.
    1. IntervalSizeMinute(~) returns the interval of the data stream. In this example, it is 15.
    2. SUM PER HOUR(60) returns the number 60 every hour (it simply makes a stream of constants)
    3. a/b returns 60 ÷ 15 = 4.  Without d), the expression returns this every 15 minutes.
    4. FIRST PER HOUR says only take the first occurrence of c) every hour. We only want it once every hour.
    5. SUM PER DAY adds up the result for the entire day

  2. Figure out how many values were really recorded for each day.

    ActualNumberOfPoints = SUM PER DAY(IF(ISNULL([Electricity:Energy]), 0,1))

    How it works:
    This expression returns a count of 1 for each good (non-null) point and sums it up in daily total.

  3. MissingNumberOfPoints= ExpectedNumberOfPoints – ActualNumberOfPoints
  4. Create a Trend Analysis of the last few days with all three calculations. Set the Time Interval to “by day” (or  higher).

    How to Create a Data Presence.png


    The dark blue column indicates that data was missing on June 2.  In fact, this facility had a power outage on that day. Notice that the current day (June 4) does not produce any missing points! Only 40 were expected as yet.
  5. Create a daily alert on MissingNumberOfPoints. Set the alert definition to send an email when the value it exceeds 0.  Alternatively, you could divide Actual by Expected and multiply by 100 to get a percentage. You could then create an alarm on the percentage.
Lt. Commander

Re: How to Create a Data Presence (Trend) Analysis

I like this idea! 

What considerations/limitations/standards are there to using this methodology for data presence?  For example:

  • Should we have 3 calculated data stream for every data stream, or should we limit to just a select few per customer/site/source for monitoring purposes?  If we should limit, what are some good ideas for managing?
  • If i care most about % available, is it best to put all of the calculation into a single data stream or break the equations down into three distinct data streams and quantify from there?  Put another way, is there any impact on performance when you cascade calculations vs nest them?
  • Should we put these 3 calculated data streams under Other?  Or should we always just put the data presence under the same category as the data stream being analyzed?  Is "Data" a new commodity?
Lieutenant JG

Re: How to Create a Data Presence (Trend) Analysis

Good questions.

  • I might suggest trying it out on sources that have been misbehaving lately, or for customers who seem particularly sensitive to data presence issues.
  • There is no problem with combining the equations into one big one. I split them out for the purpose of  clarity.
  • As for your third question, I think it's a matter of personal preference. Putting these measurements under the "Other" commodity made sense to me since I was thinking of them like special troubleshooting KPIs. I could have named them better.  Maybe I should have called them MissingCount_ElecEnergy, ExpectedCount_ElecEnergy, and ActualCount_ElecEnergy.  Then again, if a meter is offline, you only need to alert on one of its many (missing) data streams.

Re: How to Create a Data Presence (Trend) Analysis

This is really clever Marty!

Michael Schmitz

Re: How to Create a Data Presence (Trend) Analysis

Nice one!  Only gotcha that I could see is with the current handling if the interval size changes.  Otherwise, a nice little metric!

I would also caution against setting the alert to 0, as timing of data coming in could easily lead to spurious results (lots of 1 interval missing as the last interval hasnt made it in before the alert is run).

Re: How to Create a Data Presence (Trend) Analysis


Martin Mokry do you know if it'd be possible to show also how many datapoints are estimated? Don't know if the calculations engine has access to that information

Best regards!


Re: How to Create a Data Presence (Trend) Analysis

Hi Adria Casas, Im not sure if there is a way in PAM, but Im thinking it might not as that information is stored in EEM.  That said Shaun Hope​ is working on a Data Quality 'table' that is able to show you how many data points are estimated.  It will likely be outside of PAM for starters, but over time may end up with some sort of user interface.


Re: How to Create a Data Presence (Trend) Analysis

This is exactly the case, thanks Kim!

Adria Casas​ we know that customers want to see the estimated data marked clearly in PAM, and expect to introduce something like that in the near future.  No specific timeline has been set though.

Michael Schmitz