Data Center Expert Web Services Alarm Clears and Acknowledgements

rtirak · ‎2020-10-08

Currently we are working with DCE 2.7.2 and specifically I am working with the AlarmService and EventService trying to integrate the alarms from the DCE into our other monitoring product.

Using the WSDL web services I am able to actively poll the Alarms with a python script. My hang up is trying to update the alarm when it’s clearing or is acknowledged in DCE.

Right now I am making a call to the ISXCentralAlarmsService_v2_0.wsdl

getting All current alarms and there status, and creating those in our monitoring tool. However my question is what’s the proper way to handle when an alarm clears from the DCE or is Acknowledged?

What API endpoint can I use when calling to update current alarms each polling cycle?

Thanks in advance for your time and help.

Shaun · ‎2020-10-08

Hi,

I believe the getAlarmsDeltaHistory method in AlarmsService_v2.0 should fill this role - it takes a parameter secondsInPast and returns a list of alarms whose status has changed within that period. You should then be able to work from ISXCAlarm.ISXCAlarmState for a value of ACTIVE/RESOLVED. (I must confess I'm not sure how 'acknowledged' is reflected in this).

When using an absolute time period like this, it's recommended to have some overlap between polls (eg, if you poll every 5 minutes, request 330 seconds of alarms rather than 300), and de-duplicate on alarm IDs. Personally I'd be tempted to still call getAllActiveAlarms periodically (or at least when any poll has been missed) to catch-up.

(This method works best if you're working with polling periods of handfuls of minutes. If you need very short polling periods, it's better to look into the publish/subscribe ws-eventing model - or simply receiving http-posts as a DCE alarm action. If you need very long polling intervals, getAlarmHistory will work better than DeltaHistory)

rtirak · ‎2020-10-08

@Shaun Thanks for the information. To start I was looking at the Alarm Action HTTP Post as you suggested however it seems that I am unable to use any macros or add/edit values in the payload.

I then looked as you are suggesting the ws-eventing-module and the EventService with publish/subscrbe however the lack of documentation has not even allowed me to really get very far with it. Could you possibly provide me a suggestion or show me an example maybe in python or some other language is fine, of how I can use the ws-eventing-moudle successfully?

I have been unable to even use a client and successfully test Subscribing to the DCE to receive Events with say a postman api call.

After all of that I have turned to this form in hopes of getting some more information or maybe some example code that I can look at to help me get to the finish line? Any help would be appreciated.

rtirak · ‎2020-10-08

Also it does seem everywhere I look, in the community and forums I see the following methods mentioned:

Alarms Service Methods

Device Groups Service Methods

Device Service Methods

Sensor Service Methods

But no where am I able to find, much talk about the Event Service, or examples it seems.

Cory_McDonald · ‎2020-10-08

Hello @rtirak,

Data Center Expert web services documentation can be found here: https://www.apc.com/shop/us/en/products/StruxureWare-Data-Center-Expert-Web-Services-API/P-SFWEBSVC

Inside that ZIP file you will find the DCE_WS_EVENTING.doc and also the WSDL file: ISXCentralEventService_v1_0.

The Word document goes over example requests and responses and the WSDL file can be used to make the requests through a program.

Kind Regards,

Cory

rtirak · ‎2020-10-08

@Cory_McDonald Hi Corey, yes this is where I started I read over this document, and than attempted to use the wsdl service located at http://servername/eventing/ISXCentralServer

I tried to use the Documents subscription example to subscribe to events from the DCE like this:

1.1.1 Subscription Request

Request Method: POST

Request URI: /eventing/ae042c72-b667-12f3-d017-d0467f7788b2

Request Version: HTTP/1.1

Authorization: Basic

Dst Port: 80

<?xml version="1.0" encoding="UTF-8"?>

<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" xmlns:wsd="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing">

<soap:Header>

<wsa:To>http://10.218.14.81:80/eventing/ae042c72-b667-12f3-d017-d0467f7788b2</wsa:To>

<wsa:Action>http://schemas.xmlsoap.org/ws/2004/08/eventing/Subscribe</wsa:Action>

<wsa:FaultTo>

<wsa:Address>http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous </wsa:Address>

</wsa:FaultTo>

<wsa:MessageID>urn:uuid:dd8e6236-99a7-3542-49c3-402d5c5c9b5d</wsa:MessageID>

<wsa:ReplyTo>

<wsa:Address>http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous </wsa:Address>

</wsa:ReplyTo>

</soap:Header>

<soap:Body>

<wse:Subscribe xmlns:wse="http://schemas.xmlsoap.org/ws/2004/08/eventing">

<wse:EndTo>

<wsa:Address>http://159.215.12.212:8367/d283392f-9215-f305-800b-90d8c4458dfe </wsa:Address>

</wse:EndTo>

<wse:Delivery>

<wse:NotifyTo>

<wsa:Address>http://159.215.12.212:8367/f090b358-0617-51fa-a344-cf0359026634 </wsa:Address>

<wsa:ReferenceParameters>

<DeviceName>Data Center Expert</DeviceName>

</wsa:ReferenceParameters>

</wse:NotifyTo>

</wse:Delivery>

<wse:Expires>PT10M</wse:Expires>

<wse:Filter Dialect="http://schemas.xmlsoap.org/ws/2006/02/devprof/Action"> http://www.apc.com/dpws/wsdl/ISXCentralEvents-v1/AlarmEvents/AlarmStatusChanged </wse:Filter>

</wse:Subscribe>

</soap:Body>

</soap:Envelope>

In attempting to do this I am only ever able to get the following response

1. Postman just spins and and never stops and I never recieve anything back

or if I edit the xml improperly I get:

2.

?xml version="1.0" encoding="UTF-8"?>

<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" xmlns:wsd="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing">

<soap:Body>

<soap:Fault>

<soap:Value>soap:Sender</soap:Value>

<soap:SubCode>

<soap:Value>wsa:MessageInformationHeaderRequired</soap:Value>

</soap:SubCode>

</faultcode>

<soap:Reason>

<soap:Text>A required header representing a Message Addressing Property is not present</soap:Text>

</soap:Reason>

<soap:Detail>

<wsa:ProblemHeaderQName>wsa:Action</wsa:ProblemHeaderQName>

</soap:Detail>

</soap:Fault>

</soap:Body>

</soap:Envelope>

Shaun · ‎2020-10-08

I'll admit I'm getting out of my depth on this one, so bear with me if this isn't in as much detail as we might like. I've never tried this either, and so far it's giving me a whole new appreciation for websockets+json!

The uuid in the URL you're using (ae042c72-b667-12f3-d017-d0467f7788b2) is an example, and shouldn't be valid on your machine. TYou can find the end-point for your device by visiting /eventing/ISXCentralServer - but as it's generated at boot, it will change so it's best to implement discovery for this:

This is in the document as 3.8.1; Device Profile Request. Highligted in the documented is the wsa:MessageID - this should be yours to create (it's a short-lived transaction identifier. You send a message with a MessageID, and you get a response with RelatedTo header bearing the same token. It's intended this is randomly generated for each message, to match the response to its origin. If properly handled this allows for replies to come out-of-order.)

The response to this is 3.8.2 Device Profile Response. You should see your MessageID as the Header.RelatesTo. But more notably, towards the end of the reponse you'll see a Relationship section in the Body which includes wsa:Address - this should be your request URL in the Subscription Request.

I haven't found a python library that'll make our lives any easier here. I usually use 'zeep' for soap/wsdl, but WS-Eventing is a mammoth spec which is usually handled by larger frameworks (eg Apache Axis).

rtirak · ‎2020-10-08

@Shaun

So I am able to see those values that you pointed out. I guess my first question is, am I correct in my understanding that I should be able to subscribe to events from the DCE using this Service, and all devices that alarm in the DCE alarm console I would be able to see those events?

Or would I have to subscribe to every device individually in order to get the events from each and everyone even though they all also report to the DCE server?

Yes I also usually use zeep when using WSDL/SOAP, however all my efforts I had failed so far.

As you stated, "You can find the end-point for your device by visiting /eventing/ISXCentralServer - but as it's generated at boot, it will change so it's best to implement discovery for this" I did notice this thanks for pointing it as I almost forgot.

rtirak · ‎2020-10-08

@Shaun

Also I VERY VERY much appreciate any and all help. So No worries at all, take your time, and I will take any advice / help you or anyone has to offer.

Thanks!

rtirak · ‎2020-10-08

When I attempt to run the example in 3.8.1 Device Profile request. I am sending a POST request to http://myserver.com/eventing/dceUUID with Postman. In the body is a SOAP envelop like the one in 3.8.1 however mine looks like this as it has my values in it:

<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" xmlns:wsd="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing">

<soap:Header>

<wsa:To>urn:uuid:ISXCentralServer</wsa:To>

<wsa:Action>http://schemas.xmlsoap.org/ws/2004/09/transfer/Get</wsa:Action>

<wsa:FaultTo>

<wsa:Address>http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous </wsa:Address>

</wsa:FaultTo>

<wsa:MessageID>urn:uuid:331993bd-a3e6-24a3-44a0-98ec4d6a6b14</wsa:MessageID>

<wsa:ReplyTo>

<wsa:Address>http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous </wsa:Address>

</wsa:ReplyTo>

</soap:Header>

<soap:Body/>

</soap:Envelope>

I send that and I get a 500 response like this:

<?xml version="1.0" encoding="UTF-8"?>

<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" xmlns:wsd="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing">

<soap:Body>

<soap:Fault>

<soap:Value>soap:Receiver</soap:Value>

</faultcode>

<soap:Reason>

<soap:Text>More than 10000 channels open. Exception to prevent memory leak</soap:Text>

</soap:Reason>

</soap:Fault>

</soap:Body>

</soap:Envelope>

Seems pretty straight forward like I should be able to do this and get a response but I just get the above error response

Cory_McDonald · ‎2020-10-08

Hello @rtirak,

Sorry for the confusion, I should have read all of the history of the post in detail instead of skimming. I do not believe that you can use the Eventing service within SOAP UI or Postman after looking through it. I believe they need to come from an HTTP Post server so that it can accept the responses. Unfortunately I do not have any experience with HTTP Posts.

For your other question, about acknowledge alarms or hidden alarms, you can use the service for getAcknowledgementHistoryForAlarm which is outlined in the DCE_WS_DEVELOPMENT.doc file.

Start by polling the system for all active alarms:

http://{{DCE_Server_IP}}/integration/services/ISXCentralAlarmsService_v2_0

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:isx="http://www.apc.com/stdws/xsd/ISXCentralAlarms-v2">
    <soapenv:Header/>
    <soapenv:Body>
        <isx:getAllActiveAlarmsRequest>
            <!--Optional:-->
            <isx:locale>?</isx:locale>
        </isx:getAllActiveAlarmsRequest>
    </soapenv:Body>
</soapenv:Envelope>

Request Response:

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <soap:Body xmlns:ns1="http://www.apc.com/stdws/xsd/ISXCentralAlarms-v2">
        <ns1:getAllActiveAlarmsResponse>
            <ns1:ArrayOfISXCAlarm>
                <ns2:ISXCAlarm xmlns:ns2="http://www.apc.com/stdws/xsd/ISXCentral/2009/10">
                    <ns2:ISXCElement>
                        <ns2:ISXCElementType>ALARM</ns2:ISXCElementType>
                        <ns2:id>nbErrorCond_maintenance_Skc9QrKk</ns2:id>
                    </ns2:ISXCElement>
                    <ns2:activeTime>2020-07-18T14:26:24.267-04:00</ns2:activeTime>
                    <ns2:ISXCAlarmSeverity>INFORMATIONAL</ns2:ISXCAlarmSeverity>
                    <ns2:ISXCAlarmState>ACTIVE</ns2:ISXCAlarmState>
                    <ns2:description>Notifications have been disabled for maintenance on this device.</ns2:description>
                    <ns2:deviceID>B9C25C7_nbSNMPEnc7DCD36FA</ns2:deviceID>
                    <ns2:recommendedAction></ns2:recommendedAction>
                    <ns2:resolvedTime xsi:nil="true" />
                    <ns2:sensorID>B9C25C7_nbSNMPEnc7DCD36FA_MAINTENANCEERROR</ns2:sensorID>
                </ns2:ISXCAlarm>

You can then grab each of id's of the alarms and use the below steps to determine if they are still active or not.

Get Request:

GET:  http://{{DCE_Server_IP}}/integration/services/ISXCentralAlarmsService_v2_0?

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:isx="http://www.apc.com/stdws/xsd/ISXCentralAlarms-v2">
   <soapenv:Header/>
   <soapenv:Body>
      <isx:getAcknowledgmentForAlarmRequest>
         <isx:alarmID>B1db239_nbErrorCond_5AE3CEC1</isx:alarmID>
      </isx:getAcknowledgmentForAlarmRequest>
   </soapenv:Body>
</soapenv:Envelope>

Response:

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <soap:Body xmlns:ns1="http://www.apc.com/stdws/xsd/ISXCentralAlarms-v2">
        <ns1:getAcknowledgmentForAlarmResponse>
            <ns1:alarmID>B1db239_nbErrorCond_5AE3CEC1</ns1:alarmID>
            <ns1:isAlarmAlertSuppressed>false</ns1:isAlarmAlertSuppressed>
            <ns1:isAlarmHidden>false</ns1:isAlarmHidden>
        </ns1:getAcknowledgmentForAlarmResponse>
    </soap:Body>
</soap:Envelope>

You can see that there are two values returned for "isAlarmAlertSuppressed" and "isAlarmHidden".

In regards to the status on an alarm, you can see the if an alarm is still active by taking the alarmID and using that with the getAlarmByID request.

Get Request:

GET:  http://{{DCE_Server_IP}}/integration/services/ISXCentralAlarmsService_v2_0

Request Body:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:isx="http://www.apc.com/stdws/xsd/ISXCentralAlarms-v2">
    <soapenv:Header/>
    <soapenv:Body>
        <isx:getAlarmByIDRequest>
            <isx:ISXCElementID>B1db239_nbErrorCond_5AE3CEC1</isx:ISXCElementID>
            <!--Optional:-->
            <isx:locale>?</isx:locale>
        </isx:getAlarmByIDRequest>
    </soapenv:Body>
</soapenv:Envelope>

Response:

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <soap:Body xmlns:ns1="http://www.apc.com/stdws/xsd/ISXCentralAlarms-v2">
        <ns1:getAlarmByIDResponse>
            <ns1:ISXCAlarm>
                <ns2:ISXCElement xmlns:ns2="http://www.apc.com/stdws/xsd/ISXCentral/2009/10">
                    <ns2:ISXCElementType>ALARM</ns2:ISXCElementType>
                    <ns2:id>B1db239_nbErrorCond_5AE3CEC1</ns2:id>
                </ns2:ISXCElement>
                <ns2:activeTime xmlns:ns2="http://www.apc.com/stdws/xsd/ISXCentral/2009/10">2020-09-01T14:29:33.931-04:00</ns2:activeTime>
                <ns2:ISXCAlarmSeverity xmlns:ns2="http://www.apc.com/stdws/xsd/ISXCentral/2009/10">WARNING</ns2:ISXCAlarmSeverity>
                <ns2:ISXCAlarmState xmlns:ns2="http://www.apc.com/stdws/xsd/ISXCentral/2009/10">ACTIVE</ns2:ISXCAlarmState>
                <ns2:description xmlns:ns2="http://www.apc.com/stdws/xsd/ISXCentral/2009/10">The battery measured life test indicates that the battery is near the end of life. The battery should be replaced.</ns2:description>
                <ns2:deviceID xmlns:ns2="http://www.apc.com/stdws/xsd/ISXCentral/2009/10">B1db239_nbSNMPEnc06CCB693</ns2:deviceID>
                <ns2:recommendedAction xmlns:ns2="http://www.apc.com/stdws/xsd/ISXCentral/2009/10">Recommended Action: Replace all faulty batteries. You can use the APC Product Selectors (http://www.apc.com/go/direct/index.cfm?tag=selectors) page to order new batteries.</ns2:recommendedAction>
                <ns2:resolvedTime xmlns:ns2="http://www.apc.com/stdws/xsd/ISXCentral/2009/10" xsi:nil="true" />
                <ns2:sensorID xmlns:ns2="http://www.apc.com/stdws/xsd/ISXCentral/2009/10"></ns2:sensorID>
            </ns1:ISXCAlarm>
        </ns1:getAlarmByIDResponse>
    </soap:Body>
</soap:Envelope>

Using the ISXCAlarmState value, you can then know if the alarm is still active or not. When an alarm is resolved, there will also be a value for "resolvedTime".

Not sure if this will work for your setup, but you should be able to request All alarms, run a comparison with what you are still showing as active, and then request updates (getAlarmByID) for the ones that are missing from the most recent pull, which means that they should have cleared.

Regards,

Cory

rtirak · ‎2020-10-09

Hi Corey,

I appreciate your help sir!

Here is what I am currently doing with python. I am just running this every 5 minutes. Here is the decision block of the python:

#Iterate through each alarm result

for alarms in response:

#Pull Alarm ID specifically so it can be sent to getDeviceName from ID

deviceID = alarms['deviceID']

#Call getDeviceName in order to take the alarms deviceID and populate its Hostname in the UIM alarm

deviceHostName = getDeviceName(deviceID)

#If Alarm is No Longer Active Clear it from the UIM Console.

if alarms['ISXCAlarmState'] == 'RESOLVED':

#source of alarm

source = deviceHostName

#alarm subsystem id if unsure use 1.1

ss_id = '2.35'

#alarm suppression key

supp_key = alarms['deviceID'] + '/' + alarms['ISXCElement']['id']

#alarm level e.g. clear, warning, minor, major, critical

level = 'clear'

#alarm message / text detail

msg = alarms['description'] + 'HAS CLEARED'

#Send Alarm Clear

create_alarm(uim_ws, source, ss_id, supp_key, level, msg)

#Check for Alarm State and Severity to map UIM Severity properly, if Failure then critical alarm

elif alarms['ISXCAlarmState'] == 'ACTIVE' and alarms['ISXCAlarmSeverity'] == 'FAILURE':

#source of alarm

source = deviceHostName

#alarm subsystem id if unsure use 1.1

ss_id = '2.35'

#alarm suppression key

supp_key = alarms['deviceID'] + '/' + alarms['ISXCElement']['id']

#alarm level e.g. clear, warning, minor, major, critical

level = 'major'

#alarm message / text detail

msg = alarms['description']

#Send Alarm

create_alarm(uim_ws, source, ss_id, supp_key, level, msg)

#if Informational alarm then minor alarm in UIM

elif alarms['ISXCAlarmState'] == 'ACTIVE' and alarms['ISXCAlarmSeverity'] == 'WARNING' or alarms['ISXCAlarmSeverity'] == 'INFORMATIONAL':

#source of alarm

source = deviceHostName

#alarm subsystem id if unsure use 1.1

ss_id = '2.35'

#alarm suppression key

supp_key = alarms['deviceID'] + '/' + alarms['ISXCElement']['id']

#alarm level e.g. clear, warning, minor, major, critical

level = 'minor'

#alarm message / text detail

msg = alarms['description']

#Send Alarm

create_alarm(uim_ws, source, ss_id, supp_key, level, msg)

#If Alarm is No Longer Active Clear it from the UIM Console.

elif alarms['ISXCAlarmState'] != 'ACTIVE':

#source of alarm

source = deviceHostName

#alarm subsystem id if unsure use 1.1

ss_id = '2.35'

#alarm suppression key

supp_key = alarms['deviceID'] + '/' + alarms['ISXCElement']['id']

#alarm level e.g. clear, warning, minor, major, critical

level = 'clear'

#alarm message / text detail

msg = alarms['description'] + 'HAS CLEARED'

#Send Alarm Clear

create_alarm(uim_ws, source, ss_id, supp_key, level, msg)

So what I am doing is every 5 minutes requesting all alarms as you suggest taking the JSON output and checking each alarm.

Specifically one of the keys I am looking at as I believe you suggest is ISXCAlarmState I am checking that State and if the alarm is RESOLVED than I am sending the clear message to the other tool in order to clear the alarm from that tools alarm console.

I guess my question is, so when an alarm comes in to EDC and lets say its been active for 4 hours. And in that 4 hours the issue has been no fixed. The Device or DCE, does it send another separate JSON record of the alarm to the console with resolved.

Or is it updating the Original alarm record now for the ISXCAlarmState field to now read resolved?

rtirak · ‎2020-10-09

@Cory_McDonald

So if I put your suggestion in place, wont that fail when there are no active alarms? So lets use your steps with an example

So this will run("POLL") every 5 minutes.

1. It requests alarm list and resturns two alarms

2. It takes the IDs of the alarms check and sees they are still active alarms.

3. It sends the alerts to the other monitoring console so alerts are generated on the other tools console one for each alarm.

4. Script ends.

waits 5 minutes then runs again

1. It requests all active alarms and now there is only 1 that returns

2. Right here it now has no way to know which of the two alarms cleared from its last run as that info is gone, so its cant send a clear to the other tool to clear the proper alarm from the board.

Unless your talking about caching the results of each time the script runs so that it can check those values at next run time. This is where I was hoping that being able to do something with the Eventing Service might help.

Please let me know if I am miss understanding or how this would work, as if your way will, I would like to attempt to put it in place.

Thanks!

Cory_McDonald · ‎2020-10-09

@rtirak,

Yes, when I referenced comparing the current results to previous results, this would require a DB or arrays or something to store the data and compare it to the next poll to find out what is missing or added.

If reference to the SOAP Eventing service, my recommendation is that you reach out to your local sales team and request a custom service for integration. Someone should then be able to reach out to you to discuss any project costs to perform the actions. This recommendation comes due to the fact that you are likely unable to use Postman or Soap UI for testing since it requires an HTTP Post server for the requests and responses.

Regards,

Cory

rtirak · ‎2020-10-22

So for now I took @Shaun recommendation, what I have is a python script that is running every 5 minutes and I am first getting all active alarms.

than I am also then running the getAlarmHistory call for the last 330 seconds and updating any existing alarms that maybe have changed in the last 6 minutes.

than the script sits and a schedule task 5 minutes later runs it again. And that’s my polling period/run time.

I guess my question is, does this seem sound for now? Will I miss anything running this way?

also thanks again everyone for there Input.

Cory_McDonald · ‎2020-11-01

@rtirak,

I will leave it up to others to comment, as I do not have direct experience with this sort of solution. Hopefully others can comment on if a solution like this worked for them. It appears, per logic, that it should cover what you need.

Regards,

Cory

Invite a Co-worker

Invitation Sent