This question was originally posted on DCIM Support by Garry Priestland on 2016-12-02
Over the last few months I have had 4x DCE servers for different customers that have suddenly begun to send a very large number of email to email recipients. We are talking 10,000's of emails being sent to multiple recipients, at a rate of several per second. The emails show "Time Detected" on some of the events as being several days, weeks or even months before the Notification time in some cases. Also in each case, we see in the email - "Alert Level: Email - repeat 32651". On at least one of these servers there was not a single Alert Action that was set to repeat the sending of emails.
As far as I can see the emails are actually not regarding recent events but are of old events. It looks like the server may be resending multiple emails for every single alert that has ever occurred. On one occasion I tried to stop the emails by disabling every single device alarm and threshold which did not work. I then tired tom remove the Alert Actions from the Notification policies and still they did not stop. I could only stop them by deleting the email server settings from DCE. As soon as these were entered back in, the emails started again.
The way to fix it seems to be to reboot the server.
The servers were on various versions 7.2.1, 7.2.2, 7.2.5 and 7.2.7.
Obviously the main objective when this happens is to stop the emails being sent which means I cannot easily investigate the issue whilst it is happening.
I have only been able to capture logs from one of the servers too.
Any ideas what may be happening please?
This answer was originally posted on DCIM Support by John Thompson on 2016-12-02
Can you share a link to the logs that you did capture.
Did you capture these before or after you rebooted the server?
Thanks & Regards
This comment was originally posted on DCIM Support by James on 2016-12-02
Same issue happening on my DCE server. The alarms are for current ones (not notifications in the past as Garry has stated). Hopefully the only answer isn't turn it off/on.
This answer was originally posted on DCIM Support by Angela Clark on 2016-12-02
I know it doesn't really "help" but we do have this issue mentioned in the DCE 7.4.1 release notes which I think is likely what you're encountering:
Repeat Email Notifications are Sent for a Single Active Alarm
When the Data Center Expert server approaches approximately 500 days of uptime, it is possible to receive many repeat email notifications for a single active alarm. Reboot the Data Center Expert server to stop the repeat notifications.
For now unfortunately, the only fix is a reboot. It happens when a server uptime is relatively high - can you confirm that the servers you had were probably up close to the 500 days? I will add this post to our bug logged on this issue and continue to try and help drive this to resolution internally.
Discuss challenges in energy and automation with 30,000+ experts and peers.
Find answers in 10,000+ support articles to help solve your product and business challenges.
Find peer based solutions to your questions. Provide answers for fellow community members!