Sign In Help
Schneider Electric
HelpSign In
Schneider Electric Exchange
  • Home
  • Collaborate
  • Develop
  • Shop
Home Collaborate Develop Shop Log in or Register Help

Invite a Co-worker

Send a co-worker an invite to the Exchange portal.Just enter their email address and we’ll connect them to register. After joining, they will belong to the same company.
You have entered an invalid email address. Please re-enter the email address.
This co-worker has already been invited to the Exchange portal. Please invite another co-worker.
Please enter email address
Send Invite Cancel

Invitation Sent

Your invitation was sent.Thanks for sharing Exchange with your co-worker.
Send New Invite Close
  • Home
  • Collaborate
  • Exchange Community
  • :
  • EcoStruxure IT
  • :
  • EcoStruxure IT forum
  • :
  • Automating ETL import
Community Menu
  • Forums
    • By Topic
        • EcoStruxure IT
          • EcoStruxure IT forum
        • Industrial Automation
          • Industry Automation and Control Forum
          • Alliance System Integrators Forum
          • Machine Solutions in the Digital Transformation
          • EcoStruxure Automation Expert / IEC 61499 Forum
          • Industrial Edge Computing Forum
          • Level and Pressure Instrumentation Forum
          • Modicon User Group
          • PLC Club Indonesia
          • SEE Automation Club Forum
          • Fabrika ve Makina Otomasyonu Çözümleri
          • Форум по промышленной автоматизации СНГ
        • SCADA & Telemetry Solutions
          • Geo SCADA Expert Forum
          • SCADA and Telemetry Devices Forum
        • Power Distribution IEC
          • Power Distribution and Digital
          • Power Standards & Regulations
          • Paneelbouw & Energie Distributie
          • Eldistribution & Fastighetsautomation
        • Power Distribution Softwares
          • EcoStruxure Power Design Forum
          • SEE Electrical Building+ Forum
          • LayoutFAST User Group Forum
        • Wireless Information Network Solutions
          • Instrument Area Network
          • Remote Monitoring
          • Tank Level Monitoring
          • Remote Data Collection
        • Solutions for your Business
          • Solutions for Food & Beverage Forum
          • Solutions for Healthcare Forum
    • By Segment
        • Food & Beverage
          • Solutions for Food & Beverage Forum
        • Healthcare
          • Solutions for Healthcare Forum
      • EcoStruxure IT
        • EcoStruxure IT forum
      • Industrial Automation
        • Industry Automation and Control Forum
        • Alliance System Integrators Forum
        • Machine Solutions in the Digital Transformation
        • EcoStruxure Automation Expert / IEC 61499 Forum
        • Industrial Edge Computing Forum
        • Level and Pressure Instrumentation Forum
        • Modicon User Group
        • PLC Club Indonesia
        • SEE Automation Club Forum
        • Fabrika ve Makina Otomasyonu Çözümleri
        • Форум по промышленной автоматизации СНГ
      • SCADA & Telemetry Solutions
        • Geo SCADA Expert Forum
        • SCADA and Telemetry Devices Forum
      • Power Distribution IEC
        • Power Distribution and Digital
        • Power Standards & Regulations
        • Paneelbouw & Energie Distributie
        • Eldistribution & Fastighetsautomation
      • Power Distribution Softwares
        • EcoStruxure Power Design Forum
        • SEE Electrical Building+ Forum
        • LayoutFAST User Group Forum
      • Wireless Information Network Solutions
        • Instrument Area Network
        • Remote Monitoring
        • Tank Level Monitoring
        • Remote Data Collection
      • Solutions for your Business
        • Solutions for Food & Beverage Forum
        • Solutions for Healthcare Forum
      • Food & Beverage
        • Solutions for Food & Beverage Forum
      • Healthcare
        • Solutions for Healthcare Forum
  • Blogs
    • By Topic
        • Industrial Automation
          • Industrial Edge Computing Blog
          • Industry 4.0 Blog
          • Industrie du Futur France
        • SCADA & Telemetry Solutions
          • SCADA and Telemetry Blog
        • Power Distribution IEC
          • Power Events & Webinars
          • Power Foundations Blog
        • Power Distribution NEMA
          • NEMA Power Foundations Blog
        • Power Distribution Softwares
          • EcoStruxure Power Design Blog
          • SEE Electrical Building+ Blog
        • Solutions for your Business
          • Solutions for Food & Beverage Blog
          • Solutions for Healthcare Blog
          • Solutions for Retail Blog
        • Community experts & publishers
          • Publishers Community
    • By Segment
        • Food & Beverage
          • Solutions for Food & Beverage Blog
        • Healthcare
          • Solutions for Healthcare Blog
        • Retail
          • Solutions for Retail Blog
      • Industrial Automation
        • Industrial Edge Computing Blog
        • Industry 4.0 Blog
        • Industrie du Futur France
      • SCADA & Telemetry Solutions
        • SCADA and Telemetry Blog
      • Power Distribution IEC
        • Power Events & Webinars
        • Power Foundations Blog
      • Power Distribution NEMA
        • NEMA Power Foundations Blog
      • Power Distribution Softwares
        • EcoStruxure Power Design Blog
        • SEE Electrical Building+ Blog
      • Solutions for your Business
        • Solutions for Food & Beverage Blog
        • Solutions for Healthcare Blog
        • Solutions for Retail Blog
      • Community experts & publishers
        • Publishers Community
      • Food & Beverage
        • Solutions for Food & Beverage Blog
      • Healthcare
        • Solutions for Healthcare Blog
      • Retail
        • Solutions for Retail Blog
  • Ideas
        • Industrial Automation
          • Modicon Ideas & new features
        • SCADA & Telemetry Solutions
          • Geo SCADA Expert Ideas
          • SCADA and Telemetry Devices Ideas
  • Knowledge Center
    • Building Automation Knowledge Base
    • Industrial Automation Knowledge Base
    • Industrial Automation How-to videos
    • SCADA & Telemetry Solutions Knowledge Base
    • Digital E-books
    • Success Stories Corner
    • Power Talks
  • Events & Webinars
    • Innovation Talks
    • Innovation Summit
    • Let's Exchange Series
    • Technology Partners
  • Support
    • Ask Exchange
    • Leaderboard
    • Our Community Guidelines
    • Community User Guide
    • How-To & Best Practices
    • More
Join Now
How can we help?
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
51307members
Join Now
245514posts
Join Now

Automating ETL import

Options
  • Subscribe to RSS Feed
  • Mark Topic as New
  • Mark Topic as Read
  • Float this Topic for Current User
  • Bookmark
  • Subscribe
  • Mute
  • Printer Friendly Page
Back to EcoStruxure IT forum
DCIM_Support
Picard DCIM_Support
Picard
‎2020-07-03 10:41 PM
0 Likes
11
227
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content
‎2020-07-03 10:41 PM

Automating ETL import

This question was originally posted on DCIM Support by François on 2017-09-06


Hi team, 

An ETL question on behalf of a customer who tries to implement an ETL integration :

" We were able to test ETL to import / export to DCO, it is actually possible to interconnect our internal database with DCO. But it would seem that automation is impossible : in order to retrieve the serial number we have to manually "execute" the transformation :

 



And in order to associate the new equipment we need to suppress the external system previously created :



Could you please help understanding how these steps can be automated ? "

Thanks,

François

(CID:124523885)

Labels
  • Data Center Operation
Share
  • All forum topics
  • Previous Topic
  • Next Topic
11 Replies 11
DCIM_Support
Picard DCIM_Support
Picard
‎2020-07-03 10:41 PM
0 Likes
4
227
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content
‎2020-07-03 10:41 PM

Re: Automating ETL import

This answer was originally posted on DCIM Support by Ed Tarento on 2017-09-07


Hi François

My understanding of the standard import process with v8 DCO is that once you have the asset in the ETL Import DB, and the ETL Import asset is associated in the DCO GUI with the DCO asset, new or updated pieces of information that you apply to the ETL Import DB asset, should be seen on the DCO asset in the GUI once the scheduled refresh occurs.  Refresh (5 minutes in your image above) is configurable and should be carefully set according to the amount of info.  5 minute is very aggressive.

It looks like you are not using the standard DCO ETL Import process and you haven't said which version you are using.  I know that there have been some issues with older versions.  Can you try the standard process on v8?

I hope this helps

Kind regards

Ed

(CID:124524074)

Share
DCIM_Support
Picard DCIM_Support
Picard
‎2020-07-03 10:41 PM
0 Likes
0
227
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content
‎2020-07-03 10:41 PM

Re: Automating ETL import

This comment was originally posted on DCIM Support by François on 2017-09-07


Hi Ed,

Thanks for the feedback. I had a call with the customer today : DCO version is 8.1.

They are using the ETL as a two way integration between DCO and a Lotus database (with an xls file inbetween DCO and Lotus). Lotus is used to push a new equipment to DCO (IT equipment including model, name, serial number...) while DCO will push back the location to Lotus.

The exportation works as expected for them, but they are experiencing issues with the importation :

  • In the user interface, clicking on "Synchronize External Data" (Figure 1) or waiting for the next scheduled refresh does not always discover new equipments 
  • When it does, the new equipment will show up as "greyed" and with all properties filled but the serial number 
  • To import the serial number properly, they need to navigate to System Setup > External System Configuration, then edit the ETL integration and click on "Run Now" (Figure 2)
  • The "greyed" equipment cannot be associated to the model : to associate it, they need first to navigate to System Setup > External System Configuration and remove the ETL integration. 

Then, a few additional informations :

  • They are currently working with ETL under a test environment
  • They have the following message that pops up at regular intervals on the user interface "The connection to the server has been interrupted, but is now restored. Any updates from the server that were running at the time of the interruption will resume."
  • Here are the xls and ktr file they use : Files.zip
  • In a production environment, they would except the scheduled refresh to occur daily and to allow them to see under the "Integration ETL" the new equipments (including their properties) ready to be associated to the model. 

Figure 1 :

 

Figure 2 :

 

Do you have any idea of what could go wrong or how to troubleshoot it ? 

Further more, you are mentionning a "standard DCO ETL Import process". Where can I find this process ? I actually searched for an end to end process for ETL without success so far.

 

Thanks for your help,

 

François

(CID:124524513)

Share
DCIM_Support
Picard DCIM_Support
Picard
‎2020-07-03 10:41 PM
0 Likes
0
227
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content
‎2020-07-03 10:41 PM

Re: Automating ETL import

This comment was originally posted on DCIM Support by Ed Tarento on 2017-09-11


Hi Francois,

I'm on leave hence the delay in my response.  I'm not sure if or when I might be able to take a look at the ktr.   My reference to "standard ETL import process" refers to creating an ETL Import DB and adding a DCO external system to update this DB at regular intervals.  (the same also applies to export).  Therefore, in this "standard" process, an Excel sheet is not required.  Your external system (Lotus Notes), reads data from the ETL Export DB and writes and new/updated info to the ETL Import DB via SQL statements.  All of this is document here.  But I'm not sure if Notes will read/write SQL?  I've successfully used this method in the past, because it's industry standard, more straightforward, easier to debug and understand but of course it only applies to external systems which can read/write SQL. 

Cheers

Ed

(CID:124525464)

Share
DCIM_Support
Picard DCIM_Support
Picard
‎2020-07-03 10:41 PM
0 Likes
0
227
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content
‎2020-07-03 10:41 PM

Re: Automating ETL import

This comment was originally posted on DCIM Support by François on 2017-09-28


Hi Ed,

Thanks for your feedback. Following your comments, we tried again the ktr and the import.xls file prepared by the customer.

Here is what we observed (please note that the customer is using the import.xls as an intermediate between Lotus Note and DCO) :

1. Importing a new equipment works OK :

-> adding a new line in the import.xls file will result in showing a new equipment under the external system in DCO. This equipment can then be added/associated to the model and will update the attributes of the DCO object (for intance name and serial number) after a few minutes.

I noticed only a strange behaviour in DCO as you can see here :

A new line in the tooltip seems to appear at each synchronization with the external system.


2. Updating an existing equipment does not work :

-> modifying a line for an equipment in the excel file (for instance its name or serial number ) will modify neither the external system discovery, nor the equipment property in DCO. Checking at webmin, we actually noticed that modifying a property of an equipment in the excel file stops the staging database whose status swtich to "ERROR".

But we are not sure here to understand the process of modifying the properties of an equipment in DCO using the ETL import. Should it normally work to update a property in the excel file as we do ? Or do we need two transformations instead : one to add a new equipment, a second one to update an existing equipment ?

 

Then, a last question about the unique identifier of an equipment :

DCO seems to require a unique identifier for each equipment added. This is the ci_id column of the import.xls file.

Is this mandatory to add this field in the excel document ? Cannot it be automatically filled by DCO ?

Alternativelly, the customer already uses a unique identifier coming from Lotus Note for his equipments. Could we use it as the ci_id field of the import.xls file ? If yes, how can this unique identifier be displayed in DCO ? The customer indeed wants to see this identifier in the equipment properties in DCO.

 

Thanks,

François

 

 

(CID:125213327)

Share
DCIM_Support
Picard DCIM_Support
Picard
‎2020-07-03 10:41 PM
0 Likes
0
227
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content
‎2020-07-03 10:41 PM

Re: Automating ETL import

This comment was originally posted on DCIM Support by François on 2017-09-28


To add to my previous comment following aditional testing :

I was initally uploading the updates of import.xls file to the pentaho/import folder on the DCO server using the webmin interface. This leaded to a duplication of the associacted devices as shown in my previous screen shot.

I have now mounted a share foler onto DCO as pentaho/import. Uploading a new import.xls file in the share folder or modifying directly the import .xls file into the share folder seems to fix this problem of duplicated associated devices : 

 

 

This leaves open my point 2 on how to update and the question about the unique identifier.

Rgds

F 

(CID:125213521)

Share
DCIM_Support
Picard DCIM_Support
Picard
‎2020-07-03 10:41 PM
0 Likes
4
227
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content
‎2020-07-03 10:41 PM

Re: Automating ETL import

This answer was originally posted on DCIM Support by Ed Tarento on 2017-10-03


Hi François

I finally got some time to review your KTR and XLS.  They both worked for me first time including an update to the serial number.  For the first run (adding the devices) I placed the XLS in /data/pentaho/import and made sure ownership and permissions were appropriate.  On second run (update a serial number) I first removed the XLS then copied the new one in and set file ownership and permissions.  If your update is still failing, perhaps try this step

AFAIK the ci_id in the XLS is the primary and unique key for each imported Lotus Notes device.  Perhaps you could use a unique Lotus ID for each device - might have some future benefit.  On some occasions, I've used the internal DCO ID, as exported via the ETL DB but it doesn't matter as long as the ci_id is used to uniquely identify each Lotus device.  In the XLS, I believe all other fields can be duplicated, e.g. the name and serial number

My DCO is 8.1.0-70629-cf193f

I did see some strange behaviour with the second Nexus 2248TP.  DCO would only let me associate it with a rack, not a device.  The 1st device of this type EQ100002 associated just fine with a device.  

Cheers

Ed

(CID:126158817)

Share
DCIM_Support
Picard DCIM_Support
Picard
‎2020-07-03 10:42 PM
0 Likes
0
227
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content
‎2020-07-03 10:42 PM

Re: Automating ETL import

This comment was originally posted on DCIM Support by François on 2017-10-11


Hi Ed Tarento,

User permission and ownership for the file and for the transformation seems to be right :

 

 From here, modifying the serial number (and only the serial number) works just as you describe :

 

Note however that I have never been able to see the serial number reported into the equipment properties (unlike the name) as you can see here :

 

I have then noticed that modifying the ci_name does not work. Any attempt to do so result with an error :

 

Lastly, I am also noticing that I am progressively losing the association between equipments. All equipment with a the blue “I” sign were previously associated :

 

It seems that the “unassociation“ occurs every time I  add a new “import.xls” file to the pentaho/import folder and then synchronize the external system.

Cheers,

F

 

 

 

(CID:126162558)

Share
DCIM_Support
Picard DCIM_Support
Picard
‎2020-07-03 10:42 PM
0 Likes
0
227
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content
‎2020-07-03 10:42 PM

Re: Automating ETL import

This comment was originally posted on DCIM Support by Ed Tarento on 2017-10-12


What version of DCO are using?

(CID:126162904)

Share
DCIM_Support
Picard DCIM_Support
Picard
‎2020-07-03 10:42 PM
0 Likes
0
227
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content
‎2020-07-03 10:42 PM

Re: Automating ETL import

This comment was originally posted on DCIM Support by François on 2017-10-12


Same as yours : 8.1.0-70629-cf193f

This is the version I have been doing my tests with.

(CID:126163233)

Share
DCIM_Support
Picard DCIM_Support
Picard
‎2020-07-03 10:42 PM
0 Likes
0
227
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content
‎2020-07-03 10:42 PM

Re: Automating ETL import

This comment was originally posted on DCIM Support by Ed Tarento on 2017-10-16


I'll set up the PDI spoon system and see what I can find

Cheers

(CID:126164021)

Share
DCIM_Support
Picard DCIM_Support
Picard
‎2020-07-03 10:42 PM
0 Likes
0
227
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content
‎2020-07-03 10:42 PM

🔒 Closed

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.

Share
Top Experts
User Count
Jef
Commander Jef Commander
41
APC_Steve
Lt. Commander APC_Steve Lt. Commander
24
gsterling
Lt. Commander gsterling Lt. Commander
20
Cory_McDonald
Lieutenant Cory_McDonald Lieutenant
13
GavanOB
Lieutenant GavanOB Lieutenant
7
See More Top Experts
Find a Service Provider
Find a certified partner to help you address your integration, installation, maintenance and project needs.
View all Providers
Support

Have a question? Please contact us with details, and we will respond.

Contact Us
FAQ

Look through existing questions to find popular answers.

Learn More
About

Want to know more about Exchange and its possibilities?

Learn More

Full access is just steps away!

Join Exchange for FREE and get unlimited access to our global community of experts.

Connect with Peers & Experts

Discuss challenges in energy and automation with 30,000+ experts and peers.

Get Support in Our Knowledge Base

Find answers in 10,000+ support articles to help solve your product and business challenges.

Ask Questions. Give Solutions

Find peer based solutions to your questions. Provide answers for fellow community members!

Register today for FREE

Register Now

Already have an account?Log in

About Us FAQ Terms & Conditions Privacy Notice Change your cookie settings
©2020, Schneider Electric