KScope19 – Deep-Dive into Data Integration

There were a lot of big announcements about the future direction data management is taking, and except for the proposed loss of our favourite FDMEE fish, they were all absolutely fantastic, and there is so much to be excited about. I’m going to give my views on the biggest announcements and the direction Data Integration is taking, and I’ll be getting some tutorials together once these features are available. But first:

Jargon Buster

• FDMEE – on-premise Financial Data Management Extended Edition, amazing ETL tool for mappings, data import and export.
• Data Management – the cloud version of FDMEE, built into PBCS etc
• Data Integration – the shiny, fishless facelift of Data Management

Data Integration

Be ready, the new world is coming, and eventually it will replace the Data Management we all know and love. Data Integration functions the same as Data Management in the background, but it’s been designed to be a lot more user friendly. For example, rather than creating a source, target, then location, then import format, then load rule you will in the future just create an integration which encompasses most of those steps in one place.

They’ve applied the same logic to the options, which currently are spread all over the place but will now be accessible in one window. There are also some great new features coming with the facelift, such as Target Expressions which will allow you to apply several mappings at the load stage, including some mappings that were much more complicated to perform before. Target Expressions are complicated enough to deserve their own tutorial blog, so I’ll dive into those later, but another great feature is an official processing order for data maps, so no more alphabetical processing of data maps, you can get a like mapping to process first and then have explicit maps after if you want. This is a big Quality-Of-Life (QOL) update for admins and really shows Oracle are listening to feedback when it comes to data integrations.

You can also choose to skip the workbench stage, which can massively speed up integrations for day-to-day running if drill-through isn’t required, although you can then switch it back on and re-run an integration to troubleshoot any weirdness. I can see customers using this feature a lot. This is coming in version 19.05 for cloud.

You can start using Data Integration right now in PBCS, although full parity with Data Management doesn’t exist yet. As a result, you can create an integration in Data Integration and then jump into the more familiar Data Management interface to see how it all works in the background, and you can run an integration in either interface with no issues.

Data Export Updates

A recent highlight update for Data Management was the massive upgrade to flat file exports, which has given us several great options. These include:

• Import a file template to create a flat file export
• Include/Exclude the file header
• Sort the data in a file export
• Attributes can be exported into flat files
• Easy reordering of output columns
• Pivot specific dimensions into the columns
• Choosing the accumulate (aggregate) data or not based on preference
• Changing the data file parameter (we love a | in our csv files)

John Goodwin has blogged about it, so you know it’s great (check that out here) and there’s no point covering the same ground, so let’s move on!

Future Integrations to follow NetSuite to PBCS model

For those that haven’t read me gushing about the NetSuite integration, check my tutorial out here, but Mike Casey has confirmed that future direct integrations are planned to follow the NetSuite model. This means that they will support designing a query in the source system, pulling that specific query over to Data Management and mapping that query individually from other queries. This is AMAZING for importing metadata vs importing data which usually need completely different queries, all in the cloud, all on demand, all SIMPLY DELIGHTFUL

On-Premises Agent

The new on-premises agent will allow direct connections to on-premises systems, with a variety of configurable options to ensure all system admins can get on board with it. It will offer synchronous mode which is tougher to set up and requires more IT involvement but is constantly listening for requests, or asynchronous mode which can be run on a schedule and will execute any queued requests when it’s run on the server-side. Asynchronous doesn’t need a network port or any additional network infrastructure than EPM Automate needs now, so it will be less of a headache to set up.

Direct connections to on-prem databases from the cloud have been a massive request for a long time and it’s going to be a game-changer when this delightful tool comes out. It sounds seriously powerful!

Other Roadmap

Below is the roadmap which was shown at KScope19, under the usual Oracle Safe Harbour statement, so this represents their planned direction but no official commitment to release this stuff. Regardless, there’s plenty to keep guys like me excited and busy in here!

There was so much new and exciting content in the Data Management roadmap, that I haven’t even got around to discussing the direct HCM Cloud integration, including write-back, or the fact that data integration has been upgraded to allow a 5 million row limit!

Expect plenty of blogs giving blow-by-blow walkthroughs of the above once they’re released, but until next time, adios!

Mike