Hi
We're having trouble with getting large volumes of data into some of our cubes.
1 of our cubes (Listing) takes around 01h45 to load about 20 million records.
Another cube (Vendor) takes around 00h30 to load 36 million records.
Most of the time, in both scenarios, is taken up with SID generation ... is there a way to speed this up?
In the Listing scenario, the load to PSA has been split into about 15 infoPackages. The total time taken to run all of those infopackages (in 3 parallel streams) is about 15 minutes.
In the Vendor scenario, a single infopackage is used to load to the PSA, and takes around 2 hours to load into the PSA
The following picture shows one of the DTP requests for the Listing scenario:
For this example 3:16 for datapackage 8, of which 2:29 were taken up for the SID step.
Not sure why this takes so long, and if there's anything to do to speed it up.
A similar screenshot of the Vendor DTP:
That looks like similar performance to the listing DTP. However, when I look at the later requests in the Vendor DTP, they suddenly go a whole lot quicker towards the end:
I guess my questions are:
1. How do I speed up the SID generation
2. Why does SID generation get quicker when more & more records are processed, and why would that not apply in the Listing scenario when there are multiple PSA requests to load
3. Why would an infopackage take so much longer (36 million / 1.5 hours) when running as single infopackage compared to smaller packages (15 minutes in parallel / 20 million records) --> This one I can sort of understand, that the multiple smaller packets are better, but it then seems to harm the DTP performance.
Cheers,
Andrew