Data API – Advanced Integration

• Custom queries: You define a JSON query specifying the product (via Version ID) and optionally which dimensions and measures to include, any filters on dimension values, and currency conversions. If dimensions/measures lists are left empty, the API will default to all (so you can also use it to fetch full data, but usually Basic Integration is simpler for that).
• Asynchronous export processing: When you request a custom export, the API will respond immediately with an exportId and status, but the data file is not yet ready. The export is processed in the background. You will need to poll the API to check when it’s done.
• On-demand usage: You call the API only when you need specific slices of data. You can run these exports ad hoc or on a schedule, but unlike Basic exports, they are triggered by you (the client) rather than pre-generated.

1. Discover Available Products
   Before making a custom query, determine the Version ID of the product you want. Use GET /products to list all product versions you have access to. Each entry has a versionId (e.g., "123_WW_HCP_FCST_2024Q1" for a forecast dataset) and other info. Pick the relevant versionId for the data you need. (This step can be done once and cached if product versions don’t change often, or repeated if you’re unsure when a new version was released.)
2. (Optional) Get Metadata for Query Construction
   If you plan to filter or limit dimensions/measures, you may want to know what dimensions, measures, and values are available for that product:
   • To get a list of dimension names call: GET /products/{versionId}/dimensions
   • To get a list of measures (metrics): GET /products/{versionId}/measures
   • If you need the list of allowed values for a particular dimension (for example, all valid country codes, all years, etc.) call GET /products/{versionId}/dimensions//values
   • To see the currency options if applicable call: GET /products/{versionId}/currencies

   These endpoints help you build a valid query. For instance, you might discover that dimensions include “Country” or “Company Size”, and you can decide to filter one of them.

3. Build the Query JSON
   Create a JSON object with your desired query parameters. The structure should follow the schema:

   {
     "query": {
       "dimensions": [
         /* list of dimension names to include, or empty for all */
       ],
       "measures": [
         /* list of measure names to include, or empty for all */
       ],
       "filter": {
         /* optional filters on time dimensions, e.g. "Year": ["2024", "2025"] */
       },
       "currencies": [
         /* optional list of currency names (if not USD) */
       ]
     },
     "versionId": ""
   }

   For example, to request a specific subset: you might include only ["Region", "Product"] as dimensions, ["Units"] as measure, no filter (empty {}), and no alternate currency (empty array). This would yield a minimal export of just Units by Region and Product for the chosen product/version. Another scenario: you could provide a filter like "Year": ["2025"] to get data only for the year 2025. Leaving "dimensions" or "measures" empty means “include all”, which can be useful if you only want to apply a filter or currency change but not drop any fields.

4. Submit the Export Request
   Make a POST request to /exports with your JSON query in the body. Don’t forget the headers (x-api-key, Content-Type: application/json, Accept: application/json). If the request is well-formed, the API will respond with an export object containing at least an exportId and a status. Initially, the status will likely be "EXPORT_RUNNING" (or similar) indicating the export job is in progress. For example:

   {
   "exportId": "5678",
   "versionId": "123_WW_HCP_FCST_2024Q1",
   "dataset": "Hardcopy Peripherals Tracker - Forecast",
   "datasetId": 157,
   "exportedAt": "2024-08-10 12:00:00",
   "status": "EXPORT_RUNNING",
   "pregenerated": "false",
   "files": []
   }
   

   Here, pregenerated: "false" indicates this is a custom export (not a pre-created one). The files list is empty at first since no output is ready yet.

5. Poll for Completion
   The export is processed asynchronously. You need to check when it finishes. Use the GET /exports/{exportId} endpoint to retrieve the status of your specific export job. A best practice is to wait a few seconds between polls to avoid hitting rate limits (for example, poll every 5 or 10 seconds). The response structure is the same as the initial one, but once done, the status will change to "DATA_AVAILABLE" (or a similar final state) and the files array will be populated with the file info and URLs for download. If the export failed for some reason, you might see a status like "EXPORT_ERROR" (and possibly an error message in an errors field).
   Alternatively, you can call GET /exports (the list of all exports) and look for your exportId in the array – it will be included among other exports with its updated status. This is useful if you have multiple exports running and want to check all at once.
6. Download the Resulting File(s)
   Once your export’s status is Data Available, you can download the output file(s). The procedure is the same as in Basic Integration – use the file url provided in the export object, or call GET /exports/{exportId}/files/{fileId} for each file. Custom exports usually produce a single file unless the dataset inherently splits (e.g., by segment). Download the .csv.gz and extract it to get your filtered data.
7. Clean Up (Optional)
   The exports you trigger via Advanced Integration will remain listed by the API. There isn’t a strict need to delete them (indeed, the API does not provide a delete endpoint for exports; they expire or get cleaned up by IDC automatically over time). Just be aware that if you call /exports you will see both pregenerated (Basic) and your custom (Advanced) export entries. You might use the pregenerated flag or a naming convention to distinguish them in your processing.

Suppose you only need the last two years of data from a tracker to embed in a visualization. Instead of downloading the full historical dataset (maybe 10+ years) via Basic integration, you can use Advanced integration: find the product’s Version ID, then submit a query with a filter for Year = [2024, 2025]. The API will generate a smaller CSV just containing those years. You poll until it’s ready, then download that file. This might save bandwidth and processing time on your end.

• Processing Time: Large exports can take time to generate (possibly minutes for very large datasets). The API is built to handle quite large data sets, but be prepared to poll for completion. The initial request (POST) will return quickly, but the export itself might still be running. A status of “EXPORT_RUNNING” means just wait longer. Most exports finish in a reasonable time, and you can parallelize requests if needed (but see rate limits below).
• Concurrency and Rate Limits: IDC imposes rate limits to ensure fairness and stability. Notably,
  • Export registration (advanced export) is limited to 10 requests per minute. So don’t script a hundred export requests all at once – queue them or space them out.
  • Other GET requests have a higher limit 40 per minute
  • An overall limit 1 call per second.
  • If you exceed these, you’ll get HTTP 429 Too Many Requests. Also, try to avoid polling too rapidly for status (e.g., hitting the status endpoint dozens of times per second is unnecessary).
• Export Expiration: The files generated by an export might only be available for a certain period (often a few days) on the server. In practice, since Basic exports are refreshed with each release and custom exports are meant for immediate use, you should download them soon after they’re ready. Don’t rely on an export you ran weeks ago still being downloadable – if you need it again, you may re-trigger it or keep your own copy.
• Using Both Basic and Advanced: You are not limited to one mode or the other. You can mix them: use Basic integration to routinely grab all data, and occasionally use Advanced integration for special queries or incremental updates. They use the same underlying endpoints (the difference is primarily which endpoints you call and how you use them).
• Data Consistency: If you trigger an advanced export for a product right when a new version is being released by IDC, note that a Basic export might supersede it. Generally, it’s best to know which version you want (via Version ID). The Version IDs change with each official release. Advanced exports always tie to a specific Version ID, so you won’t accidentally mix data from different releases in one export.