SFTP – Trackers, Spending Guides, CPE, and Wallet Products

Once you have your SFTP credentials from IDC, connecting to the SFTP server is straightforward using any standard SFTP client or library.

• Server Hostname: sftp.idc.com. (This is the address of IDC’s SFTP server.)
• Port: 22 (default SFTP/SSH port).
• Username & Key: Use the username provided by IDC, and authenticate with the SSH private key file given to you. IDC generally uses key-based auth, so you may not have a typical password. Instead, your key itself is your credential, possibly secured with a passphrase (IDC will supply that if applicable). Keep the private key file safe and do not share it.
• Client Software: You can use GUI tools like FileZilla, WinSCP, or command-line tools (sftp, scp, rsync) or programming libraries (Paramiko for Python, SSH.NET for .NET, JSch for Java, etc.). In your client, when configuring the connection:
  • Enter the host (sftp.idc.com) and port (22).
  • Enter the username.
  • Instead of a password, specify the path to your private key file. In some clients you might need to convert the key to the appropriate format (e.g., WinSCP supports .ppk keys for PuTTY; IDC can provide a .ppk or you can convert an OpenSSH key to .ppk).
  • If the key has a passphrase, enter it when prompted.
• Connection Test: After setting up, connect. If everything is correct, you should establish an SFTP session and be able to list directories on the server. If you cannot connect:
  • Double-check you’re using SFTP (not FTP) and the correct port.
  • Verify the key is correctly loaded and the passphrase (if any) is right.
  • Ensure your IP isn’t blocked by firewall – you might need to allow outbound SFTP or ask IDC if they require your IP for whitelisting (usually not, but good to check if connection issues persist).
  • If using a corporate network, ensure proxies or security appliances allow SFTP.

Once connected, you are ready to navigate the directories and download files.

IDC’s data SFTP server organizes data in a logical directory structure. Upon logging in, you’ll typically see one or a few top-level folders. Key ones include:

• Data and Analytics/ – This directory contains data files for all IDC data products (the same products you’d access via the Query Tool API – Trackers and Spending Guides or via Knowledge Platform – Wallet products). Under this, you will see subfolders or files grouped by product lines.
• Under Data and Analytics folder, there may be a subfolder called Query Tool/ which includes the bulk of subscription-based data product exports. (Think of this as analogous to the Query Tool data exports in the API.)
• Inside Query Tool/ folder, you might find files for each product. In some setups, IDC might further categorize by program or geography, but often the file names themselves carry the identification.
• Knowledge Platform/ folder– If you have subscriptions to IDC’s Knowledge Platform data products, those files may reside here. Knowledge Platform folder will contain information from the following products:
  • Wallet: Spend by Tech Provider
  • Wallet: Spend by Market
  • Channel Partner Ecosystem

To access documents and PDF files please refer to SFTP Research Documents or Research Document REST API sections based on your preference. This includes documents that are part of the data subscription (for example a taxonomy document for the PCD Tracker).

IDC filenames are quite descriptive. A typical data file name might look like:

[Geography]_[ProductName]_[DataType]_[Period]_delivered-[Date]_[Part]_[Client].csv.gz

For example:

• WW_WLAN_FCST_Forecast_2024_delivered-18-Dec-2024_1_ClientX.csv.gz:
  • WW (worldwide scope),
  • WLAN_FCST (a code for the product, here WLAN Forecast),
  • Forecast_2024 (indicates dataset type and year/period),
  • delivered-18-Dec-2024 (the date IDC delivered this file),
  • _1 (if multiple parts, this is part 1),
  • Client (some files may include an identifier for the client or subscription).
• Another example from the Query Tool exports:
  IDC_Quarterly_Workstation_Tracker_Final_Historical_2023Q3_delivered-08-Nov-2023_1.csv.gz:
  • It includes IDC, the product name “Quarterly Workstation Tracker”, “Final Historical 2023Q3” indicating the content and period, the delivery timestamp, and part number.

[ClientName]_[Product]_FDE_[YYYYMMDDhhmmss].csv (FDE might stand for Full Data Export). For example: Wallet_data_timestamp_ClientShortName1.zip. This format includes a timestamp to the second.

It’s not critical to parse all parts of the filename, but they help identify the content. The “delivered-<date>” part is useful to see when the file was released by IDC, and the suffix number _1, _2, ... tells you if there are multiple files for the same release (common if the dataset is split by segments or simply too large, so IDC chunks it).

Note: The exact files you see will depend on your subscriptions. If you only subscribe to certain products, you will only see those. If a product gets updated, a new file with a new date will appear. Old files might remain for a time or be moved/archived – confirm with IDC, but generally you should always pick up the newest “delivered-” file for each product.

Different IDC products have different update frequencies:

• Some trackers are quarterly, some semiannual, some monthly, etc. Each time a product is refreshed by IDC’s research team, a new file will appear on SFTP (and correspondingly, via API).
• The Wallet products are updated every 6 months
  • The old files are removed
  • New files appear in the product folder as the data is updated in the Knowledge Platform
• To know when updates occur for a given product, you can refer to the product’s Delivery Schedule on IDC’s portal (if available) – either in the MyIDC Dashboard or under IDC.com – myIDC – Data . Also, the Info page in Query Tool might list that product’s planned release calendar (e.g., Q1 release in March, Q2 in June, etc.).
• If you don’t have access to that info, you can ask IDC or rely on notification emails.

Two main approaches:

• Polling: As mentioned earlier, you can script a routine to list files in your SFTP directory at intervals (say, daily) and detect new files that were not present before. This can be done with an SFTP ls command or using file metadata (like date or name). Once a new file is found, download it and process it. This approach ensures you catch updates even if you miss or don’t read the email notifications.
• Notifications: IDC generally sends out an email to clients when a new data release is delivered (especially for major products). The email typically outlines what was updated. Make sure IDC has the correct contacts (emails) for these notices. As per internal notes, you should “share emails of people from your company who should be notified when changes occur” so that product updates or structural changes are communicated. Relying on email alone isn’t automated, but it’s a helpful cue for your team to then trigger the download process.
  To setup or to adjust a list of email addresses to be notified refer to the Staying up to Date with Recent Changes.

The SFTP may contain only the latest version of each product or possibly a few historical files. IDC does not guarantee keeping old files indefinitely on SFTP – usually, they focus on the latest release (since clients only have access to latest data by contract). If you see multiple releases’ files, that might be temporary. It’s wise to archive the files you download on your end if you need to keep historical versions.

If you are manually handling files:

• On Windows, use a tool like 7-Zip or WinRAR to open .gz files.
• On MacOS, you can just double-click or use Archive Utility to extract.
• On Linux, use gunzip or gzip -d in the shell (most distros have it pre-installed).
• Programmatically, many languages have libraries to handle gzip and CSV extraction.

Remember extracting large files requires sufficient disk space; a 1 GB .gz might become 10 GB CSV when extracted – ensure you have space and possibly clean up files after processing to save storage.

To effectively use SFTP integration, automation and notifications go hand-in-hand:

• Automation Scripts: IDC provided a sample Python “mirroring” script in the documentation which essentially connects to the SFTP and downloads everything to a local directory (skipping files you already have). You can adapt such a script to your needs. The pseudocode of the process:
  • Connect to sftp.idc.com using Paramiko or another SSH library with your credentials.
  • Recursively list directories and files from the remote root (or the specific folder you want).
  • For each file, if it doesn’t exist locally or is newer on the server, download it. Preserve the directory structure locally.
  • Close the connection when done.

  The sample script even suggests scheduling it (e.g., via cron or Windows Task Scheduler) to run every night, keeping your local data repository in sync with IDC’s server. This is a robust solution to ensure you don’t miss anything.

• Notification Emails: As mentioned, IDC emails can alert you to new data or changes. For example, if IDC decides to add a new column to a dataset, they will email ahead of the release. Or they might announce that the next quarter’s data is now available on SFTP. It’s good practice to have those emails go to a distribution list or at least to the person monitoring the integration. The email’s content can be a prompt to verify that your automated process picked up the new files.
• Monitoring & Logging: In your automation, implement logging. Keep logs of when you last checked, what files were downloaded, and any errors encountered. This helps in auditing and troubleshooting if something was missed.
• Client Support: If an automated download fails or a file is corrupted, you can always re-download or contact IDC’s support. The SFTP is quite stable, but network issues can happen. Ensure your script can handle transient network failures (e.g., retry downloading a file if the first attempt fails).
• Passphrase Automation: If your private key has a passphrase, using it in an unattended script may require handling the passphrase input. One way is to use key agents or to securely store the passphrase in an environment variable or config that your script can read. (Some clients choose to use a key without passphrase for automation behind firewalls but weigh the security implications – consult IDC if needed.)

For an example how to automate mirroring of the SFTP server please refer to the SFTP Automation section.

In summary, combining an automated script with IDC’s notifications will give you a robust pipeline: IDC releases new data (and maybe emails you), your script (scheduled) picks it up, and your internal systems get updated data without manual intervention.

Here are some common issues and tips for SFTP integration:

• Cannot Connect / Authentication Failed: Double-check the host (sftp.idc.com), port (22), and that you’re using SFTP (SSH File Transfer), not FTP. Ensure your username is correct and that you’ve configured the private key properly in your client. If using a GUI client, make sure to select “SFTP” as protocol and point to the key file. If you get a prompt for password, it might mean the key wasn’t used – ensure the client knows about the key. If the key has a passphrase, you must provide it. If still failing, the key or username might not be activated or could be wrong; contact IDC to verify credentials.
• Permission Denied / No Access to Files: If you can login but not see any files or get permission errors, it could be that your user is chrooted to a certain directory. You should start at a root where you can see at least Data and Analytics folder. If you see nothing, confirm with IDC that your account is pointed to the right location and that files have been published. It could also mean no files are available yet (for example, if your subscription hasn’t had a release since SFTP was set up).
• Partial or Corrupt Downloads: If a file download was interrupted (network glitch), you might have a truncated file. Enable resume support if using a tool that supports it (like rsync or certain SFTP clients). Or redownload the file entirely. Always verify file sizes – IDC can provide expected sizes or you can compare to see if you got a full file. Gzip files have checksums; if a .gz is corrupted, gunzip will typically complain. Just re-fetch in that case.
• File Format Issues: If you open a CSV and it looks garbled or the delimiter is wrong: IDC uses commas for CSV and a period for decimal, etc. Ensure your CSV reader is configured correctly. Also check encoding (should be UTF-8). If using Excel to open, large files might be too big; consider using a database or CSV processing tool for large data.
• No New Files / Missing Updates: If you expected a new data release but don’t see a new file on SFTP, a few things to do:
  • Check if the release was possibly delayed or renamed. Cross-check any IDC announcement.
  • Make sure your script is not filtering incorrectly. For instance, if it only checks file names containing a certain year/quarter, a new format might be missed.
  • It could also be a permission or setup issue – perhaps the file is in a different folder (e.g., under Knowledge Platform rather than Query Tool). Search the directory structure for keywords.
  • As a fallback, contact IDC support to ask if the file has been delivered to SFTP yet.
• Support Channel: If you run into issues you cannot solve (connection problems, file issues, etc.), IDC’s support is available. You can email idc_support@idc.com for assistance, describing your problem (be sure to include your company name, and any relevant details like the SFTP username or product name). IDC’s team can help troubleshoot account issues or resend credentials if needed.

By addressing these common issues, you can ensure a smooth operation of your SFTP-based data feed. The SFTP approach is generally low-maintenance once set up – it’s been successfully used by clients to automate data ingestion with minimal ongoing effort.