# SDMX-HD (Health Domain) v1.0 documentation
Version 2009.04.WHO.03

## Getting Started

## Setup the environment

1. Create a folder called "SDMX" as your working folder in which all SDMX-HD files will be placed.
2. Unzip the "SDMX-HD.v1.0.zip" file into the "SDMX" folder. You should now see the following folder structure:

   - SDMX-HD.v1.0 MyWork - This folder is your working area in which you will create the SDMX-HD files to exchange with other systems.
   - SDMX-HD.v1.0 sample1 - This folder contains an example of SDMX-HD files used to exchange indicator definitions.
   - SDMX-HD.v1.0 sample2 - This folder contains an example of SDMX-HD files used to exchange indicator definitions and data.
   - SDMX-HD.v1.0 Testbed - This folder is created by the "CreateTESTBED.bat" file which copies SDMX-HD.v1.0 sample1 or 2 + SDMX-HD-v1.0 folders into a unique folder to resolve cross-references between files (*.xsd and *.xml SDMX files referrenced as external)
   - SDMX-HD-v1.0 - This folder contains the SDMX-HD v1.0 common and custom artifacts used as building blocks for SDMX-HD messages. .. ..

## Exchanging indicator definitions

Setting up the MyWork folder

1. Copy the sub folders and files of the "SDMX-HD-v1.0" folder to the "SDMX-HD.v1.0 MyWork" folder so you have a folder structure like this:

   The following files are not needed in your "SDMX-HD.v1.0 MyWork" folder as they are not required in the SDMX-HD message (delete them if you wish):
   - GettingStarted.html
   - index.html
   - foldercontents.xml
   - foldercontents.xsl

2. Rename "TemplateDSD.xml" file to DSD.xml
3. Rename "TemplateMSD.xml" file to MSD.xml
4. Rename "TemplateGMS.xml" file to GMS.xml
5. Rename "TemplateCDS.xml" file to CDS.xml (only needed if data is transmitted)

Create your DSD.xml file | Codelists

1. Open the DSD.xml file with the XML editor of your choice.
2. Open the "COMMON/SDMX-HD/v1.0/organisationschemes/ORGS_COMMON+SDMX-HD+1.0.xml" file and check your organization is defined in the list. Add it if it is not!
3. Open the "additional/CL_INDICATOR.xml" codelist and populate it with the name of the indicators to exchange
4. Update any common codelist that have isFinal=false with any items you need (i.e. that are referrenced from another file)
5. Create any "custom" codelist you need in the "CUSTOM/WHO/v1.0/codelists" folder.
6. Add these new custom codelist to the DSD file e.g here is custom Age Group codelist:
   <structure:CodeList id="CL_AGROUP" agencyID="WHO" version="1.0" isFinal="false" urn="urn:sdmx:org.sdmx.infomodel.codelist=WHO:CL_AGROUP" isExternalReference="true" uri="./CUSTOM/WHO/v1.0/codelists/CL_AGROUP+WHO+1.0.xml"><structure:Name xml:lang="en">Age Group</structure:Name></structure:CodeList>
   Note that the agencyID needs to be changed to your agency, matching the one in the OrganisationScheme file.

| Hierachical Codelists

1. In the VOLATILE_CODELIST_HIERARCHY HierachicalCodeList use the CodelistRef element to create aliases to each codelist you wish to create a structure for.
2. For each hierachy in the HierachicalCodeList create the structures you need using the CodeRef element which represents a node in the structure. Each CodeRef element has a CodelistAliasRef to reference the alias you created and a CodeID which references the codelist code value the alias points to.

| Concept Schemes

1. If you need to create a new Concept that is NOT already defined in the following two files:
   1. COMMON\SDMX-HD\v1.0\conceptschemes\COMMON+SDMX-HD+1.0.xml
   2. COMMON\SDMX-HD\v1.0\conceptschemes\COMMON_GEN+SDMX-HD+1.0.xml
      1. Create a new ConceptScheme element to hold the list of your custom concepts
      2. Add the concepts into the ConceptScheme. Here is an example of a concept called TOPIC, added to the list of custom WHO concepts:

             <structure:ConceptScheme id="CS_CUSTOM" agencyID="WHO" version="1.0" isFinal="false"
             urn="urn:sdmx:org.sdmx.infomodel.conceptscheme.Concept=WHO:CS_CUSTOM">
               <structure:Name xml:lang="en">WHO Custom Concepts</structure:Name>
               <structure:Description xml:lang="en">This Concept Scheme contains
               un-harmonized data elements.</structure:Description>
               <structure:Concept id="TOPIC" agencyID="WHO" version="1.0"
               urn="urn:sdmx:org.sdmx.infomodel.conceptscheme.Concept=WHO:CS_CUSTOM_WHO[1.0].TOPIC" >
             	<structure:Name xml:lang="en">Health Care Topic.</structure:Name>
             	<structure:Description xml:lang="en">Health Care Topic.</structure:Description>
             	<structure:TextFormat textType="String" />
               </structure:Concept>
               ...
             </structure:ConceptScheme>

| Key Family Component Dimensions

1. Many of the common dimensions you need have been added to the TemplateDSD.xml and commented out. Simply uncomment the dimensions you need.
2. Should you need to add a new custom dimension, first check the list of existing custom dimensions (meant to be overriden by agency) and uncomment the one you need. Or add a new one if not listed.
3. Check the following attributes are correct:

   1. conceptSchemeRef
   2. conceptVersion
   3. conceptSchemeAgency

   These attributes need to match the ones specified for your concept

   1. codelist
   2. codelistVersion
   3. codelistAgency

   These attributes need to match the ones specified for your codelist

| Groups
| No changes needed for indicator definitions
| Attributes
| No changes needed for indicator definitions

Create your MSD.xml file | Codelists

1. Open the MSD.xml file with the XML editor of your choice.
2. Update any codelists that have isFinal=false with additional items that you need.
3. Create any "custom" codelist you need in the "CUSTOM/WHO/v1.0/codelists" folder.
4. Add these new custom codelist to the MSD file e.g here is custom NEWLIST codelist:
   <structure:CodeList id="CL_NEWLIST" agencyID="WHO" version="1.0" isFinal="false" urn="urn:sdmx:org.sdmx.infomodel.codelist=WHO:CL_NEWLIST" isExternalReference="true" uri="./CUSTOM/WHO/v1.0/codelists/CL_NEWLIST+WHO+1.0.xml"><structure:Name xml:lang="en">NewList</structure:Name></structure:CodeList>

| Concept Schemes

1. If you need to create a new concept that is NOT already defined in the following two files:
   1. COMMON\SDMX-HD\v1.0\conceptschemes\COMMON+MSD+SDMX-HD+1.0.xml
   2. COMMON\SDMX-HD\v1.0\conceptschemes\COMMON_GEN+SDMX-HD+1.0.xml
      1. Create a new ConceptScheme element to hold the list of your custom concepts
      2. Add the concepts into the ConceptScheme. Here is an example of a concept called TEST, added to the list of custom WHO concepts:

             <structure:ConceptScheme id="CS_CUSTOM_MSD_WHO" agencyID="WHO" version="1.0" isFinal="true"
              urn="urn:sdmx:org.sdmx.infomodel.conceptscheme.Concept=WHO:CS_CUSTOM_MSD_WHO">
               <structure:Name xml:lang="en">WHO Custom Concepts</structure:Name>
               <structure:Description xml:lang="en">This Concept Scheme contains un-harmonized
                metadata elements.</structure:Description>
               <structure:Concept id="TEST" agencyID="WHO" version="1.0"
               urn="urn:sdmx:org.sdmx.infomodel.conceptscheme.Concept=WHO:CS_CUSTOM_MSD_WHO[1.0].TEST" >
             	<structure:Name xml:lang="en">TEST.</structure:Name>
             	<structure:Description xml:lang="en">TEST</structure:Description>
             	<structure:TextFormat textType="String" />
               </structure:Concept>
               ...
             </structure:ConceptScheme>

| IdentifierComponent
| No changes needed for indicator definitions
| ReportStructure Metadata Attributes

For each metadata attribute linked to the indicator definition (i.e. to the CL_INDICATOR code list), you must create a new MetadataAttribute describing this metadata attribute. Here is an example for Preferred Source (PSOURCE):

    <structure:MetadataAttribute conceptRef="PSOURCE" usageStatus="Conditional"
    	conceptSchemeAgency="SDMX-HD" conceptVersion="1.0" conceptSchemeRef="CS_COMMON_MSD"
    	representationScheme="CL_PSOURCE" representationSchemeAgency="SDMX-HD">
    	<structure:TextFormat textType="String" />
    </structure:MetadataAttribute>

Create your GMS.xml file | AttributeValueSet

1. Open the GMS.xml file with the XML editor of your choice.
2. For each indicator metadata row, repeat the following steps:
   1. Locate the AttributeValueSet element and set the ComponentValue value attribute to the IdentifierComponent ID specified in the MSD.
   2. Also change the ? to the codelist value this attributeSets metadata will apply to. So metadata for the first indicator (represented by code 0 in CL_INDICATOR code list), would look like this:

          <gen:TargetValues>
          	<gen:ComponentValue component="IC_INDICATOR" object="Codelist">0</gen:ComponentValue>
          </gen:TargetValues>

   3. Repeat this process for each concept you wish to retain metadata for:
      1. Set the ReportedAttribute conceptID to the concept for which you wish to create metadata for e.g. _CODE, DTYPE etc.
      2. Now set the Value element of the ReportedAttribute to the codelist value for your metadata or the free text value if no codelist exists. String values should be encapsulated into CData tags to avoid XML errors due to string content.

top

## Exchanging indicator definitions and data

Setting up the MyWork folder

1. Identical to "Exchanging indicator definitions" section above

To exchange indicator data, you can start with the same DSD.xml, MSD.xml and GMS.xml that you use to exchange indicator definitions. It is normal that these three files are almost identical as the data structure defining indicators does not change regardless of whether your SDMX-HD message contains indicator data or not. As a matter of fact, only a few changes are required to the DSD.xml. Most of the work involved with including indicator data in the SDMX-HD message consists of constructing the CDS.xml and its accompanying CDS.xsd.

Create your DSD.xml file | Codelists
| Identical to "Exchanging indicator definitions" section above
| Hierachical Codelists
| Identical to "Exchanging indicator definitions" section above
| Compact Schemes
| Identical to "Exchanging indicator definitions" section above
| Key Family Component Dimensions
| Identical to "Exchanging indicator definitions" section above
| Groups
| Uncomment the PROGRAMS group element as indicator data recorded in the CDS.xml will be grouped by PROGRAMS.
| Attributes

1. To create new attributes just add the Attribute element and complete the attributes as follows:
   1. conceptSchemeRef - References the ConceptsScheme that contains the concept you wish to use
   2. conceptRef - References the concept for this attribute. The concept ID also become the attribute name used in the CDS.xml file
   3. conceptVersion - The Version of the concept
   4. conceptSchemeAgency - The AgencyID of the concept
   5. attachmentLevel - At which level do you want to attached this attribute : Observation, Series, Group or DataSet
   6. assignmentStatus - "Mandatory" or "Conditional"

Create your MSD.xml file | Codelists
| Identical to "Exchanging indicator definitions" section above
| Compact Schemes
| Identical to "Exchanging indicator definitions" section above
| IdentifierComponent
| Identical to "Exchanging indicator definitions" section above
| ReportStructure Metadata Attributes
| Identical to "Exchanging indicator definitions" section above

Create your GMS.xml file | AttributeValueSet
| Identical to "Exchanging indicator definitions" section above

Create your CDS.xsd file | CDS Schema Validation File

1. The details of this process are listed in the SDMX-HD standard documentation in section C2 "XSD schema validation".

Create your CDS.xml file | DataSet

1. The dimensions plus all the attributes listed in the DSD attached to the DataSet must be added to the DataSet elemnent. Add the Mandatory attributes to this level.

| Group

1. PROGRAMS - specify the PROG attribute and set its value to the CL_PROGRAM codelist value to represent the PROGRAM by which you want Series (indicators) to be grouped by.

| Series

1. Each Series represents an indicator. As such, it must have the following attributes defined:
   - PROG - To identify which PROGRAMS group this indicator belongs to
   - INDICATOR - To identify which INDICATOR this Series represent
   - DISEASE - To identify which DISEASE this indicator belongs to
   - GEOGRAPHIC_PLACE_NAME - To identify which Geographical place Observations represent
   - ORGANIZATION - To identify which Organization provided the Observations
   - VALUE_TYPE - To identify the type of observation values for this indicator : Values or Targets
   - PERIODICITY - How often this data should be gathered
   - UNIT - The units of measure for the observation values
   - REPEATS - How many observation values can we have for this indicator. 0 = unlimited.
2. You then need to add the attributes defining disaggregations specified for this indicator (if any). For exmaple
   - GENDER - The DSD has GENDER specified as dimension, so a value of "_ALL" means all gender
   - AGROUP - The DSD has AGROUP specified as dimension, so a value of "5" means "15+"
   - GLOCATION - The DSD has GLOCATION specified as dimension, so a value of "3" means "Urban"

| Observation

1. Each Observation represents an indicator value and so must have the following attributes set:
   - OBS_VALUE - The indicator data point.
   - TIME_PERIOD - The time period codelist value that defines when this observation value is for.
   - DATE_COLLECT - The date when the data was collected

top

## Package the SDMX-HD message as a ZIP archive

1. The folder structure described above should be respected when packaging files as a ZIP archive. All files described above, including *.xsd and all *.xml files externally referenced by your DSD.xml and MSD.xml must be included in the package.
2. In effect, just zip the entire "SDMX-HD.v1.0 MyWork" folder contents

   It is suggested to adopt helpful *.zip naming conventions, such as a file name built like this:
   SYSTEM_ORGANIZATION_GEOPLACE_YYYYMMDDHHMMSS.zip, where
   - SYSTEM: is the system (or application) creating the SDMX-HD message
   - ORGANIZATION: is the organization reporting the data
   - GEOPLACE: is the geographic place for which data is reported
   - YYYYMMDDHHMMSS represent the file creation timestamp with YYYY= 4-digit year; MM= 2-digit month, etc...

top

## Avoiding common mistakes

First and foremost, PLEASE VALIDATE YOUR XML FILES PRIOR TO SENDING THEM TO ANOTHER ORGANIZATION. Failure to validate your XML files will likely results in errors that will prevent a successful data exchange as the recipient organization will not be able to import your file. "Stylus Studio 6 XML Professional Edition" and metadata technology SDMX Offline Tools have been successfully used by WHO to validate XML documents against the published schemas.
The following bullets points list common mistakes that should be avoided in order to produce compliant XML files:

- SDMX-HD zip messages must be self-contained: all concepts, codelists and hierarchies that your data or metadata references must be included in the zip file, either as inline information within the DSD.xml or MSD.xml files themselves, or as external references pointing to another file contained in the zip package. E.g. if you use a standard SDMX-HD code list:
  <structure:CodeList id="CL_GENDER" isExternalReference="true" uri="./COMMON/SDMX-HD/v1.0/codelists/CL_GENDER+SDMX-HD+1.0.xml"/>
  this requires the CL_GENDER+SDMX-HD+1.0.xml file to be included in the zip package, under the appropriate directory structure, i.e. in the COMMON/SDMX-HD/v1.0/codelists subfolder
- Use the correct namespaces and always include all *.xsd files with your message: message: The DSD.xml, MSD.xml and CMS.xml documents use standard SDMX-ML namespaces (see sample files), and in addition, the CDS.xml file must use the following WHO SDMX-HD namespace pointing to your own customized version of the CDS.xsd schema. The xsi:schemaLocation must use relative paths and the CDS.xsd:
  <CompactData xmlns:ns="urn:sdmx:org.sdmx.infomodel.keyfamily.KeyFamily=WHO:SDMX-HD:compact" xsi:schemaLocation="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/message SDMXMessage.xsd urn:sdmx:org.sdmx.infomodel.keyfamily.KeyFamily=WHO:SDMX-HD:compact CDS.xsd"
- Many SDMX-HD v1.0 references files cannot be modified: the SDMX-HD v1.0 standard provides final version of many files that cannot be modified (e.g. conceptschemes, codelists, etc..). Such files are marked with the isFinal="true" attribute and must be used without any modification. If such file does not cover your needs, create your own data structure (e.g. conceptscheme, codelist, etc..) in replacement and override the built-in SDMX-HD concept or codelist by pointing to your new structure accordingly (see samples for details). However, data and metadata structures marked with the isFinal="false" attribute or without the attribute at all can be freely modified.
- Do not use local date formats: the XSD specification requires date and datetime to comply with a specific schema, in the form of YYYY-MM-DDTHH:MM:SS. Using any other format will result into a validation error.
- Respect tag ordering: many SDMX schemas require the XML tags to appear in a specific order (within xs:sequence XSD declarations). Wrong tag ordering will likely result into an error such as "Element 'XXX' is unexpected according to content model of parent element 'YYY'.
- XML is a case-sensitive format: not only must you respect case for elements and attributes, you should also be consistent in your naming convention. If you use a gender code list defined as <structure:CodeList id ="CL_GENDER"/> you should use the same case for all references to it.
- Avoid empty optional elements: some elements are optional, but if present cannot be empty.