Java Tutorials // Example – Sdmx Data Writer


The following demo will write data to an output file. In this example the output format is SDMX Structure Specific v2.1 (in SDMX v2.0 this was called Compact). The output format can be changed by un-commenting one of the other DataWriter instances. The output files resulting from this demo can be found at the following links:

  Structure Specific 2.1
  Generic 2.1
  HTML (writer engine created for demo purposes)

The key advantage of using the SdmxSource DataWriterEngine interface is that the code calling the interface methods does not need to know about the output format. This decoupling allows users to write out SDMX without having to learn SDMX! It is very simple to add new output formats by just creating a new implementation of the interface, no other code needs to change, this demonstrates the key advantage of a loosely coupled system. Implementations do not need to write SDMX. In the following example there is an implementation that writes out an HTML table (and you will notice all the code id’s are resolved to labels!).

The SdmxSource implementations support all formats and versions of SDMX (except Utility) and multiple datasets are supported. The implementations use streaming technology. By streaming data very little information is held in memory at any one time, therefore there is no limit to the amount of data that can be written.


Demo Project

The following link will download a demo project from which the following example is taken.
DEMO DATA WRITER PROJECT

Adding Data Writer capabilities to your project

The Data Writer Interface lives in the SdmxApi project. There are various implementations in the SdmxDataParser project and the SdmxEdiParser project. The following dependencies are used in the demo project.

<!-- WEAVING -->
<dependency>
  <groupId>org.aspectj</groupId>
  <artifactId>aspectjrt</artifactId>
  <version>1.6.11</version>
</dependency>

<!-- REQUIRED FOR APPLICATION CONTEXT -->
<dependency>
  <groupId>javax.servlet</groupId>
  <artifactId>servlet-api</artifactId>
  <version>2.5</version>
  <scope>provided</scope>
</dependency>

<!-- DATA PARSER -->
<dependency>
  <groupId>org.sdmxsource</groupId>
  <artifactId>SdmxDataParser</artifactId>
  <version>1.5.6.6</version>
</dependency>

The SdmxDataParser dependency has dependencies on the SdmxApi (amongst other) projects, and all dependencies will be automatically download by the IDE and included on the build path of the project.

Creating a Data Writer Engine

A DataWriterEngine expects an OutputStream on creation, this defines where the output is written to. The advantage of using an OutputStream is that the user of the DataWriterEngine can decide exactly where the data is written to. The user may provide a FileOutputStream (if they want the output to go to a file), a ByteArrayOutputStream (output to memory) or any other implementation of OutputStream. Remember to close() the DataWriterEngine on completion, as this causes the Engine to flush all the information and close the stream. A DataWriterEngine expects the first method call to be startDataSet(…,…) this method requires a DataStructureBean.

/**
 * Creates a compact data writer version 2.1 SDMX (aka StructureSpecific).
 * A dataset is started as an APPEND dataset
 */
public void createDataWriter(DataStructureBean dsd, OutputStream out) {
  DataWriterEngine dwe = new CompactDataWriterEngine(SDMX_SCHEMA.VERSION_TWO_POINT_ONE, out);
  DatasetHeaderBean header = new DatasetHeaderBeanImpl("DSREF001", DATASET_ACTION.APPEND, dsRef);
  dwe.startDataset(dsd, header);
}

Obtaining a DataStructureBean

A DataStructureBean defines the data in terms of its dimensionality. It is a structural artefact in SDMX terms and can be obtained in many different ways. Typically the DataStructureBean will either be obtained from a file, from a web service, or programtically created using Java Code. In all cases the means of retrieving a structural element such as a DataStructureBean can be decoupled from the program code by using a SdmxBeanRetrievalManager. This interface defines methods for retrieving SdmxBeans, and the implementation can decide where to get the Bean from. The implemenation can be ‘wired into’ the code by using Spring’s Inversion of Control (IOC). An InMemoryRetrievalManager is a good candidate to use if the structures are coming from a location that can be defined by a URI (such as a file, or a RESTful API).

Writing Data

To write data, simply create a DataWriterEngine implementation and make the appropriate method calls. In the following example the data that is written to the output file is hardcoded in the method call, but this will typically be more dynamically generated. Notice that the DataWriterEngine has the method writeAttributeValue(…,…) which is able to determine the attachment depending on the current context. If the previous method call was writeSeries, then the attribute is a series level attribute, if the previous call was writeObservation then it will be treated as an Observation level attribute. Note that the DataWriterEngines in the following example will not validate the data being written.


/**
 * Writes some data to a file in one of many supported formats
 */
public void writeData() throws Exception {
  //1. Create File
  File f = new File("output.xml");
  f.createNewFile();
  OutputStream out= new FileOutputStream(f);

  //2. Get Data Structure Definition
  MaintainableRefBean ref = new MaintainableRefBeanImpl(null, "PGI", null);
  DataStructureBean dsd = beanRetrievalManager.getDataStructure(ref);

  //3. Get Data Writer & Output to File
  DataWriterEngine writerEngine = getDataWriter(dsd, out);

  writeData(writerEngine, dsd);
}

/**
 * Get an implementation of DataWriterEngine.
 * The implementation will define the output format
 * (commented out code is included to show examples of returning other implementations
 * that will output the data in another format / version)
 */
private DataWriterEngine getDataWriter(DataStructureBean dsd, OutputStream out) {
  return new CompactDataWriterEngine(SDMX_SCHEMA.VERSION_TWO_POINT_ONE, out);
  //return new HTMLDataWriterEngine(dsd, out);
  //return new GenericDataWriterEngine(SDMX_SCHEMA.VERSION_TWO_POINT_ONE, out);
  //return new GenericDataWriterEngine(SDMX_SCHEMA.VERSION_TWO, out);
  //return new EDIDataWriterEngineImpl(out);
}

/**
 * Write some data to the DataWriterEngine.
 * This method does not know or need to know about the output format.
 */
private void writeData(DataWriterEngine dwe, DataStructureBean dsd) {
	DatasetStructureReferenceBean dsRef = new DatasetStructureReferenceBeanImpl(dsd.asReference());
	DatasetHeaderBean headerOne = new DatasetHeaderBeanImpl("DSREF001", DATASET_ACTION.APPEND, dsRef);

	//Start a New Dataset for the Data Structure Definition
	dwe.startDataset(dsd, headerOne);

	dwe.startSeries();
	dwe.writeSeriesKeyValue("REF_AREA", "213");
	dwe.writeSeriesKeyValue("PGI_CONCEPT", "003");
	dwe.writeSeriesKeyValue("DATASOURCE", "BOP");
	dwe.writeSeriesKeyValue("UNITOFMEASURE", "N_M");
	dwe.writeSeriesKeyValue("FREQ", "A");

	//Attribute written out as a Series Level Attribute
	dwe.writeAttributeValue("TIME_FORMAT", "P1Y");

	//Write Observation using ISO8601 Standard Date/Time format
	dwe.writeObservation("2009", "182.2");
	//Attribtue written as an Observation Level Attribute
	dwe.writeAttributeValue("OBS_STATUS", "A");

	//Write Observation using Date Object and telling the writer which format to output in
	Calendar cal = Calendar.getInstance();
	cal.set(2010, 01, 01);
	dwe.writeObservation(cal.getTime(), "183.3", TIME_FORMAT.YEAR);
	dwe.writeAttributeValue("OBS_STATUS", "A");

	//New Series
	dwe.startSeries();
	dwe.writeSeriesKeyValue("REF_AREA", "213");
	dwe.writeSeriesKeyValue("PGI_CONCEPT", "003");
	dwe.writeSeriesKeyValue("DATASOURCE", "BOP");
	dwe.writeSeriesKeyValue("UNITOFMEASURE", "N_M");
	dwe.writeSeriesKeyValue("FREQ", "M");
	dwe.writeAttributeValue("TIME_FORMAT", "P1Y");

	dwe.writeObservation("2009-01", "10.2");
	dwe.writeAttributeValue("OBS_STATUS", "A");
	dwe.writeObservation("2010-02", null);
	dwe.writeAttributeValue("OBS_STATUS", "M");

	dwe.close();
}

In the above tutorial a beanRetrievalManager is u