Java // Documentation // Readable / Writeable Data Location

Sdmxsource uses streaming technology to process large files. To read data as a Stream an InputStream must be used, and to output data as a Stream an OutputStream must be used. The problem with an InputStream is that it can not be reset reliably (depending on implementation, reseting an InputStream is usually not supported). This means that for processes that need to read from a Stream, and re-read from the same Stream, an InputStream can not be relied upon.

To overcome this limitation, sdmxsource uses temporary files to stream information from. In a lot of circumstances the calling application will want to call a method passing in an InputStream, or file location, or possibly a URL. Instead of exposing multiple interfaces taking different parameters for essentially the same method, sdmxsource uses a ReadableDataLocation interface. The ReadableDataLocation Interface defines two methods, one to get a new instance of an InputStream on each method call, and one to close all the underlying resources. The implementation of ReadableDataLocation provided by sdmxsource is ReadableDataLocationTmp, and can be constructed from a String, a File, a URL, a URI, or an InputStream. If created from a URL or an InputStream, the ReadableDataLocationTmp will create a local temporary file which it copies the data to. On closing the ReadableDataLocationTmp the temporary file will be deleted.

Conversely there exists the WriteableDataLocation. This extends ReadableDataLocation by providing the means to get the OutputStream so that the underlying source can be written to. The implementation provided by sdmxsource is the WriteableDataLocationTmp and is intended to be constructed using the default constructor (although other constructors can be used). The default (no argument) constructor will create a temporary file, to which the interface will offer methods to get both the OutputStream to write the data, and the InputStream to read the data. On closing the ReadableDataLocationTmp the temporary file will be deleted.


Examples

Showing how a new InputStream can be obtained from a single ReadableDataLocation

/**
 * Copies the input stream to 2 output streams
 * @param is
 * @param out1
 * @param out2
 */
private void copyStreams(InputStream is, OutputStream out1, OutputStream out2) {
  ReadableDataLocation rdl = new ReadableDataLocationTmp(is);
  StreamUtil.copyStream(rdl.getInputStream(), out1);
  StreamUtil.copyStream(rdl.getInputStream(), out2);
}

Showing a ReadableDataLocation being used to obtain information about the data available
in the provided InputStream

  /**
   * Demonstrates how the InputStream from a ReadableDataLocation can be read by
   * two different methods
   */
  private void processMessage(InputStream is) {
    ReadableDataLocation rdl = new ReadableDataLocationTmp(is);
    MESSAGE_TYPE messageType = SdmxMessageUtil.getMessageType(rdl);
    SDMX_SCHEMA schemaVersion = SdmxMessageUtil.getSchemaVersion(rdl);

    if(messageType == MESSAGE_TYPE.STRUCTURE) {
	//Process structure document
    } else if(messageType == MESSAGE_TYPE.QUERY) {
	//Process query document
    }
  }

Showing a WriteableDataLocation being used to both write data to, and read data from

/**
 * Processes SdmxBeans the result is written to an OutputStream.
 * A ReadableDataLocation is returned giving access
 * to an InputStream which can be used to read the message
 */
private ReadableDataLocation processBeans(SdmxBeans beans) {
  WriteableDataLocation wdl = new WriteableDataLocationTmp();
  OutputStream out = wdl.getOutputStream();
  processMessage(out);
  return wdl;
}

Brining it all togeather

/**
 * Process structures, write the response to the output stream
 * @param is
 * @param out
 */
private void processMessage(InputStream is, OutputStream out) {
  ReadableDataLocation input = new ReadableDataLocationTmp(is);
  SdmxBeans beans = structureParsingManager.parseStructures(input).getStructureBeans(false);
  ReadableDataLocation result = processBeans(beans);
  try {
    StreamUtil.copyStream(result.getInputStream(), out);
  } finally {
    //Ensure the result is closed so that the underlying resources are
    //deleted from the file system
    result.close();
  }
}

/**
 * Processes SdmxBeans the result is written to an OutputStream.
 * A ReadableDataLocation is returned giving access to an InputStream
 * which can be used to read the message
 */
private ReadableDataLocation processBeans(SdmxBeans beans) {
  WriteableDataLocation wdl = new WriteableDataLocationTmp();
  OutputStream out = wdl.getOutputStream();
  processMessage(out);
  return wdl;
}