Communications Part 5: Submission This is the fifth of a series of posts that discuss communication between the Client Tool and the EPA Host System. Perhaps you remember the days when the quarterly reports were put on a floppy disk and mailed to the EPA. Things have come a long way from then. Now submission of data is all electronic. With ECMPS, submission has also been enhanced. This post will cover the basics of the communication between the Client Tool and the EPA Host System during submission.

In the previous four posts in this series, the following topics were covered. It is not necessary to have read the earlier posts, but if you have not read the first one in this series, it might be good to read that before continuing.

1. Basics of communication between the Client Tool and EPA Host System.
2. Initial Synchronization.
3. Basics of Synchronization.
4. Synchronization of Submitted Data

The Submit Module provides the functionality in the Client Tool that transmits data from the Client Tool to the EPA Host System. The data which are sent to the EPA Host System are monitoring plan, QA/Cert, emissions, and evaluation data. These data are transmitted to the EPA Host System using XML files. The XML files are saved on the EPA Host System as raw files of data. Then as part of the submission the data from the XML files are loaded on to the EPA Host System and stored as records in the tables of the EPA Host System Oracle database.

All communication between the Client Tool and the EPA Host System uses Secure Socket Layer (SSL) with which you are familiar if you have ever used a secure Web site such as your bank's Web site. The same security layer is employed by ECMPS. This means that it is unnecessary to use third-party security software such as SecuRemote.

The process of communication takes place as follows. The user clicks the button in the Submission Module which begins the submission process. The Client Tool lets the EPA Host System know that a submission is about to begin. The Client Tool provides the EPA Host System with a description of what data are to be submitted in the submission set. For example, a submission set may include a monitoring plan and QA tests. The EPA Host System responds by providing what are called submission IDs. These numbers uniquely identify the data which are submitted to the EPA Host System in the submission set. In this example, the monitoring plan data and the QA tests would have two different submission IDs.

The Client Tool then gathers up the data to be submitted and produces an XML file for the data (monitoring plan and QA tests) as well as the associated evaluation data. All of these XML files are submitted to the EPA Host System. Each XML file is archived on the EPA Host System. Finally, the Client Tool communicates with the EPA Host System that all of the data in the submission set have been sent. This signals the EPA Host to begin processing the XML files.

The EPA Host System processes the XML files by parsing the data in the files into scratch tables. From there, the data are loaded into the official tables. After all of the data have been loaded into the official tables, the EPA Host System marks the processing as complete.

After the Client Tool has communicated to the EPA Host System that the data are ready to be processed, the Client Tool begins to poll the EPA Host System. The Client Tool checks to see if the processing of the data is complete. It will continue to poll every 30 seconds or so until it determines that the processing is complete. As soon as it receives that confirmation, the Client Tool sets the submission status to complete in the Client Tool. This information is displayed in the Submission Log.

That is all there is to the communication between the Client Tool and the EPA Host System. The next post on communication will cover the communication which takes place during Retrieve.

Labels: Communication

The ECMPS Client Tool Preview (CTP) testing session is in full swing. Each week more stakeholders are signing up to begin working with test copies of their data in order to prepare for the eventual transition to ECMPS. Along with the testing, we continue to receive a steady stream of technical support questions. Here are a couple of those questions from the mailbag.

Missing Old Probe Information

Question
If we changed out our probes, discarded the old ones, and do not have any data related to the old probes, how will get past the errors that state the probe data does not cover the entire evaluation period?

Answer
The only way to clear the error is to have an active probe component defined for the entire period that the monitoring system has been in use since January 1st of 2003. To define a valid probe component all you need is a Component ID, Manufacturer Name, and Serial Number. The purpose of having the manufacturer and serial number information is so that electronically represented components can be identified in the field. It would be much better if you can find the information, but if the probe components are not longer in use it, is less important to have the actual data.

I would suggest adding the old probes with a manufacturer and serial number of "Unknown". Then add the the new probes with the known manufacturer and serial number information. The old probes should be ended on the correct date and hour in the system component record, and the new probes should be begin on the correct date and hour in the system component record.

For more information about adding a probe component, view the tutorial, "ECMPS Monitoring Plan Corrections, Part 2: Adding a Probe" on the Tutorials page.

QA/Cert Data Baseline

Question
Do I need to retrieve the QA/Cert data from the EPA Host System initially to have some "base line" of previously submitted data, or can I just start by adding the QA/Cert data from this quarter going forward?

Answer
During the initial synchronization, the ECMPS Client Tool retrieves what is termed Test History data. These data are the essential QA/Cert data which the Client Tool needs from all previously submitted QA/Cert data. The Test History data are a subset of all of the previously submitted QA/Cert data. The Client Tool needs these data in order to perform the data evaluations in the Client Tool which require data from previously submitted QA/Cert data. For example, the bias adjustment factor (BAF). You can view these data through the Test History Report which is found in the QA/Cert Module of the Client Tool. The data which are retrieved in the initial synchronization are all of the previously submitted QA/Cert data from 2003 to the present. Going forward the Test History data will automatically be in your Client Tool because it is generated in the Client Tool every time an evaluation is run on new QA/Cert data.

If you would like to view all of the details of the previously submitted tests, use the Retrieve module to retrieve all of the data (e.g. run data, summary data) for any of the QA/Cert data which have been previously submitted.

For more information about the initial synchronization, view the previous post on initial synchronization. (This post does not use the term Test History data because it had not been adopted at that time, but Test History data is part of supplementary data.)

Labels: Communication, Mailbag, Synchronization

This is the fourth of a series of posts that discuss communication between the Client Tool and the EPA Host System. The first post described the basics of communication between the Client Tool and EPA Host System. The second post covered initial synchronization which takes place the first time the Client Tool is opened after installation. The third post discussed the basics of synchronization.

This post will cover the synchronization of submitted data. The topics covered will include in what instances does synchronization of submitted data matter and what happens when submitted data is synchronized.

I would suggest reading the third post in this series before continuing with this one if you have not already read it or it has been a while since you read it. That post lays out some of the basic concepts of synchronization which are helpful to know for understanding the information in this post.

Submitted data are the data which are submitted to the EPA Host System from the Client Tool. The origin of the data is through data submissions from affected sources, and the data includes monitoring plan, QA test and certification, and emissions data.

Synchronization of submitted data really only matters when there are different installations of the Client Tool which access the same data. The reason for that is that synchronization of submitted data is based on comparing the data on the EPA Host System to the data which are in the Client Tool database. If only one installation of the Client Tool (e.g., one Client Tool database) ever accesses a certain facility's data, the data will always be in sync. The data can also always be in sync even if multiple users work with the same data. This will be the case for a network installation of the Client Tool in which multiple workstations access a single Client Tool database installed on the company network. The reason is that there is one Client Tool database which is kept in sync with the Client Tool no matter how many users access the data.

For the case of when synchronization matters, here is an example of what occurs to keep separate installations of the Client Tool in sync with submitted data. User A works with Facility X. User A corrects some part of the monitoring plan and submits the modified monitoring plan to the EPA Host System. User B also works with Facility X. After the EPA Host System has received the modified monitoring plan submitted by User A, User B logs on to the EPA Host System through the Client Tool. At this point User B has a monitoring plan for Facility X that is different than the monitoring plan for Facility X which is found on the EPA Host System. The Client Tool automatically recognizes that the monitoring plan data is out of sync. The monitoring plan data for Facility X is automatically retrieved from the EPA Host System and put in User B's Client Tool database. After the data is retrieved, User B's Client Tool is in sync with the EPA Host System.

Just like the synchronization of meta data, synchronization of submitted data automatically takes place when certain actions are taken by the user. These actions are logging in to the EPA Host System, retrieving data through the Retrieve Module, and submitting data through the Submit Module. Prior to an evaluation through the Evaluate Module, the user is given the option to synchronize the user's data, but the user is not required to synchronize. At any of these times, the Client Tool performs a synchronization with the EPA Host System to determine if any of the data which are associated with that user have changed on the EPA Host System. If there are differences in the submitted data between the EPA Host System and the Client Tool, the Client Tool retrieves the updated submitted data. The Client Tool will remain in sync until there are any further submitted data changes on the EPA Host System.

For more information on synchronization, view the PowerPoint presentation on synchronization by clicking here.

Labels: Communication

This is the third of several posts that discuss communication between the Client Tool and the EPA Host System. The first post described the basics of communication between the Client Tool and EPA Host System. The second post covered initial synchronization which takes place the first time the Client Tool is opened after installation.

In this post, the basics of synchronization will be discussed. These basics include the two types of data which can be synchronized, the more common form of synchronization, and when synchronization takes place.

Synchronization takes place in order to ensure that the Client Tool and the EPA Host System are in sync. There are two types of data which can be out of sync. The first data are what might be called submitted data. Submitted data are the data which are submitted to the EPA Host System from the Client Tool. The origin of the data is through data submissions from affected sources, and the data includes monitoring plan, QA test and certification, and emissions data. The second data are what might be called meta data. Meta data assist the operation of the Client Tool, and the source of the meta data is the EPA. A good example of this type of data would be what is called submission access data or the data which allows sources to submit emissions data during the reporting month.

There is only one way in which the Client Tool and the EPA Host System will ever not be in sync for the submitted data. That way is if there is more than one Client Tool which submits the same data for a particular configuration to the EPA Host System. In other words, if there is only one Client Tool which submits monitoring plan, QA and certification, and emissions data for a particular configuration, the Client Tool will never be out of sync with regard to submitted data for that configuration.

There is one additional qualification. The multiple Client Tool scenario can only be out of sync with regard to the first type of data if the multiple Client Tools use separate databases. If the Client Tools share a database, the submitted data will always been in sync for all Client Tools which access the data in the common database.

The more likely scenario for many sources will be that the meta data is out of sync. For example, if a user has not opened the Client Tool for several weeks, the user might find that when the user opens the Client Tool at the beginning of the reporting period that the submission access data are out of sync. During the synchronization, the Client Tool will automatically retrieve the submission access data which will allow the user to submit emissions data during the reporting period.

Synchronization automatically takes place when certain actions are taken by the user. These actions are logging in to the EPA Host System, retrieving data through the Retrieve Module, and submitting data through the Submit Module. Prior to an evaluation through the Evaluate Module, the user is given the option to synchronize the user's data, but the user is not required to synchronize. At any of these times, the Client Tool performs a synchronization with the EPA Host System to determine if any of the data which are associated with that user have changed on the EPA Host System. If there are differences in the meta data between the EPA Host System and the Client Tool, the Client Tool retrieves the updated meta data. The Client Tool will remain in sync until there are any further meta data changes on the EPA Host System.

The next post on communication will cover the synchronization of submitted data.

Labels: Communication

In part one of this series on communication, some basics about communication between the Client Tool and the EPA Host System were described. This post will provide information about the first method of communication--initial synchronization.

The initial synchronization takes place whenever the Client Tool database needs to be initialized. The database needs to be initialized because it is empty. The most typical instance of when the database is empty is is when you first start up the Client Tool after it has been installed. As soon as you start ECMPS and the database is empty, you will be prompted for your user information including your CBS User Name.



At the next screen, you will need to supply your CBS password. Upon successfully logging in, the initial synchronization will begin.



During the initial synchronization, the Client Tool will initialize the Client Tool database by retrieving data from the EPA Host System. The data retrieved will depend upon the access to the data to which the user has. This access is set in the CAMD Business System (CBS). At the Stakeholder meeting on December 5th, more information will be provided about ECMPS agents and access to the data.

While the data is being retrieved, a screen will appear which lists the data that is being loaded into the Client Tool database.



The first data loaded is the facility and unit inventory data. These data will be all of the facilities and units for which the user has a relationship as either a representative or an ECMPS agent. The next data which are loaded are the monitoring plan data and any associated supplementary data. Again, the data loaded is the data for which the user has a relationship as established in CBS.

The supplementary data includes used identifier data, submission access data, QA supplemental data, QA certification event data, test extension and exemption data, and operating supplemental data. Used identifier data is data which insures the referential integrity of stack, system, component, and formula identifiers. Submission access data determines when emissions data can be submitted to the EPA Host System. QA supplemental data is the test results for QA tests. QA certification event data and test extension and exemption data are exactly what their names are. Operating supplemental data are the EPA accepted emissions values. Note, the supplemental data cannot be edited in the Client Tool. Instead, it is retrieved to the Client Tool in order to assist with the evaluation of the data.

At the completion of the initial synchronization, the Client Tool database has been populated with all of the data which are necessary for the user to begin to use the Client Tool. All of the monitoring plan data will be identical to what are on the EPA Host System. Similarly, the certification event data, test extension and exemption data, and the QA and emissions supplementary data in the Client Tool will be identical to what is on the EPA Host System.

If monitoring plan data which the user is expecting to see are not visible, the user should log on to the CAMD Business System to verify that the user has a relationship with that data.

The next post will cover synchronization after the initial synchronization.

Labels: Communication

Part 1: Basics of Communication

There are two main parts to ECMPS. There is the Client Tool and the EPA Host System. The ECMPS Client Tool, which is a desktop application, connects to the EPA Host System, which is an Oracle database, through Web services. The purpose of the Web services is to transmit data between the two parts. This post and several that will follow are dedicated to explaining all of the various methods of communication that can take place between the two major pieces of ECMPS. This post will cover basic communication information.

Before describing some of the specifics of communication in later posts, it might be good to mention a couple of basic concepts about all communication between the Client Tool and the EPA Host System.

First, communication between the Client Tool and the EPA Host System takes place over the Internet. As will be described in the next post, it is not possible to complete the installation of the Client Tool without an Internet connection. However, once the Client Tool has been set up, it is possible to work with your data without an Internet connection, although this does limit some of the functionality of the Client Tool. For example, without an Internet connection, it is not possible to submit data to the EPA, retrieve data from the EPA, or synchronize. Later posts will provide more information about these methods of communication.

Second, all communication with the EPA Host System is initiated by the Client Tool. In other words, a user has to perform some action which triggers the Client Tool to communicate with the EPA Host System. The EPA Host System never initiates communication between the two parts.

Third, communication between the Client Tool and the EPA Host System takes place using Secure Socket Layer (SSL). This means that all of the data which is transmitted is encrypted using the same type of system that secure Web sites use (e.g., https). For example, using SSL protects the CBS password when it is transmitted from the Client Tool to the EPA Host System during authentication.

That covers the basics of communication. The next post will describe initial synchronization.

Labels: Communication