[WP3] Evaluating Oracle Streams to federate Astro-Wise

[Danny R. Boxhoorn]

Hello Joel and Fëdor,

Over the last weeks I've been thinking about how we want to set up
a database federation for Astro-Wise with Oracle Streams.

First of all, the idea is that each database in the federation has a complete
copy of all the relevant data. Users populate their local database as they
do now, but in addition this data is copied to all other nodes.
Whether to always copy all data or only subsets, how often, etc. are questions
that have to be answered separately. The minimum, and most difficult to
implement, functionality that is needed is the ability to copy data to
all databases in the federation.

The goal of the evaluation should be to implement a copy mechanism and
see how well it works. This includes checks how often and which subsets.

What has become clear to me is that we are going for a "Messaging"
implementation. It could turn out that Streams Messaging is not suited for the
task or requires too much maintenance. In that case a Python Messaging
implementation has to be considered.

The steps that we need to take are

1) Have two separate databases each of our sites.
   Make sure that Oracle Streams works from one database to the other for
   a simple table.

2) Get the Streams Messaging example in Chapter 18 of the Streams guide
   working on each of our sites.

3) Get Streams Messaging working for a simple object table on each of
   our sites. For example the Astro-Wise "Chip" object table.
   Note that we do not care about DDL. Streaming DML has to work, and if we
   get Streaming DDL to work that is a bonus.

4) Get Streams Messaging working for a more complex object containing
   references on each of our sites. "BiasFrame" is probably a nice example.
   Again, we do not care about DDL.

5) Activate Messaging from the Python code or via triggers.

6) Connect our databases and enable peer-to-peer messaging.

I'm currently busy with step 2 and I would like to ask you what you think
of the above steps and at which step you are.

The are many open questions, such as how to add an new node to an existing
federation and how to bootstrap it. And many, many more.
This is just a sketchy workplan, so just tell me what you think.
If you have ideas how to organize the three of us, please do so.
I've been thinking about phoning each other on a regular basis or
a short status e-mail every other day to see how we are progressing.
Perhaps it's too early for that. If you have an opinion, just put it
forward!

When the mailing-lists are working again, I suggest that we use the `wp3'
mailing list to send our discussions to. I'll tell you when it's open
for subscription again.
For your information, the url is http://listman.astro-wise.org/

We also need a place in CVS to put everything useful we create and
I suggest we use opipe/Database for that.
Joel, do you already have CVS write access?
If not, just follow the instructions in

   http://www.astro-wise.org/howto/cvs.html

I assume they are still correct. [)

One final thing that I recommend is to look at the "International Virtual
Observatory Alliance - Interoperability Meeting 2003" presentations.
See

   http://ivoa.net/twiki/bin/view/IVOA/InterOpOct2003

The sections on VO Query Language, Data Model, Web & Grid services and
others are typically ones that we should keep an eye on.
If there's something interesting or useful on the "database federation/grid"
front we should bring it to each others attention.

That should be enough for now.

Ciao!

                                                   Danny